WEBVTT

00:00:00.000 --> 00:00:01.972 align:middle line:90%
[SQUEAKING]

00:00:01.972 --> 00:00:03.451 align:middle line:90%
[RUSTLING]

00:00:03.451 --> 00:00:04.930 align:middle line:90%
[CLICKING]

00:00:04.930 --> 00:00:16.280 align:middle line:90%


00:00:16.280 --> 00:00:17.400 align:middle line:90%
RAMA RAMAKRISHNAN: OK.

00:00:17.400 --> 00:00:17.900 align:middle line:90%
All right.

00:00:17.900 --> 00:00:20.520 align:middle line:90%
Let's get going.

00:00:20.520 --> 00:00:22.280 align:middle line:90%
Today is going to be packed.

00:00:22.280 --> 00:00:24.880 align:middle line:84%
I'm going to spend the first
roughly half of the lecture

00:00:24.880 --> 00:00:27.920 align:middle line:90%
on actually building a model--

00:00:27.920 --> 00:00:29.840 align:middle line:90%
a Keras model in Colab--

00:00:29.840 --> 00:00:32.360 align:middle line:84%
to solve the heart disease
problem we saw earlier,

00:00:32.360 --> 00:00:34.880 align:middle line:84%
and then switch gears
halfway, and then

00:00:34.880 --> 00:00:37.660 align:middle line:84%
talk about how to solve
image classification.

00:00:37.660 --> 00:00:40.850 align:middle line:84%
So we're going to
do two Colabs today.

00:00:40.850 --> 00:00:42.100 align:middle line:90%
I've been talking about Colab.

00:00:42.100 --> 00:00:43.740 align:middle line:90%
Colab, I've been teasing you.

00:00:43.740 --> 00:00:45.300 align:middle line:90%
We'll actually do Colabs today.

00:00:45.300 --> 00:00:46.020 align:middle line:90%
All right.

00:00:46.020 --> 00:00:47.722 align:middle line:90%
So summary of where we were.

00:00:47.722 --> 00:00:49.680 align:middle line:84%
By the way, I've shut
off the lights in the top

00:00:49.680 --> 00:00:51.480 align:middle line:84%
because, when I
switch to Colab, it's

00:00:51.480 --> 00:00:53.360 align:middle line:84%
going to be much better for you
folks, particularly the folks

00:00:53.360 --> 00:00:54.950 align:middle line:84%
in the back, to
be able to see it.

00:00:54.950 --> 00:00:57.840 align:middle line:84%
But I hope you can see
the slide right now.

00:00:57.840 --> 00:00:59.890 align:middle line:90%
Yes?

00:00:59.890 --> 00:01:00.390 align:middle line:90%
OK.

00:01:00.390 --> 00:01:01.010 align:middle line:90%
Great.

00:01:01.010 --> 00:01:04.836 align:middle line:84%
So this is just a quick recap
of what we did last class.

00:01:04.836 --> 00:01:07.283 align:middle line:84%
Broadly speaking,
training a neural network

00:01:07.283 --> 00:01:09.950 align:middle line:84%
essentially is no different than
training other kinds of models.

00:01:09.950 --> 00:01:12.870 align:middle line:84%
We have a bunch of parameters,
i.e., weights and biases,

00:01:12.870 --> 00:01:15.170 align:middle line:84%
and we need to use
the data to find

00:01:15.170 --> 00:01:16.990 align:middle line:90%
good values of those weights.

00:01:16.990 --> 00:01:18.190 align:middle line:90%
And what does good mean?

00:01:18.190 --> 00:01:20.130 align:middle line:84%
Typically, it means
that we define

00:01:20.130 --> 00:01:23.570 align:middle line:84%
some measure of discrepancy
between what the model predicts

00:01:23.570 --> 00:01:26.330 align:middle line:84%
for a given set of weights and
what the right answer is, what

00:01:26.330 --> 00:01:27.910 align:middle line:90%
the ground truth answer is.

00:01:27.910 --> 00:01:30.910 align:middle line:84%
And then we try to find weights
that minimize this discrepancy.

00:01:30.910 --> 00:01:31.430 align:middle line:90%
That's it.

00:01:31.430 --> 00:01:33.970 align:middle line:84%
And this notion of discrepancy
is called a loss function.

00:01:33.970 --> 00:01:37.250 align:middle line:84%
So broadly speaking, the
overall training flow

00:01:37.250 --> 00:01:38.830 align:middle line:90%
is that you define some network.

00:01:38.830 --> 00:01:39.533 align:middle line:90%
It has an input.

00:01:39.533 --> 00:01:40.950 align:middle line:84%
It goes through a
bunch of layers.

00:01:40.950 --> 00:01:42.230 align:middle line:84%
You come up with
some predictions.

00:01:42.230 --> 00:01:43.272 align:middle line:90%
You take the predictions.

00:01:43.272 --> 00:01:44.590 align:middle line:90%
You take the true values.

00:01:44.590 --> 00:01:46.910 align:middle line:84%
And then those two go
into the loss function,

00:01:46.910 --> 00:01:48.365 align:middle line:90%
i.e., the discrepancy function.

00:01:48.365 --> 00:01:49.990 align:middle line:84%
And then you come up
with a loss score.

00:01:49.990 --> 00:01:52.110 align:middle line:84%
And then you send
it to the optimizer,

00:01:52.110 --> 00:01:55.290 align:middle line:84%
which then proceeds to calculate
the gradient of this loss

00:01:55.290 --> 00:01:57.190 align:middle line:84%
function with respect
to all the parameters.

00:01:57.190 --> 00:01:59.900 align:middle line:84%
And then it updates all the
weights using that gradient,

00:01:59.900 --> 00:02:01.940 align:middle line:90%
and then the process repeats.

00:02:01.940 --> 00:02:02.640 align:middle line:90%
That's it.

00:02:02.640 --> 00:02:04.420 align:middle line:90%
So that is the training flow.

00:02:04.420 --> 00:02:05.500 align:middle line:90%
Quick recap.

00:02:05.500 --> 00:02:08.820 align:middle line:84%
Now, we also talked about
the optimization algorithm

00:02:08.820 --> 00:02:12.620 align:middle line:84%
we're going to use, which
is called gradient descent.

00:02:12.620 --> 00:02:16.360 align:middle line:84%
And gradient descent, as you
noticed, in each iteration,

00:02:16.360 --> 00:02:20.100 align:middle line:84%
every data point is being
used to make predictions

00:02:20.100 --> 00:02:21.700 align:middle line:84%
and therefore to
calculate the loss,

00:02:21.700 --> 00:02:23.900 align:middle line:84%
and then to calculate
the gradient.

00:02:23.900 --> 00:02:27.300 align:middle line:84%
And then we pointed out that
gradient descent is actually not

00:02:27.300 --> 00:02:30.420 align:middle line:84%
as good as something called
stochastic gradient descent--

00:02:30.420 --> 00:02:32.660 align:middle line:84%
stochastic gradient
descent-- where,

00:02:32.660 --> 00:02:35.320 align:middle line:84%
instead of taking
all the points,

00:02:35.320 --> 00:02:38.460 align:middle line:84%
we just randomly choose
a small number of points,

00:02:38.460 --> 00:02:41.900 align:middle line:84%
pretend for a moment as if those
are the only points we have,

00:02:41.900 --> 00:02:45.240 align:middle line:84%
make predictions, calculate
loss, calculate gradient,

00:02:45.240 --> 00:02:46.420 align:middle line:90%
and go on.

00:02:46.420 --> 00:02:49.300 align:middle line:84%
So that was the basic idea
behind stochastic gradient

00:02:49.300 --> 00:02:50.860 align:middle line:90%
descent.

00:02:50.860 --> 00:02:52.120 align:middle line:90%
Two different kinds of things.

00:02:52.120 --> 00:02:54.860 align:middle line:84%
Now, what it means is
that when we actually

00:02:54.860 --> 00:02:58.300 align:middle line:84%
start training the model,
as we will in a few minutes,

00:02:58.300 --> 00:03:00.740 align:middle line:84%
because we only take a
few points at a time,

00:03:00.740 --> 00:03:03.380 align:middle line:84%
we have to be a bit
careful in what's going on.

00:03:03.380 --> 00:03:04.840 align:middle line:84%
And I want to make
sure you clearly

00:03:04.840 --> 00:03:07.200 align:middle line:84%
understand what the differences
are before we actually

00:03:07.200 --> 00:03:08.300 align:middle line:90%
get to the Colab.

00:03:08.300 --> 00:03:10.560 align:middle line:90%
OK?

00:03:10.560 --> 00:03:11.140 align:middle line:90%
All right.

00:03:11.140 --> 00:03:14.800 align:middle line:84%
So there is the
notion of an epoch.

00:03:14.800 --> 00:03:17.760 align:middle line:84%
An epoch essentially just
means that we make one

00:03:17.760 --> 00:03:19.880 align:middle line:90%
pass through the training data.

00:03:19.880 --> 00:03:20.960 align:middle line:90%
All the training data.

00:03:20.960 --> 00:03:22.660 align:middle line:90%
We make one pass through it.

00:03:22.660 --> 00:03:27.280 align:middle line:84%
And so what is one pass is
that if you have something

00:03:27.280 --> 00:03:31.080 align:middle line:84%
like gradient descent, one
pass means every data point

00:03:31.080 --> 00:03:32.460 align:middle line:90%
is sent through the network.

00:03:32.460 --> 00:03:34.820 align:middle line:84%
We calculate its predictions,
calculate the loss,

00:03:34.820 --> 00:03:36.740 align:middle line:90%
calculate the gradient.

00:03:36.740 --> 00:03:39.020 align:middle line:84%
We run every training
sample through it.

00:03:39.020 --> 00:03:42.300 align:middle line:84%
We calculate the gradient,
which is just this thing here.

00:03:42.300 --> 00:03:46.400 align:middle line:84%
I mean, I will sometimes
say d of loss, dW.

00:03:46.400 --> 00:03:48.260 align:middle line:84%
Derivative of loss
with respect to W.

00:03:48.260 --> 00:03:50.680 align:middle line:84%
Sometimes I might use
this nabla symbol.

00:03:50.680 --> 00:03:52.640 align:middle line:90%
These are all interchangeable.

00:03:52.640 --> 00:03:54.720 align:middle line:84%
So we'll calculate
the gradient, and then

00:03:54.720 --> 00:03:57.330 align:middle line:84%
we update using some
version of this.

00:03:57.330 --> 00:04:01.130 align:middle line:84%
But we just do it once
at the end of the epoch.

00:04:01.130 --> 00:04:03.930 align:middle line:84%
Because if you have 10 billion
data points, every one of them

00:04:03.930 --> 00:04:06.330 align:middle line:84%
flows through, you get
10 billion outputs.

00:04:06.330 --> 00:04:08.080 align:middle line:84%
And then we calculate
the epoch just once.

00:04:08.080 --> 00:04:10.205 align:middle line:84%
At the end of this thing,
we calculate the gradient

00:04:10.205 --> 00:04:10.890 align:middle line:90%
and update once.

00:04:10.890 --> 00:04:13.330 align:middle line:90%
One update per epoch.

00:04:13.330 --> 00:04:14.970 align:middle line:90%
Yes?

00:04:14.970 --> 00:04:18.610 align:middle line:84%
Now, in stochastic gradient
descent, what we do

00:04:18.610 --> 00:04:21.930 align:middle line:84%
is that we process
the data in batches.

00:04:21.930 --> 00:04:24.890 align:middle line:84%
Small numbers of
points at a time.

00:04:24.890 --> 00:04:27.150 align:middle line:84%
And these are called--
technically speaking,

00:04:27.150 --> 00:04:28.872 align:middle line:90%
they're called minibatches.

00:04:28.872 --> 00:04:29.830 align:middle line:90%
I don't know about you.

00:04:29.830 --> 00:04:31.455 align:middle line:84%
I just get tired of
saying minibatches,

00:04:31.455 --> 00:04:34.090 align:middle line:84%
so I'm just going to say
batches from this point on.

00:04:34.090 --> 00:04:34.690 align:middle line:90%
OK?

00:04:34.690 --> 00:04:37.650 align:middle line:84%
And in fact, that is widely
done in the literature.

00:04:37.650 --> 00:04:40.670 align:middle line:84%
So we'll have to
process it in batches.

00:04:40.670 --> 00:04:42.190 align:middle line:90%
So we take the training data.

00:04:42.190 --> 00:04:44.690 align:middle line:84%
And then we divide
it up into batches--

00:04:44.690 --> 00:04:47.810 align:middle line:84%
batch 1, batch 2, all the
way till the final batch.

00:04:47.810 --> 00:04:52.010 align:middle line:84%
And so what we do
is, for each batch,

00:04:52.010 --> 00:04:54.380 align:middle line:84%
we basically do
gradient descent.

00:04:54.380 --> 00:04:56.980 align:middle line:84%
For each batch, we take
batch 1, and then we

00:04:56.980 --> 00:04:59.815 align:middle line:84%
run just the training
samples in that batch

00:04:59.815 --> 00:05:01.440 align:middle line:84%
through the network
to get predictions.

00:05:01.440 --> 00:05:03.800 align:middle line:84%
We calculate the gradient,
we update the parameters,

00:05:03.800 --> 00:05:05.040 align:middle line:90%
and then we go to batch 2.

00:05:05.040 --> 00:05:07.320 align:middle line:84%
Then we go to batch 3
and so on and so forth.

00:05:07.320 --> 00:05:11.280 align:middle line:84%
So pictorially, this is how
it's going to look like.

00:05:11.280 --> 00:05:14.220 align:middle line:84%
Let's say the first
batch is, say, 32 points.

00:05:14.220 --> 00:05:15.672 align:middle line:90%
We take those 32 points.

00:05:15.672 --> 00:05:17.880 align:middle line:84%
We run it through the network,
get all the stuff out.

00:05:17.880 --> 00:05:20.500 align:middle line:84%
We calculate the gradient,
update the weights.

00:05:20.500 --> 00:05:25.300 align:middle line:84%
So when we now get to batch
2, the weights have changed.

00:05:25.300 --> 00:05:26.773 align:middle line:90%
They have been updated.

00:05:26.773 --> 00:05:28.940 align:middle line:84%
And then we do the same
thing with batch 2, batch 3,

00:05:28.940 --> 00:05:31.258 align:middle line:84%
and all the way till we get
to the end of the thing.

00:05:31.258 --> 00:05:32.800 align:middle line:84%
And when we are done
with this thing,

00:05:32.800 --> 00:05:36.300 align:middle line:84%
this whole thing
is called a what?

00:05:36.300 --> 00:05:38.100 align:middle line:90%
An epoch.

00:05:38.100 --> 00:05:40.140 align:middle line:90%
This whole thing is an epoch.

00:05:40.140 --> 00:05:42.780 align:middle line:90%
OK?

00:05:42.780 --> 00:05:43.710 align:middle line:90%
All right.

00:05:43.710 --> 00:05:45.460 align:middle line:84%
So the question, of
course, is that if you

00:05:45.460 --> 00:05:47.085 align:middle line:84%
have a bunch of data
points, and you're

00:05:47.085 --> 00:05:49.180 align:middle line:84%
going to run stochastic
gradient descent on it,

00:05:49.180 --> 00:05:54.550 align:middle line:84%
in a particular epoch, how many
batches are going to be there?

00:05:54.550 --> 00:05:56.410 align:middle line:84%
How many batches are
going to be there?

00:05:56.410 --> 00:05:58.410 align:middle line:84%
Now, Keras is going to
calculate all this stuff.

00:05:58.410 --> 00:05:59.290 align:middle line:84%
You don't have to
worry about it.

00:05:59.290 --> 00:06:01.590 align:middle line:84%
But you just need to understand
exactly what happens.

00:06:01.590 --> 00:06:04.070 align:middle line:84%
So my philosophy, by
the way, is that you

00:06:04.070 --> 00:06:06.088 align:middle line:84%
have to know the details
of what's going on.

00:06:06.088 --> 00:06:08.630 align:middle line:84%
If you don't know the details,
and if you haven't figured out

00:06:08.630 --> 00:06:10.840 align:middle line:84%
at least once, you
will not actually

00:06:10.840 --> 00:06:12.590 align:middle line:84%
be able to think new
and creative thoughts

00:06:12.590 --> 00:06:14.510 align:middle line:90%
for a new problem.

00:06:14.510 --> 00:06:17.630 align:middle line:84%
It's because the concepts are
not manipulable in your head

00:06:17.630 --> 00:06:18.570 align:middle line:90%
yet.

00:06:18.570 --> 00:06:19.070 align:middle line:90%
OK.

00:06:19.070 --> 00:06:23.930 align:middle line:90%


00:06:23.930 --> 00:06:27.350 align:middle line:90%
Please use the microphone.

00:06:27.350 --> 00:06:31.430 align:middle line:84%
AUDIENCE: So when we talk about
SDG and we are talking about--

00:06:31.430 --> 00:06:32.890 align:middle line:84%
we are only taking
some part of it,

00:06:32.890 --> 00:06:35.910 align:middle line:84%
is it what we are saying is that
we only take some variables,

00:06:35.910 --> 00:06:37.930 align:middle line:84%
or we're only taking
some part of the data?

00:06:37.930 --> 00:06:40.283 align:middle line:84%
RAMA RAMAKRISHNAN: We
are taking some rows.

00:06:40.283 --> 00:06:41.950 align:middle line:84%
AUDIENCE: We're taking
only-- oh, right.

00:06:41.950 --> 00:06:42.310 align:middle line:90%
So that--

00:06:42.310 --> 00:06:43.310 align:middle line:90%
RAMA RAMAKRISHNAN: Data points.

00:06:43.310 --> 00:06:44.570 align:middle line:84%
AUDIENCE: That means
the batch itself.

00:06:44.570 --> 00:06:45.695 align:middle line:90%
RAMA RAMAKRISHNAN: Exactly.

00:06:45.695 --> 00:06:48.130 align:middle line:84%
So for example, let's say
you have 1,000 data points,

00:06:48.130 --> 00:06:51.250 align:middle line:84%
1,000 rows of observations,
1,000 patients in the heart

00:06:51.250 --> 00:06:54.650 align:middle line:84%
disease example, or 1,000 images
that you're trying to classify.

00:06:54.650 --> 00:06:58.470 align:middle line:84%
You take, let's say, 32 of those
images, 32 of those patients,

00:06:58.470 --> 00:06:59.610 align:middle line:90%
and that's a batch.

00:06:59.610 --> 00:07:01.562 align:middle line:84%
Then you go to the next
32, then the next 32,

00:07:01.562 --> 00:07:03.770 align:middle line:84%
and so on and so forth,
until you run out of patients

00:07:03.770 --> 00:07:05.250 align:middle line:90%
or run out of images.

00:07:05.250 --> 00:07:06.912 align:middle line:84%
AUDIENCE: And each
iterative time you

00:07:06.912 --> 00:07:09.370 align:middle line:84%
are updating with the weights,
the weights that you've got?

00:07:09.370 --> 00:07:09.650 align:middle line:90%
RAMA RAMAKRISHNAN: Correct.

00:07:09.650 --> 00:07:12.210 align:middle line:84%
AUDIENCE: And it means
you keep correcting it

00:07:12.210 --> 00:07:13.410 align:middle line:90%
or keep moving towards--

00:07:13.410 --> 00:07:14.440 align:middle line:84%
RAMA RAMAKRISHNAN: You're
basically updating the weights

00:07:14.440 --> 00:07:15.107 align:middle line:90%
as you go along.

00:07:15.107 --> 00:07:17.138 align:middle line:90%
AUDIENCE: Updating the weights.

00:07:17.138 --> 00:07:18.930 align:middle line:84%
AUDIENCE: And what we
are calling the epoch

00:07:18.930 --> 00:07:20.730 align:middle line:84%
is ultimately the
equation of loss function

00:07:20.730 --> 00:07:21.990 align:middle line:90%
that we are trying to get.

00:07:21.990 --> 00:07:23.630 align:middle line:84%
RAMA RAMAKRISHNAN:
No, an epoch-- see,

00:07:23.630 --> 00:07:28.050 align:middle line:84%
the thing to remember is that
here, this whole thing is called

00:07:28.050 --> 00:07:30.850 align:middle line:84%
an epoch, because we
have to do one full pass

00:07:30.850 --> 00:07:32.210 align:middle line:90%
through the training data.

00:07:32.210 --> 00:07:37.250 align:middle line:84%
But within that epoch, we
update the weights many times.

00:07:37.250 --> 00:07:39.690 align:middle line:84%
Basically, we update the
weights as many times

00:07:39.690 --> 00:07:40.580 align:middle line:90%
as we have batches.

00:07:40.580 --> 00:07:44.010 align:middle line:90%


00:07:44.010 --> 00:07:46.410 align:middle line:90%
All right.

00:07:46.410 --> 00:07:48.630 align:middle line:84%
So to go here-- let's
say, for example--

00:07:48.630 --> 00:07:50.880 align:middle line:84%
basically, the idea is that
you take the training set,

00:07:50.880 --> 00:07:54.917 align:middle line:84%
you divide it by the batch size,
and you choose the batch size.

00:07:54.917 --> 00:07:56.000 align:middle line:90%
You choose the batch size.

00:07:56.000 --> 00:07:58.500 align:middle line:84%
And we'll talk about, well, How
do you choose that? later on.

00:07:58.500 --> 00:07:59.583 align:middle line:90%
You choose the batch size.

00:07:59.583 --> 00:08:02.580 align:middle line:84%
And once you choose the size,
just divide it and round it up.

00:08:02.580 --> 00:08:04.960 align:middle line:84%
So for example, as you
will see in the Colab,

00:08:04.960 --> 00:08:08.180 align:middle line:84%
a training set is going
to be 194 patients.

00:08:08.180 --> 00:08:10.980 align:middle line:84%
And then we're going to
choose a batch size of 32.

00:08:10.980 --> 00:08:14.660 align:middle line:84%
And we typically tend to
choose batch sizes of 32, 64,

00:08:14.660 --> 00:08:16.420 align:middle line:84%
and things like that
because it actually

00:08:16.420 --> 00:08:18.960 align:middle line:84%
aligns very well with the
nature of the parallel hardware

00:08:18.960 --> 00:08:20.740 align:middle line:90%
we're going to use.

00:08:20.740 --> 00:08:24.280 align:middle line:90%
And so here, 32 and so on.

00:08:24.280 --> 00:08:26.140 align:middle line:90%
So divide 194 by 32.

00:08:26.140 --> 00:08:27.440 align:middle line:90%
You get 6-point-something.

00:08:27.440 --> 00:08:30.120 align:middle line:90%
You round it up to 7.

00:08:30.120 --> 00:08:32.700 align:middle line:84%
And so what that means is that
the first six batches will

00:08:32.700 --> 00:08:34.700 align:middle line:90%
have 32 samples each.

00:08:34.700 --> 00:08:37.799 align:middle line:84%
And then the final batch
has only two samples left.

00:08:37.799 --> 00:08:38.679 align:middle line:90%
And that's OK.

00:08:38.679 --> 00:08:41.622 align:middle line:84%
It can be a nice, little,
small batch at the end.

00:08:41.622 --> 00:08:43.539 align:middle line:84%
There's nothing that
says that every batch has

00:08:43.539 --> 00:08:44.372 align:middle line:90%
to be the same size.

00:08:44.372 --> 00:08:46.980 align:middle line:90%


00:08:46.980 --> 00:08:47.700 align:middle line:90%
That's it.

00:08:47.700 --> 00:08:48.320 align:middle line:90%
Epochs.

00:08:48.320 --> 00:08:48.970 align:middle line:90%
Batches.

00:08:48.970 --> 00:08:53.110 align:middle line:90%
[AUDIO OUT]

00:08:53.110 --> 00:08:54.390 align:middle line:90%
AUDIENCE: And are you--

00:08:54.390 --> 00:08:57.890 align:middle line:84%
for each batch, you run
through the whole network,

00:08:57.890 --> 00:08:58.630 align:middle line:90%
all the layers?

00:08:58.630 --> 00:09:00.670 align:middle line:90%
Or each layer is one batch?

00:09:00.670 --> 00:09:01.630 align:middle line:90%
RAMA RAMAKRISHNAN: No.

00:09:01.630 --> 00:09:03.970 align:middle line:84%
For a batch, you run it
through the entire network.

00:09:03.970 --> 00:09:06.630 align:middle line:84%
So the way to think about
it is that you take a batch.

00:09:06.630 --> 00:09:10.910 align:middle line:84%
Just momentarily, you assume
that's all the data you have.

00:09:10.910 --> 00:09:13.510 align:middle line:84%
Just run it through the network,
because unless you run it

00:09:13.510 --> 00:09:15.130 align:middle line:84%
through every layer
of the network,

00:09:15.130 --> 00:09:17.230 align:middle line:90%
you can't get a prediction.

00:09:17.230 --> 00:09:19.670 align:middle line:84%
Unless you get a prediction,
you can't calculate the loss.

00:09:19.670 --> 00:09:20.730 align:middle line:84%
And unless you
calculate the loss,

00:09:20.730 --> 00:09:22.105 align:middle line:84%
you can't calculate
the gradient.

00:09:22.105 --> 00:09:24.808 align:middle line:84%
Unless you calculate gradient,
you can't update the weights.

00:09:24.808 --> 00:09:25.850 align:middle line:90%
AUDIENCE: The last thing.

00:09:25.850 --> 00:09:27.690 align:middle line:84%
But if you're
using all the data,

00:09:27.690 --> 00:09:29.910 align:middle line:84%
just doing the gradient
descent, then you just

00:09:29.910 --> 00:09:31.885 align:middle line:84%
go through the
network once, right?

00:09:31.885 --> 00:09:33.010 align:middle line:90%
RAMA RAMAKRISHNAN: Exactly.

00:09:33.010 --> 00:09:34.630 align:middle line:84%
So in gradient
descent, one epoch

00:09:34.630 --> 00:09:37.710 align:middle line:84%
is one pass and
one weight update.

00:09:37.710 --> 00:09:40.055 align:middle line:84%
And in stochastic
gradient descent,

00:09:40.055 --> 00:09:41.430 align:middle line:84%
the number of
updates you make is

00:09:41.430 --> 00:09:43.150 align:middle line:84%
equal to the number
of batches you have,

00:09:43.150 --> 00:09:46.670 align:middle line:84%
which ends up being the
training set divided

00:09:46.670 --> 00:09:50.285 align:middle line:90%
by the batch size rounded up.

00:09:50.285 --> 00:09:51.660 align:middle line:84%
AUDIENCE: Sorry,
just to confirm.

00:09:51.660 --> 00:09:53.440 align:middle line:84%
So initially, when
you introduced

00:09:53.440 --> 00:09:55.440 align:middle line:84%
the concept of batches,
the whole purpose

00:09:55.440 --> 00:09:57.160 align:middle line:84%
was not to run
through all the data

00:09:57.160 --> 00:10:00.140 align:middle line:84%
and be able to do some
prediction from a subset.

00:10:00.140 --> 00:10:02.640 align:middle line:84%
So now, the advantage
is that after batch 1,

00:10:02.640 --> 00:10:05.560 align:middle line:84%
we are using more accurate
coefficient to run

00:10:05.560 --> 00:10:06.980 align:middle line:90%
through batch 2 and so on?

00:10:06.980 --> 00:10:08.560 align:middle line:84%
That's really the
advantage of it?

00:10:08.560 --> 00:10:08.920 align:middle line:90%
RAMA RAMAKRISHNAN: Exactly.

00:10:08.920 --> 00:10:10.200 align:middle line:84%
AUDIENCE: Or is there
something else here?

00:10:10.200 --> 00:10:11.220 align:middle line:84%
RAMA RAMAKRISHNAN:
Perfectly said.

00:10:11.220 --> 00:10:12.620 align:middle line:90%
That's exactly the advantage.

00:10:12.620 --> 00:10:14.360 align:middle line:84%
So we take a small
amount of data,

00:10:14.360 --> 00:10:17.060 align:middle line:84%
and we say, hey, we know
this is not all the data,

00:10:17.060 --> 00:10:18.617 align:middle line:84%
it's just a small
subset of the data.

00:10:18.617 --> 00:10:20.700 align:middle line:84%
So therefore, it's not
going to be super accurate.

00:10:20.700 --> 00:10:21.908 align:middle line:90%
It's going to be approximate.

00:10:21.908 --> 00:10:23.280 align:middle line:90%
But it's OK.

00:10:23.280 --> 00:10:26.800 align:middle line:84%
So we still tend to move
in the right direction.

00:10:26.800 --> 00:10:29.057 align:middle line:84%
So instead of waiting for
the whole thing to get done

00:10:29.057 --> 00:10:30.640 align:middle line:84%
and then updating
it, we're just going

00:10:30.640 --> 00:10:33.600 align:middle line:90%
to update it as we go along.

00:10:33.600 --> 00:10:34.400 align:middle line:90%
All right.

00:10:34.400 --> 00:10:35.740 align:middle line:90%
Yes.

00:10:35.740 --> 00:10:37.060 align:middle line:90%
AUDIENCE: Another question.

00:10:37.060 --> 00:10:40.800 align:middle line:84%
Is it that doing this
process, SGD, will

00:10:40.800 --> 00:10:42.680 align:middle line:84%
render us a more
better solution,

00:10:42.680 --> 00:10:44.940 align:middle line:84%
or it requires
less compute power?

00:10:44.940 --> 00:10:45.940 align:middle line:90%
RAMA RAMAKRISHNAN: Both.

00:10:45.940 --> 00:10:46.420 align:middle line:90%
AUDIENCE: Both.

00:10:46.420 --> 00:10:47.420 align:middle line:90%
RAMA RAMAKRISHNAN: Both.

00:10:47.420 --> 00:10:49.780 align:middle line:84%
And the reasons for both
are in the previous lecture.

00:10:49.780 --> 00:10:50.347 align:middle line:90%
Yeah.

00:10:50.347 --> 00:10:52.180 align:middle line:84%
And I'm saying that
instead of repeating it,

00:10:52.180 --> 00:10:54.280 align:middle line:84%
just because I'm very
pressed for time today.

00:10:54.280 --> 00:10:55.540 align:middle line:90%
That's why.

00:10:55.540 --> 00:10:56.040 align:middle line:90%
All right.

00:10:56.040 --> 00:10:56.620 align:middle line:90%
Cool.

00:10:56.620 --> 00:10:58.740 align:middle line:90%
So that's what we have.

00:10:58.740 --> 00:11:01.146 align:middle line:90%
Are we good?

00:11:01.146 --> 00:11:01.646 align:middle line:90%
OK.

00:11:01.646 --> 00:11:04.580 align:middle line:84%
So now we come to the last
step before we actually

00:11:04.580 --> 00:11:07.740 align:middle line:84%
fire up the Colab, which is
overfitting and regularization.

00:11:07.740 --> 00:11:11.260 align:middle line:84%
So if you remember from your
machine-learning background,

00:11:11.260 --> 00:11:13.390 align:middle line:84%
when your model gets
more and more complex--

00:11:13.390 --> 00:11:17.992 align:middle line:90%


00:11:17.992 --> 00:11:19.940 align:middle line:84%
use a simple model,
then use a more complex

00:11:19.940 --> 00:11:21.540 align:middle line:90%
model, and so on and so forth--

00:11:21.540 --> 00:11:25.002 align:middle line:84%
what happens to the error on
the training data, typically?

00:11:25.002 --> 00:11:26.960 align:middle line:84%
What happens to the error
on the training data?

00:11:26.960 --> 00:11:28.480 align:middle line:84%
So let's say you have a
simple regression model.

00:11:28.480 --> 00:11:29.440 align:middle line:90%
You get some error.

00:11:29.440 --> 00:11:30.500 align:middle line:84%
And then you have
a regression model

00:11:30.500 --> 00:11:32.560 align:middle line:84%
in which you use all kinds
of interaction terms,

00:11:32.560 --> 00:11:33.980 align:middle line:84%
you use logarithms,
and this and that,

00:11:33.980 --> 00:11:35.240 align:middle line:90%
and make it super complicated.

00:11:35.240 --> 00:11:36.100 align:middle line:84%
What do you think
is going to happen

00:11:36.100 --> 00:11:37.640 align:middle line:84%
to the error on
the training data?

00:11:37.640 --> 00:11:38.377 align:middle line:90%
AUDIENCE: Reduce.

00:11:38.377 --> 00:11:39.460 align:middle line:90%
RAMA RAMAKRISHNAN: Reduce.

00:11:39.460 --> 00:11:42.810 align:middle line:84%
Basically, it's going to go down
as the model gets more complex.

00:11:42.810 --> 00:11:43.992 align:middle line:90%
Correct?

00:11:43.992 --> 00:11:45.530 align:middle line:84%
Now, of course,
comes the punchline,

00:11:45.530 --> 00:11:47.030 align:middle line:84%
which is, what do
you think is going

00:11:47.030 --> 00:11:49.270 align:middle line:90%
to happen to the training data?

00:11:49.270 --> 00:11:50.370 align:middle line:90%
I showed you the answer.

00:11:50.370 --> 00:11:52.933 align:middle line:90%


00:11:52.933 --> 00:11:54.350 align:middle line:84%
Basically, what's
going to happen,

00:11:54.350 --> 00:11:56.010 align:middle line:84%
typically, at
least conceptually,

00:11:56.010 --> 00:11:57.270 align:middle line:84%
is that it's going to
get better and better.

00:11:57.270 --> 00:11:59.030 align:middle line:84%
At some point, it's
going to bottom-out,

00:11:59.030 --> 00:12:00.850 align:middle line:84%
and it's going to
start climbing again.

00:12:00.850 --> 00:12:03.790 align:middle line:84%
And so we typically
refer to this phenomenon

00:12:03.790 --> 00:12:05.430 align:middle line:84%
here, when it starts
to climb again,

00:12:05.430 --> 00:12:08.190 align:middle line:84%
as overfitting because
the model is essentially

00:12:08.190 --> 00:12:11.190 align:middle line:84%
fitting to the idiosyncrasies
of the training data

00:12:11.190 --> 00:12:13.790 align:middle line:84%
as opposed to
generalizing patterns.

00:12:13.790 --> 00:12:15.710 align:middle line:84%
And then, in this
thing, we call it

00:12:15.710 --> 00:12:17.360 align:middle line:84%
underfitting because
still there's

00:12:17.360 --> 00:12:18.610 align:middle line:90%
a lot of potential to improve.

00:12:18.610 --> 00:12:22.390 align:middle line:84%
And we really are hoping to find
the sweet spot in the middle.

00:12:22.390 --> 00:12:25.470 align:middle line:84%
That's the basic idea of
overfitting, underfitting.

00:12:25.470 --> 00:12:28.910 align:middle line:84%
And the way we-- and to relate
this to neural networks,

00:12:28.910 --> 00:12:31.550 align:middle line:84%
as you see, as you've
learned so far,

00:12:31.550 --> 00:12:33.990 align:middle line:84%
you have to learn smart
representations of the input

00:12:33.990 --> 00:12:34.690 align:middle line:90%
data.

00:12:34.690 --> 00:12:37.550 align:middle line:84%
And to do that, I have
argued that you need to have

00:12:37.550 --> 00:12:39.130 align:middle line:90%
lots of layers in your network.

00:12:39.130 --> 00:12:41.550 align:middle line:84%
The more layers you have,
the better things get.

00:12:41.550 --> 00:12:45.500 align:middle line:84%
GPT-3, for example, has
96 layers, if I recall.

00:12:45.500 --> 00:12:46.700 align:middle line:90%
More layers the better.

00:12:46.700 --> 00:12:49.160 align:middle line:84%
But more layers means
more parameters.

00:12:49.160 --> 00:12:51.760 align:middle line:84%
More parameters means more
complexity to the model,

00:12:51.760 --> 00:12:55.240 align:middle line:84%
and therefore more
chance of overfitting.

00:12:55.240 --> 00:12:58.400 align:middle line:84%
So it's really important
in neural networks

00:12:58.400 --> 00:13:00.520 align:middle line:84%
that we think about
regularization.

00:13:00.520 --> 00:13:02.200 align:middle line:84%
And regularization,
you will recall

00:13:02.200 --> 00:13:04.020 align:middle line:84%
from your machine-learning
background,

00:13:04.020 --> 00:13:06.200 align:middle line:84%
is the way we handle
the risk of overfitting

00:13:06.200 --> 00:13:09.240 align:middle line:84%
and try to find models
that fit just right.

00:13:09.240 --> 00:13:12.240 align:middle line:84%
And so several
regularization methods

00:13:12.240 --> 00:13:13.960 align:middle line:84%
have been developed
over the years.

00:13:13.960 --> 00:13:16.100 align:middle line:84%
And we are going to
use only two of them.

00:13:16.100 --> 00:13:18.680 align:middle line:84%
The first one is
called early stopping.

00:13:18.680 --> 00:13:21.480 align:middle line:84%
And this has been
famously referred to

00:13:21.480 --> 00:13:23.960 align:middle line:84%
by Geoff Hinton, who
is one of the pioneers,

00:13:23.960 --> 00:13:25.340 align:middle line:84%
or as he's more
colorfully known,

00:13:25.340 --> 00:13:28.360 align:middle line:84%
one of the godfathers
of deep learning--

00:13:28.360 --> 00:13:30.160 align:middle line:84%
he also won the Turing
a few years ago--

00:13:30.160 --> 00:13:32.940 align:middle line:90%
as a "beautiful free lunch."

00:13:32.940 --> 00:13:34.220 align:middle line:90%
That's what he calls it.

00:13:34.220 --> 00:13:35.720 align:middle line:90%
So the idea is very simple.

00:13:35.720 --> 00:13:37.337 align:middle line:90%
We take a validation set.

00:13:37.337 --> 00:13:38.920 align:middle line:84%
We take the training
data, we split it

00:13:38.920 --> 00:13:40.740 align:middle line:84%
into a training and
a validation set,

00:13:40.740 --> 00:13:43.260 align:middle line:84%
and then we just keep
doing gradient descent.

00:13:43.260 --> 00:13:44.607 align:middle line:90%
Bup, bup, bup, bup, bup, bup.

00:13:44.607 --> 00:13:47.190 align:middle line:84%
The training will hopefully keep
on getting better and better.

00:13:47.190 --> 00:13:49.130 align:middle line:90%
Lower and lower error.

00:13:49.130 --> 00:13:50.730 align:middle line:84%
And then we just
keep track of what's

00:13:50.730 --> 00:13:52.428 align:middle line:90%
going on in the validation set.

00:13:52.428 --> 00:13:53.970 align:middle line:84%
And then, at some
point, if it starts

00:13:53.970 --> 00:13:56.510 align:middle line:84%
to flatten out and
start to climb,

00:13:56.510 --> 00:13:59.833 align:middle line:84%
we just say, OK, that's
when we stop training.

00:13:59.833 --> 00:14:02.250 align:middle line:84%
And what we're going to do in
the Colab is actually run it

00:14:02.250 --> 00:14:03.950 align:middle line:84%
through the whole thing,
see where it flattens out,

00:14:03.950 --> 00:14:05.650 align:middle line:84%
and then we say, OK, that's
where we should stop.

00:14:05.650 --> 00:14:07.530 align:middle line:84%
But of course, you don't want
to go all the way to the end

00:14:07.530 --> 00:14:10.410 align:middle line:84%
and then go back and say, well,
I want to stop at the 10th

00:14:10.410 --> 00:14:11.370 align:middle line:90%
epoch.

00:14:11.370 --> 00:14:13.290 align:middle line:84%
And there are ways you
can use Keras to be

00:14:13.290 --> 00:14:14.670 align:middle line:90%
very efficient about this.

00:14:14.670 --> 00:14:17.030 align:middle line:84%
But the fundamental idea is
you take the training data,

00:14:17.030 --> 00:14:19.230 align:middle line:84%
split it into training
and validation,

00:14:19.230 --> 00:14:21.530 align:middle line:84%
and just track what's going
on in the validation set

00:14:21.530 --> 00:14:23.890 align:middle line:84%
to see whether this kind
of bottoming out happens.

00:14:23.890 --> 00:14:26.650 align:middle line:84%
So this is called
early stopping.

00:14:26.650 --> 00:14:28.850 align:middle line:84%
And the other way
we're going to do--

00:14:28.850 --> 00:14:30.850 align:middle line:90%
this is called early stopping.

00:14:30.850 --> 00:14:32.530 align:middle line:90%
We're looking for this part.

00:14:32.530 --> 00:14:34.650 align:middle line:84%
The other thing
is called dropout.

00:14:34.650 --> 00:14:39.963 align:middle line:84%
And I'm going to come back to
dropout on Wednesday's lecture

00:14:39.963 --> 00:14:42.130 align:middle line:84%
because that's the first
time we're going to use it.

00:14:42.130 --> 00:14:43.750 align:middle line:84%
And so I'll come back
to dropout and tell you

00:14:43.750 --> 00:14:44.650 align:middle line:90%
exactly how it works.

00:14:44.650 --> 00:14:46.410 align:middle line:84%
It's a very, very
clever strategy,

00:14:46.410 --> 00:14:47.770 align:middle line:90%
but we will not use it today.

00:14:47.770 --> 00:14:48.990 align:middle line:90%
We'll use it on Wednesday.

00:14:48.990 --> 00:14:52.090 align:middle line:90%
So in summary, what do we do?

00:14:52.090 --> 00:14:53.450 align:middle line:90%
We get the data ready.

00:14:53.450 --> 00:14:54.898 align:middle line:90%
We design the network--

00:14:54.898 --> 00:14:56.690 align:middle line:84%
number of hidden layers,
number of neurons,

00:14:56.690 --> 00:14:57.770 align:middle line:90%
and so on and so forth.

00:14:57.770 --> 00:14:59.410 align:middle line:90%
We pick the right output layer.

00:14:59.410 --> 00:15:01.590 align:middle line:90%
We pick the right loss function.

00:15:01.590 --> 00:15:03.730 align:middle line:90%
We choose an optimizer.

00:15:03.730 --> 00:15:05.310 align:middle line:84%
As I mentioned
earlier, SGD comes

00:15:05.310 --> 00:15:08.390 align:middle line:84%
in lots of flavors, lots
of variations on the theme.

00:15:08.390 --> 00:15:12.650 align:middle line:84%
And empirically, much like
for hidden-layer neurons

00:15:12.650 --> 00:15:15.950 align:middle line:84%
we tend to use ReLU as
the activation function,

00:15:15.950 --> 00:15:17.710 align:middle line:84%
for optimization, we
tend to use a flavor

00:15:17.710 --> 00:15:20.830 align:middle line:84%
of SGD called Adam
as the default

00:15:20.830 --> 00:15:22.050 align:middle line:90%
because it's really good.

00:15:22.050 --> 00:15:24.270 align:middle line:84%
So we'll use Adam,
as you will see.

00:15:24.270 --> 00:15:27.810 align:middle line:84%
We typically use either
early stopping or dropout.

00:15:27.810 --> 00:15:30.830 align:middle line:84%
And then we just fire it
up and start training Keras

00:15:30.830 --> 00:15:31.830 align:middle line:90%
in TensorFlow.

00:15:31.830 --> 00:15:32.330 align:middle line:90%
All right.

00:15:32.330 --> 00:15:33.810 align:middle line:90%
So that is the training loop.

00:15:33.810 --> 00:15:35.560 align:middle line:84%
Now I'm going to switch
gears and give you

00:15:35.560 --> 00:15:37.390 align:middle line:84%
a quick intro to
Teras and Kensor--

00:15:37.390 --> 00:15:39.480 align:middle line:90%
Teras and KensorFlow.

00:15:39.480 --> 00:15:42.020 align:middle line:84%
Keras and Tensor-- no,
TensorFlow and Keras.

00:15:42.020 --> 00:15:43.080 align:middle line:90%
Thank you.

00:15:43.080 --> 00:15:44.955 align:middle line:84%
And then we'll actually
fire up the Colab.

00:15:44.955 --> 00:15:46.330 align:middle line:84%
So first of all,
what's a tensor?

00:15:46.330 --> 00:15:49.815 align:middle line:90%


00:15:49.815 --> 00:15:50.440 align:middle line:90%
AUDIENCE: Yeah.

00:15:50.440 --> 00:15:53.800 align:middle line:84%
Just a quick question
on the previous thing.

00:15:53.800 --> 00:15:55.600 align:middle line:84%
If you're looking
at the validation

00:15:55.600 --> 00:15:58.360 align:middle line:84%
sets to avoid overfitting,
but aren't you

00:15:58.360 --> 00:16:01.840 align:middle line:84%
actually overfitting
because you're

00:16:01.840 --> 00:16:04.880 align:middle line:84%
kind of using the validation
set as a training set?

00:16:04.880 --> 00:16:05.530 align:middle line:90%
Or not.

00:16:05.530 --> 00:16:06.780 align:middle line:90%
RAMA RAMAKRISHNAN: No, no, no.

00:16:06.780 --> 00:16:10.200 align:middle line:84%
The validation set is never
used to calculate any gradients.

00:16:10.200 --> 00:16:14.360 align:middle line:84%
It's only used to calculate
accuracy and loss.

00:16:14.360 --> 00:16:14.860 align:middle line:90%
Yeah.

00:16:14.860 --> 00:16:17.680 align:middle line:84%
It's kept aside and only
used for evaluation, not

00:16:17.680 --> 00:16:19.320 align:middle line:90%
for training.

00:16:19.320 --> 00:16:22.360 align:middle line:90%
That's what keeps you honest.

00:16:22.360 --> 00:16:23.160 align:middle line:90%
All right.

00:16:23.160 --> 00:16:25.180 align:middle line:84%
And this will become clear when
we actually go into the Colab.

00:16:25.180 --> 00:16:25.972 align:middle line:90%
So what's a tensor?

00:16:25.972 --> 00:16:28.600 align:middle line:90%


00:16:28.600 --> 00:16:30.560 align:middle line:90%
All right.

00:16:30.560 --> 00:16:33.240 align:middle line:90%
OK.

00:16:33.240 --> 00:16:35.013 align:middle line:84%
AUDIENCE: Tensor is
the input data, which

00:16:35.013 --> 00:16:36.180 align:middle line:90%
you're giving to the system.

00:16:36.180 --> 00:16:39.190 align:middle line:84%
It could be in various formats
like, for instance, an image.

00:16:39.190 --> 00:16:41.870 align:middle line:84%
It could be-- like we
call it a 4D tensor.

00:16:41.870 --> 00:16:44.850 align:middle line:84%
If it's time series
data, it's 3D.

00:16:44.850 --> 00:16:47.310 align:middle line:84%
And typically, if we
just send numbers in,

00:16:47.310 --> 00:16:51.610 align:middle line:84%
it becomes a vector,
which would coincide.

00:16:51.610 --> 00:16:56.050 align:middle line:84%
It gives the value
of the variable

00:16:56.050 --> 00:16:58.710 align:middle line:84%
as well as-- the values of the
variables associated to it,

00:16:58.710 --> 00:17:06.102 align:middle line:84%
as well as the information
you want [INAUDIBLE].

00:17:06.102 --> 00:17:08.310 align:middle line:84%
RAMA RAMAKRISHNAN: You're
kind of on the right track,

00:17:08.310 --> 00:17:09.410 align:middle line:90%
but not entirely.

00:17:09.410 --> 00:17:11.490 align:middle line:84%
It's actually a simpler
concept than that.

00:17:11.490 --> 00:17:13.268 align:middle line:90%
So--

00:17:13.268 --> 00:17:14.810 align:middle line:84%
AUDIENCE: A matrix,
but generalizable

00:17:14.810 --> 00:17:16.165 align:middle line:90%
with higher dimensions.

00:17:16.165 --> 00:17:17.290 align:middle line:90%
RAMA RAMAKRISHNAN: Correct.

00:17:17.290 --> 00:17:19.970 align:middle line:84%
That's also actually
correct, but incomplete.

00:17:19.970 --> 00:17:23.609 align:middle line:84%
The reason is because it can
be simpler than a matrix.

00:17:23.609 --> 00:17:24.982 align:middle line:90%
It's not matrix or higher.

00:17:24.982 --> 00:17:26.190 align:middle line:90%
It actually could be simpler.

00:17:26.190 --> 00:17:30.090 align:middle line:84%
In fact, you take a number,
it's actually a tensor.

00:17:30.090 --> 00:17:30.630 align:middle line:90%
All right.

00:17:30.630 --> 00:17:33.130 align:middle line:84%
The simplest case of
a tensor is a number.

00:17:33.130 --> 00:17:36.340 align:middle line:84%
The next simpler case is
a vector, which is a list.

00:17:36.340 --> 00:17:40.780 align:middle line:90%
The next higher case is a table.

00:17:40.780 --> 00:17:42.880 align:middle line:90%
So these are all tensors.

00:17:42.880 --> 00:17:45.700 align:middle line:84%
So tensors, basically,
are a generalization

00:17:45.700 --> 00:17:49.940 align:middle line:84%
of the notion of both a
number, a vector, and a table

00:17:49.940 --> 00:17:53.060 align:middle line:90%
to higher dimensions.

00:17:53.060 --> 00:17:58.620 align:middle line:84%
So you can think of a tensor
as having what are called--

00:17:58.620 --> 00:18:01.700 align:middle line:84%
every tensor has
something called a rank.

00:18:01.700 --> 00:18:04.240 align:middle line:90%
So a number is just a number.

00:18:04.240 --> 00:18:06.160 align:middle line:84%
It doesn't have a
dimensionality to it.

00:18:06.160 --> 00:18:08.140 align:middle line:90%
So it has got rank 0.

00:18:08.140 --> 00:18:08.940 align:middle line:90%
OK?

00:18:08.940 --> 00:18:11.840 align:middle line:84%
While a vector, it's
a list of numbers.

00:18:11.840 --> 00:18:16.060 align:middle line:84%
You can write it down, top to
bottom, and it's one dimension.

00:18:16.060 --> 00:18:18.920 align:middle line:84%
So that dimension, that one
dimension, is called a rank.

00:18:18.920 --> 00:18:20.540 align:middle line:90%
So it's called rank 1.

00:18:20.540 --> 00:18:26.500 align:middle line:84%
A table is 2D, two dimensional,
so it's called rank 2.

00:18:26.500 --> 00:18:32.100 align:middle line:84%
And you can have a rank 3,
which is just a bunch of tables.

00:18:32.100 --> 00:18:34.440 align:middle line:84%
A bunch of tables
is a rank 3 tensor.

00:18:34.440 --> 00:18:37.160 align:middle line:90%
We also think of it as a cube.

00:18:37.160 --> 00:18:41.600 align:middle line:84%
So these things are very
useful because, obviously, we

00:18:41.600 --> 00:18:44.240 align:middle line:90%
are all familiar with vectors.

00:18:44.240 --> 00:18:46.080 align:middle line:84%
As you will see
very shortly later

00:18:46.080 --> 00:18:49.040 align:middle line:84%
in this class, black and
white, grayscale images,

00:18:49.040 --> 00:18:52.320 align:middle line:84%
are usually represented using
tables of numbers, like this.

00:18:52.320 --> 00:18:54.830 align:middle line:84%
Color images are represented
using three tables.

00:18:54.830 --> 00:18:57.360 align:middle line:90%


00:18:57.360 --> 00:19:00.360 align:middle line:84%
Can you think of what
might be representable

00:19:00.360 --> 00:19:05.720 align:middle line:84%
as a tensor of rank 4,
meaning, every element

00:19:05.720 --> 00:19:08.890 align:middle line:84%
of a tensor of rank 4 is
actually a color picture?

00:19:08.890 --> 00:19:11.880 align:middle line:90%


00:19:11.880 --> 00:19:12.620 align:middle line:90%
Shout it out.

00:19:12.620 --> 00:19:13.440 align:middle line:90%
AUDIENCE: Video.

00:19:13.440 --> 00:19:13.800 align:middle line:90%
RAMA RAMAKRISHNAN: Video.

00:19:13.800 --> 00:19:14.480 align:middle line:90%
Exactly.

00:19:14.480 --> 00:19:15.340 align:middle line:90%
What is the video?

00:19:15.340 --> 00:19:19.100 align:middle line:84%
A video is basically a
stream of color images.

00:19:19.100 --> 00:19:20.560 align:middle line:90%
A color video.

00:19:20.560 --> 00:19:25.240 align:middle line:84%
So each element of that
stream, the first dimension

00:19:25.240 --> 00:19:28.360 align:middle line:84%
of the tensor is
which frame it is,

00:19:28.360 --> 00:19:31.200 align:middle line:84%
and then everything else
is the actual frame.

00:19:31.200 --> 00:19:36.758 align:middle line:84%
So the way I think about
these tensors always is--

00:19:36.758 --> 00:19:39.930 align:middle line:84%
if you take a tensor, you
can just think of it as a--

00:19:39.930 --> 00:19:44.130 align:middle line:84%
you can think of a tensor as
being this array, which has

00:19:44.130 --> 00:19:46.690 align:middle line:90%
all these axes or dimensions.

00:19:46.690 --> 00:19:49.210 align:middle line:84%
This is the first one,
this is the second one,

00:19:49.210 --> 00:19:51.210 align:middle line:84%
this is the third
one, and so on.

00:19:51.210 --> 00:19:54.650 align:middle line:90%
This is a tensor of rank 4.

00:19:54.650 --> 00:19:57.450 align:middle line:90%
1, 2, 3, 4.

00:19:57.450 --> 00:20:00.350 align:middle line:90%
And so if you have a vector--

00:20:00.350 --> 00:20:02.730 align:middle line:84%
so you can imagine,
if it's just a vector,

00:20:02.730 --> 00:20:06.012 align:middle line:84%
you can imagine the vector
actually living like this.

00:20:06.012 --> 00:20:06.970 align:middle line:90%
Just a list of numbers.

00:20:06.970 --> 00:20:10.210 align:middle line:90%


00:20:10.210 --> 00:20:11.290 align:middle line:90%
But if it's just--

00:20:11.290 --> 00:20:16.270 align:middle line:84%
if it is just a 2D,
a rank 2 tensor,

00:20:16.270 --> 00:20:18.010 align:middle line:90%
which is just like that--

00:20:18.010 --> 00:20:21.250 align:middle line:90%
which is just like that.

00:20:21.250 --> 00:20:24.970 align:middle line:84%
So this thing becomes like
that, and that thing becomes

00:20:24.970 --> 00:20:26.090 align:middle line:90%
like that.

00:20:26.090 --> 00:20:28.690 align:middle line:84%
So for example, if
this is a 7 comma 3,

00:20:28.690 --> 00:20:35.167 align:middle line:84%
that means that there
are 7 rows and 3 columns.

00:20:35.167 --> 00:20:36.000 align:middle line:90%
So you get the idea.

00:20:36.000 --> 00:20:37.417 align:middle line:84%
So the way you
think about tensors

00:20:37.417 --> 00:20:39.460 align:middle line:84%
is always as if there's
open square bracket,

00:20:39.460 --> 00:20:41.460 align:middle line:84%
a bunch of things, a
closed square bracket.

00:20:41.460 --> 00:20:43.860 align:middle line:84%
And that's really what
a tensor object is.

00:20:43.860 --> 00:20:48.412 align:middle line:84%
So what that means is that
anytime you have a tensor--

00:20:48.412 --> 00:20:50.620 align:middle line:84%
anytime you have a tensor,
however complicated it is,

00:20:50.620 --> 00:20:54.460 align:middle line:84%
you can always create a more
complicated tensor by-- if you

00:20:54.460 --> 00:20:56.120 align:middle line:84%
want to take a list
of those tensors--

00:20:56.120 --> 00:20:59.300 align:middle line:84%
let's say that you
have a list of videos.

00:20:59.300 --> 00:21:03.580 align:middle line:84%
Each video is a rank 4 tensor,
which means a list of videos

00:21:03.580 --> 00:21:04.700 align:middle line:90%
is what rank?

00:21:04.700 --> 00:21:05.495 align:middle line:90%
AUDIENCE: 5.

00:21:05.495 --> 00:21:06.620 align:middle line:90%
RAMA RAMAKRISHNAN: Exactly.

00:21:06.620 --> 00:21:15.220 align:middle line:84%
So a tensor of rank, say, 10, is
just a list of rank 9 tensors.

00:21:15.220 --> 00:21:17.620 align:middle line:84%
So that is the most
important thing

00:21:17.620 --> 00:21:19.620 align:middle line:84%
you need to understand
about tensors.

00:21:19.620 --> 00:21:21.980 align:middle line:84%
So at any point in time,
if I give you a tensor,

00:21:21.980 --> 00:21:24.800 align:middle line:84%
you can just iterate through
the first dimension of it,

00:21:24.800 --> 00:21:26.880 align:middle line:90%
the first aspect of it.

00:21:26.880 --> 00:21:28.880 align:middle line:84%
And as you go through
each one of these values--

00:21:28.880 --> 00:21:35.550 align:middle line:84%
so for example, here,
yeah, that can do it.

00:21:35.550 --> 00:21:42.710 align:middle line:84%
So if you have this
tensor here, and if you

00:21:42.710 --> 00:21:46.910 align:middle line:84%
want to create a more
complicated tensor, no problem.

00:21:46.910 --> 00:21:48.690 align:middle line:84%
So you add another
dimension here.

00:21:48.690 --> 00:21:52.038 align:middle line:90%


00:21:52.038 --> 00:21:54.590 align:middle line:84%
And now it just becomes--
this dimension, let's say,

00:21:54.590 --> 00:21:55.360 align:middle line:90%
has 9 values.

00:21:55.360 --> 00:21:58.110 align:middle line:90%


00:21:58.110 --> 00:21:59.630 align:middle line:90%
0, 1, all the way to 9.

00:21:59.630 --> 00:22:01.008 align:middle line:90%
So you put 0 here.

00:22:01.008 --> 00:22:02.050 align:middle line:90%
And then what do you get?

00:22:02.050 --> 00:22:04.310 align:middle line:84%
This whole tensor
is a rank 4 tensor.

00:22:04.310 --> 00:22:05.930 align:middle line:90%
And you put a 1 here.

00:22:05.930 --> 00:22:07.210 align:middle line:90%
It's another rank 4 tensor.

00:22:07.210 --> 00:22:08.030 align:middle line:90%
You put a 2 here.

00:22:08.030 --> 00:22:09.510 align:middle line:90%
Another rank 4 tensor.

00:22:09.510 --> 00:22:12.950 align:middle line:84%
So every tensor, you
take the first element.

00:22:12.950 --> 00:22:14.790 align:middle line:84%
It's just a list,
but it's a list

00:22:14.790 --> 00:22:17.910 align:middle line:90%
of the next down-rank tensor.

00:22:17.910 --> 00:22:20.630 align:middle line:84%
Now, this tensor
concept is actually

00:22:20.630 --> 00:22:22.830 align:middle line:90%
something Einstein came up with.

00:22:22.830 --> 00:22:27.990 align:middle line:84%
And so it's simultaneously
kind of easy to understand

00:22:27.990 --> 00:22:29.355 align:middle line:90%
and also slippery.

00:22:29.355 --> 00:22:30.730 align:middle line:84%
So I would actually
encourage you

00:22:30.730 --> 00:22:32.897 align:middle line:84%
to read the book, which has
a really good discussion

00:22:32.897 --> 00:22:33.510 align:middle line:90%
of tensors.

00:22:33.510 --> 00:22:36.650 align:middle line:84%
And the more you practice
with it, the easier it'll get.

00:22:36.650 --> 00:22:39.090 align:middle line:84%
So if you feel you kind of
understood but not quite,

00:22:39.090 --> 00:22:40.850 align:middle line:90%
you're not alone.

00:22:40.850 --> 00:22:42.010 align:middle line:90%
It happens to all of us.

00:22:42.010 --> 00:22:44.690 align:middle line:84%
You have to pay the price
and go through the crucible.

00:22:44.690 --> 00:22:46.070 align:middle line:90%
OK?

00:22:46.070 --> 00:22:48.610 align:middle line:90%
All right.

00:22:48.610 --> 00:22:52.240 align:middle line:84%
So to come back to this,
that's what we have.

00:22:52.240 --> 00:22:54.777 align:middle line:90%


00:22:54.777 --> 00:22:56.610 align:middle line:84%
And we already talked
about a rank 4 tensor.

00:22:56.610 --> 00:22:57.770 align:middle line:90%
It's a video.

00:22:57.770 --> 00:23:00.470 align:middle line:84%
So 2.2, the text has
a lot more detail.

00:23:00.470 --> 00:23:02.690 align:middle line:90%
You should definitely read it.

00:23:02.690 --> 00:23:06.930 align:middle line:84%
So here, TensorFlow
is a library.

00:23:06.930 --> 00:23:09.370 align:middle line:84%
And as you can imagine,
neural networks, tensors

00:23:09.370 --> 00:23:11.130 align:middle line:84%
come in and go
through the network

00:23:11.130 --> 00:23:12.410 align:middle line:90%
and go out the other end.

00:23:12.410 --> 00:23:16.330 align:middle line:84%
And since tensors capture
everything-- numbers, lists,

00:23:16.330 --> 00:23:17.790 align:middle line:90%
tables, and so on and so forth--

00:23:17.790 --> 00:23:19.870 align:middle line:84%
it's just tensors flowing
from input to output.

00:23:19.870 --> 00:23:21.250 align:middle line:90%
Hence, it's called TensorFlow.

00:23:21.250 --> 00:23:22.732 align:middle line:84%
And it gives you
a couple of things

00:23:22.732 --> 00:23:24.190 align:middle line:84%
which are really,
really important,

00:23:24.190 --> 00:23:25.190 align:middle line:90%
which is why we use it.

00:23:25.190 --> 00:23:27.130 align:middle line:84%
The first one is that
it will automatically

00:23:27.130 --> 00:23:31.300 align:middle line:84%
calculate gradients for you of
arbitrarily complicated loss

00:23:31.300 --> 00:23:32.200 align:middle line:90%
functions.

00:23:32.200 --> 00:23:34.700 align:middle line:84%
You don't have to calculate the
gradient because calculating

00:23:34.700 --> 00:23:36.740 align:middle line:90%
the gradient is very painful.

00:23:36.740 --> 00:23:39.165 align:middle line:84%
It will automatically calculate
the gradients for you.

00:23:39.165 --> 00:23:40.040 align:middle line:90%
That's the best part.

00:23:40.040 --> 00:23:41.080 align:middle line:84%
You don't have to
use the chain rule.

00:23:41.080 --> 00:23:42.400 align:middle line:90%
You don't have to do anything.

00:23:42.400 --> 00:23:45.280 align:middle line:84%
The second thing it will do, it
gives you all these optimizers,

00:23:45.280 --> 00:23:47.245 align:middle line:84%
including SGD and
all its variations,

00:23:47.245 --> 00:23:49.620 align:middle line:84%
so you don't have to worry
about the optimization itself.

00:23:49.620 --> 00:23:51.660 align:middle line:84%
You can just pick and
choose what you want.

00:23:51.660 --> 00:23:54.347 align:middle line:84%
Third, if you have
a lot of servers,

00:23:54.347 --> 00:23:56.180 align:middle line:84%
it will actually take
the computational load

00:23:56.180 --> 00:23:57.972 align:middle line:84%
and distribute it across
all those servers.

00:23:57.972 --> 00:24:00.100 align:middle line:84%
People here with a
CS background know

00:24:00.100 --> 00:24:02.620 align:middle line:84%
that parallelizing
computation is actually

00:24:02.620 --> 00:24:04.583 align:middle line:90%
a very difficult problem.

00:24:04.583 --> 00:24:07.000 align:middle line:84%
There are things which are
called embarrassingly parallel.

00:24:07.000 --> 00:24:08.460 align:middle line:90%
Many things are not.

00:24:08.460 --> 00:24:10.480 align:middle line:84%
And it's actually quite
tricky to figure it out.

00:24:10.480 --> 00:24:11.813 align:middle line:90%
You don't have to figure it out.

00:24:11.813 --> 00:24:13.560 align:middle line:90%
TensorFlow will figure it out.

00:24:13.560 --> 00:24:15.460 align:middle line:84%
And then finally, I
talked about the fact

00:24:15.460 --> 00:24:18.060 align:middle line:84%
that there are these things
called GPUs, Graphics Processing

00:24:18.060 --> 00:24:20.420 align:middle line:84%
Units, which are
parallel hardware.

00:24:20.420 --> 00:24:24.460 align:middle line:84%
And so, even if you have just
one computer but it has GPUs,

00:24:24.460 --> 00:24:25.860 align:middle line:84%
there's a particular
way in which

00:24:25.860 --> 00:24:27.910 align:middle line:84%
you have to take
your computation

00:24:27.910 --> 00:24:29.750 align:middle line:84%
and organize it
to really exploit

00:24:29.750 --> 00:24:31.630 align:middle line:90%
the fact that you have a GPU.

00:24:31.630 --> 00:24:35.307 align:middle line:84%
And so TensorFlow will actually
do it for you, out of the box,

00:24:35.307 --> 00:24:35.890 align:middle line:90%
automatically.

00:24:35.890 --> 00:24:37.470 align:middle line:84%
You don't have to worry
about any of that stuff.

00:24:37.470 --> 00:24:39.650 align:middle line:84%
So those are all the
advantages of this thing.

00:24:39.650 --> 00:24:42.530 align:middle line:84%
By the way TPU is called
a Tensor Processing Unit.

00:24:42.530 --> 00:24:46.170 align:middle line:84%
It's something that-- you can
think of it as Google's GPU.

00:24:46.170 --> 00:24:48.630 align:middle line:84%
They came up with their
own variation on the theme.

00:24:48.630 --> 00:24:52.910 align:middle line:84%
Now, Keras sits on
top of TensorFlow.

00:24:52.910 --> 00:24:55.230 align:middle line:90%
This is the hardware you have.

00:24:55.230 --> 00:24:57.350 align:middle line:84%
TensorFlow sits on
top of the hardware.

00:24:57.350 --> 00:25:00.830 align:middle line:84%
Keras sits on top of
TensorFlow, and it basically

00:25:00.830 --> 00:25:02.890 align:middle line:84%
gives you a whole bunch
of convenience features.

00:25:02.890 --> 00:25:05.870 align:middle line:84%
So for example, it gives
you the notion of a layer.

00:25:05.870 --> 00:25:09.890 align:middle line:84%
We already saw, keras.dense
is a dense layer.

00:25:09.890 --> 00:25:11.370 align:middle line:84%
It gives you the
notion of a layer,

00:25:11.370 --> 00:25:13.370 align:middle line:84%
it gives you the notion
of activation functions,

00:25:13.370 --> 00:25:14.330 align:middle line:90%
and so on and so forth.

00:25:14.330 --> 00:25:17.190 align:middle line:84%
It gives you easy ways to
preprocess the data, easy ways

00:25:17.190 --> 00:25:19.470 align:middle line:84%
to train the model,
report on metrics,

00:25:19.470 --> 00:25:22.110 align:middle line:84%
calculate validation loss,
validation accuracy, training

00:25:22.110 --> 00:25:24.130 align:middle line:84%
loss, all the metrics
we care about.

00:25:24.130 --> 00:25:25.880 align:middle line:84%
And then it also gives
you a whole library

00:25:25.880 --> 00:25:28.240 align:middle line:84%
of pre-trained models that
you can just use and adapt

00:25:28.240 --> 00:25:30.000 align:middle line:90%
for your particular problem.

00:25:30.000 --> 00:25:32.320 align:middle line:84%
So it gives you a whole
bunch of conveniences.

00:25:32.320 --> 00:25:33.780 align:middle line:84%
And that's why
it's very popular.

00:25:33.780 --> 00:25:35.440 align:middle line:84%
And by the way, many
of you might also

00:25:35.440 --> 00:25:38.160 align:middle line:84%
be familiar with PyTorch,
which is a fantastic framework

00:25:38.160 --> 00:25:39.420 align:middle line:90%
as well for deep learning.

00:25:39.420 --> 00:25:42.520 align:middle line:84%
And the reason we chose to go
with TensorFlow for this course,

00:25:42.520 --> 00:25:44.600 align:middle line:84%
rather than PyTorch,
is because we

00:25:44.600 --> 00:25:48.320 align:middle line:84%
wanted to make the course
accessible to folks

00:25:48.320 --> 00:25:50.480 align:middle line:84%
who don't have a ton of
programming background

00:25:50.480 --> 00:25:53.840 align:middle line:84%
before coming to the class, and
PyTorch is a bit more demanding

00:25:53.840 --> 00:25:55.260 align:middle line:90%
from a CS perspective.

00:25:55.260 --> 00:25:57.807 align:middle line:84%
It requires more knowledge of
object-oriented programming,

00:25:57.807 --> 00:26:00.140 align:middle line:84%
which is why we decided to
go with TensorFlow and Keras,

00:26:00.140 --> 00:26:03.980 align:middle line:84%
because I think it's actually
as powerful, in many ways,

00:26:03.980 --> 00:26:07.060 align:middle line:84%
and it's a little
easier to get going.

00:26:07.060 --> 00:26:08.680 align:middle line:90%
So that's what we have here.

00:26:08.680 --> 00:26:10.120 align:middle line:84%
And one other thing
I will mention

00:26:10.120 --> 00:26:12.680 align:middle line:84%
is that there are three ways
in which you can use Keras.

00:26:12.680 --> 00:26:14.400 align:middle line:90%
There are three kinds of APIs--

00:26:14.400 --> 00:26:16.300 align:middle line:84%
sequential, functional,
subclassing.

00:26:16.300 --> 00:26:20.180 align:middle line:84%
And we'll almost exclusively
use the functional API.

00:26:20.180 --> 00:26:22.680 align:middle line:84%
And in fact, the model we built
for heart disease prediction

00:26:22.680 --> 00:26:24.130 align:middle line:90%
uses the functional API.

00:26:24.130 --> 00:26:27.410 align:middle line:84%
And so just read 7.2.2 of the
textbook to understand in detail

00:26:27.410 --> 00:26:28.630 align:middle line:90%
how the API works.

00:26:28.630 --> 00:26:30.650 align:middle line:84%
I find in my own work,
the functional API

00:26:30.650 --> 00:26:31.790 align:middle line:90%
is basically all I need.

00:26:31.790 --> 00:26:34.290 align:middle line:84%
I don't need to do anything
more complicated than that.

00:26:34.290 --> 00:26:37.010 align:middle line:84%
And as you will see as
you work on the homeworks

00:26:37.010 --> 00:26:40.410 align:middle line:84%
and on your project, it's
sort of a beautifully designed

00:26:40.410 --> 00:26:43.150 align:middle line:84%
LEGO-block environment
for doing these things.

00:26:43.150 --> 00:26:47.462 align:middle line:84%
And you can create very
complicated models very easily.

00:26:47.462 --> 00:26:50.110 align:middle line:84%
There is a whole bunch of
stuff here on these websites,

00:26:50.110 --> 00:26:51.450 align:middle line:90%
so check them out.

00:26:51.450 --> 00:26:54.870 align:middle line:90%
Lots of Colabs are available.

00:26:54.870 --> 00:26:57.370 align:middle line:84%
So now, if you go back to the
neural model for heart disease

00:26:57.370 --> 00:27:00.290 align:middle line:84%
prediction, this is what we
came up with in the last class.

00:27:00.290 --> 00:27:03.910 align:middle line:84%
We had an input layer, one
dense layer with 16 neurons,

00:27:03.910 --> 00:27:06.270 align:middle line:84%
ReLU neurons, an output
layer with a sigmoid,

00:27:06.270 --> 00:27:07.930 align:middle line:84%
and then, boom,
that was a model.

00:27:07.930 --> 00:27:10.250 align:middle line:90%
So let's train this model.

00:27:10.250 --> 00:27:13.690 align:middle line:84%
And so the training checklist is
that we have already done this.

00:27:13.690 --> 00:27:16.170 align:middle line:90%
Hidden layer of 16 neurons.

00:27:16.170 --> 00:27:16.848 align:middle line:90%
Sigmoid.

00:27:16.848 --> 00:27:18.890 align:middle line:84%
We need to use an appropriate
loss function based

00:27:18.890 --> 00:27:19.973 align:middle line:90%
on the type of the output.

00:27:19.973 --> 00:27:22.950 align:middle line:84%
What loss function
should we use?

00:27:22.950 --> 00:27:26.510 align:middle line:90%
What is the output here?

00:27:26.510 --> 00:27:28.670 align:middle line:84%
It's a binary
classification problem.

00:27:28.670 --> 00:27:30.450 align:middle line:84%
So what should the
loss function be?

00:27:30.450 --> 00:27:33.310 align:middle line:90%


00:27:33.310 --> 00:27:34.990 align:middle line:90%
I kind of heard it somewhere.

00:27:34.990 --> 00:27:35.930 align:middle line:90%
You can shout it out.

00:27:35.930 --> 00:27:37.550 align:middle line:90%
AUDIENCE: [INAUDIBLE]

00:27:37.550 --> 00:27:39.590 align:middle line:84%
RAMA RAMAKRISHNAN: The
output is a sigmoid.

00:27:39.590 --> 00:27:40.730 align:middle line:90%
The loss function.

00:27:40.730 --> 00:27:42.750 align:middle line:90%
AUDIENCE: Binary cross-entropy.

00:27:42.750 --> 00:27:45.250 align:middle line:84%
RAMA RAMAKRISHNAN:
Binary cross-entropy.

00:27:45.250 --> 00:27:48.290 align:middle line:84%
Remember, if you're predicting
a number, an arbitrary number,

00:27:48.290 --> 00:27:50.522 align:middle line:84%
you can use something
like mean squared error.

00:27:50.522 --> 00:27:52.230 align:middle line:84%
If you're predicting
a probability, which

00:27:52.230 --> 00:27:55.030 align:middle line:84%
has to be compared to
a 0, 1 output, which

00:27:55.030 --> 00:27:57.090 align:middle line:84%
is what binary
classification is all about,

00:27:57.090 --> 00:27:58.990 align:middle line:90%
we use binary cross-entropy.

00:27:58.990 --> 00:28:00.890 align:middle line:90%
So that's what we do here.

00:28:00.890 --> 00:28:03.750 align:middle line:90%
So we do binary cross-entropy.

00:28:03.750 --> 00:28:06.170 align:middle line:90%
And then we will go with Adam.

00:28:06.170 --> 00:28:07.910 align:middle line:84%
And then we will
use early stopping

00:28:07.910 --> 00:28:10.670 align:middle line:90%
to make sure we don't overfit.

00:28:10.670 --> 00:28:13.230 align:middle line:84%
I know this-- OK, I promise,
this is literally the last slide

00:28:13.230 --> 00:28:15.190 align:middle line:90%
before I go to the Colab.

00:28:15.190 --> 00:28:17.350 align:middle line:84%
I feel like one of
those used car salesmen.

00:28:17.350 --> 00:28:19.470 align:middle line:90%
Wait, there is more!

00:28:19.470 --> 00:28:20.920 align:middle line:90%
So anyway.

00:28:20.920 --> 00:28:24.640 align:middle line:84%
So don't worry if you don't
understand every detail of what

00:28:24.640 --> 00:28:25.760 align:middle line:90%
I'm going to go through.

00:28:25.760 --> 00:28:28.672 align:middle line:84%
I'm going to link to the Colab
as soon as the class is over.

00:28:28.672 --> 00:28:30.380 align:middle line:84%
But once you get your
hands on the Colab,

00:28:30.380 --> 00:28:32.920 align:middle line:84%
make sure you actually go
through every line in Colab.

00:28:32.920 --> 00:28:35.337 align:middle line:84%
What I typically do when I'm
trying to learn something new

00:28:35.337 --> 00:28:38.680 align:middle line:90%
is I'll actually cut and paste.

00:28:38.680 --> 00:28:39.483 align:middle line:90%
I won't do that.

00:28:39.483 --> 00:28:41.900 align:middle line:84%
I won't actually cut and paste
the code and run it myself.

00:28:41.900 --> 00:28:43.805 align:middle line:90%
I will retype the code.

00:28:43.805 --> 00:28:46.180 align:middle line:84%
If you retype the code as
opposed to cutting and pasting,

00:28:46.180 --> 00:28:48.120 align:middle line:84%
trust me, you will
learn a lot more.

00:28:48.120 --> 00:28:52.320 align:middle line:84%
So I strongly encourage
you to do it that way.

00:28:52.320 --> 00:28:55.392 align:middle line:84%
And so all the Colabs you're
going to publish in the class,

00:28:55.392 --> 00:28:57.600 align:middle line:84%
the first thing you should
do is you should just make

00:28:57.600 --> 00:29:00.120 align:middle line:90%
your own copy of the notebook.

00:29:00.120 --> 00:29:01.740 align:middle line:90%
Copy to Drive.

00:29:01.740 --> 00:29:03.880 align:middle line:84%
And then, if you're
using anything other

00:29:03.880 --> 00:29:07.160 align:middle line:84%
than today's Colab, anything
involving natural language

00:29:07.160 --> 00:29:09.520 align:middle line:84%
processing or vision, you
probably should use a GPU.

00:29:09.520 --> 00:29:11.040 align:middle line:90%
So just go into--

00:29:11.040 --> 00:29:14.840 align:middle line:84%
you go in here, choose
the runtime to be GPU,

00:29:14.840 --> 00:29:17.020 align:middle line:84%
and then you start your
notebook, and you're done.

00:29:17.020 --> 00:29:18.728 align:middle line:84%
And the second time
onwards, you can just

00:29:18.728 --> 00:29:20.032 align:middle line:90%
go directly to this step.

00:29:20.032 --> 00:29:21.490 align:middle line:84%
You don't have to
do all this stuff

00:29:21.490 --> 00:29:23.322 align:middle line:90%
for that particular notebook.

00:29:23.322 --> 00:29:25.530 align:middle line:84%
There are numerous tutorials,
like five-minute videos

00:29:25.530 --> 00:29:27.128 align:middle line:90%
and so on, on how to use Colab.

00:29:27.128 --> 00:29:27.670 align:middle line:90%
Just do that.

00:29:27.670 --> 00:29:30.210 align:middle line:84%
I'm not going to
spend time on it here.

00:29:30.210 --> 00:29:31.490 align:middle line:90%
All right.

00:29:31.490 --> 00:29:31.990 align:middle line:90%
OK.

00:29:31.990 --> 00:29:35.010 align:middle line:84%
So I just ran it
a few hours ago.

00:29:35.010 --> 00:29:36.510 align:middle line:84%
I'm not going to
run every cell now

00:29:36.510 --> 00:29:37.510 align:middle line:84%
because this is going
to take some time.

00:29:37.510 --> 00:29:39.468 align:middle line:84%
It's going to get in the
way of the class time.

00:29:39.468 --> 00:29:41.970 align:middle line:84%
But I'm going to just go
through it slowly and explain

00:29:41.970 --> 00:29:42.970 align:middle line:90%
what's going on.

00:29:42.970 --> 00:29:45.750 align:middle line:84%
So here, this is just an
introduction to the data set.

00:29:45.750 --> 00:29:48.730 align:middle line:84%
We already saw this
introduction last week.

00:29:48.730 --> 00:29:51.370 align:middle line:90%
We have whatever-- 303 patients.

00:29:51.370 --> 00:29:52.430 align:middle line:90%
Heart patients.

00:29:52.430 --> 00:29:55.850 align:middle line:84%
We have a whole bunch
of variables here--

00:29:55.850 --> 00:29:59.870 align:middle line:84%
age, demographics, and a whole
bunch of biomarker information.

00:29:59.870 --> 00:30:02.850 align:middle line:90%
And this is our target variable.

00:30:02.850 --> 00:30:03.770 align:middle line:90%
0 or 1.

00:30:03.770 --> 00:30:05.530 align:middle line:90%
Heart disease, yes or no.

00:30:05.530 --> 00:30:09.770 align:middle line:84%
And so, by the way, just some
technical preliminaries here,

00:30:09.770 --> 00:30:11.637 align:middle line:84%
basically every time
we load these things,

00:30:11.637 --> 00:30:13.470 align:middle line:84%
we're actually going
to load these packages.

00:30:13.470 --> 00:30:15.512 align:middle line:84%
So you can see here, these
are the two key things

00:30:15.512 --> 00:30:16.110 align:middle line:90%
we need to do.

00:30:16.110 --> 00:30:18.140 align:middle line:90%
We import TensorFlow first.

00:30:18.140 --> 00:30:20.280 align:middle line:84%
And then, from within
TensorFlow, we import Keras.

00:30:20.280 --> 00:30:24.085 align:middle line:84%
That's what these
two lines do here.

00:30:24.085 --> 00:30:26.460 align:middle line:84%
And folks who have done data
science and machine learning

00:30:26.460 --> 00:30:28.553 align:middle line:90%
a bit before, you'll know this.

00:30:28.553 --> 00:30:31.380 align:middle line:84%
We will actually load
the three packages

00:30:31.380 --> 00:30:33.780 align:middle line:84%
that were just most commonly
used right, which is

00:30:33.780 --> 00:30:36.660 align:middle line:90%
NumPy, Pandas, and Matplotlib.

00:30:36.660 --> 00:30:40.220 align:middle line:84%
NumPy, because it's very easy
for manipulating matrices

00:30:40.220 --> 00:30:42.020 align:middle line:90%
and arrays and tensors.

00:30:42.020 --> 00:30:45.800 align:middle line:84%
Pandas because, oftentimes, you
get some data from somewhere,

00:30:45.800 --> 00:30:48.130 align:middle line:84%
you need to massage it
and wrangle it to a point

00:30:48.130 --> 00:30:49.880 align:middle line:84%
where you can actually
feed it into Keras.

00:30:49.880 --> 00:30:51.120 align:middle line:90%
So you need pandas for that.

00:30:51.120 --> 00:30:55.173 align:middle line:84%
And Matplotlib because you just
want to plot these loss curves

00:30:55.173 --> 00:30:57.340 align:middle line:84%
and accuracy curves to see
whether early stopping is

00:30:57.340 --> 00:30:57.960 align:middle line:90%
needed.

00:30:57.960 --> 00:31:00.100 align:middle line:90%
So that's why we use it.

00:31:00.100 --> 00:31:01.350 align:middle line:90%
So we import all these things.

00:31:01.350 --> 00:31:03.600 align:middle line:84%
And then, I guess the other
thing you have to remember

00:31:03.600 --> 00:31:06.140 align:middle line:84%
is that when we are training
these deep-learning models,

00:31:06.140 --> 00:31:08.580 align:middle line:84%
there is randomness in
the process, which enters

00:31:08.580 --> 00:31:10.140 align:middle line:90%
in a few different places.

00:31:10.140 --> 00:31:13.340 align:middle line:84%
So clearly, the starting
values for these weights

00:31:13.340 --> 00:31:14.180 align:middle line:90%
are going to be--

00:31:14.180 --> 00:31:16.600 align:middle line:84%
the weights are going to
be randomly initialized.

00:31:16.600 --> 00:31:20.560 align:middle line:84%
And therefore, that's obviously
a source of randomness.

00:31:20.560 --> 00:31:22.183 align:middle line:84%
Now, we talked
about how you take--

00:31:22.183 --> 00:31:24.100 align:middle line:84%
when you're doing
stochastic gradient descent,

00:31:24.100 --> 00:31:27.440 align:middle line:84%
you take all the data, and then
you randomly choose batches

00:31:27.440 --> 00:31:30.500 align:middle line:84%
right from this data till we
finish a whole pass through it.

00:31:30.500 --> 00:31:33.100 align:middle line:84%
Well, that immediately
raised the question, well,

00:31:33.100 --> 00:31:34.780 align:middle line:84%
what do you mean
by randomly choose?

00:31:34.780 --> 00:31:37.080 align:middle line:84%
So typically, what we
do in practice is that--

00:31:37.080 --> 00:31:39.200 align:middle line:84%
and Keras will take care
of all this for you--

00:31:39.200 --> 00:31:42.340 align:middle line:84%
you basically take the data and
just shuffle it once, randomly.

00:31:42.340 --> 00:31:45.020 align:middle line:84%
And then you just go first
32, next 32, next 32, next 32,

00:31:45.020 --> 00:31:46.090 align:middle line:90%
like that.

00:31:46.090 --> 00:31:47.960 align:middle line:84%
But it is a source
of randomness.

00:31:47.960 --> 00:31:51.340 align:middle line:84%
And then when we split the data
into train, validation, testing,

00:31:51.340 --> 00:31:53.120 align:middle line:84%
and so on, particularly
if you want

00:31:53.120 --> 00:31:55.480 align:middle line:84%
to look for early
stopping and overfitting,

00:31:55.480 --> 00:31:57.540 align:middle line:84%
we need to again split
the data randomly.

00:31:57.540 --> 00:31:59.300 align:middle line:84%
And that's another
source of randomness.

00:31:59.300 --> 00:32:01.240 align:middle line:84%
And then when we
do dropout, which

00:32:01.240 --> 00:32:03.260 align:middle line:84%
we will talk about
on Wednesday-- again,

00:32:03.260 --> 00:32:06.260 align:middle line:84%
dropout has a little bit
of a random element to it,

00:32:06.260 --> 00:32:09.100 align:middle line:84%
and so that's another
source of randomness.

00:32:09.100 --> 00:32:12.140 align:middle line:84%
So all this means is that if
you're working with these models

00:32:12.140 --> 00:32:14.640 align:middle line:84%
and if you want to build a model
and you want to hand it off

00:32:14.640 --> 00:32:17.450 align:middle line:84%
to someone so that they
can reproduce your results,

00:32:17.450 --> 00:32:20.890 align:middle line:84%
well, you better make sure
that you make it easy for them

00:32:20.890 --> 00:32:22.110 align:middle line:90%
to replicate what you have.

00:32:22.110 --> 00:32:24.970 align:middle line:84%
And the way you do it is by
sending a random seed for all

00:32:24.970 --> 00:32:25.830 align:middle line:90%
these things.

00:32:25.830 --> 00:32:27.650 align:middle line:84%
And the way you
do it is by having

00:32:27.650 --> 00:32:30.730 align:middle line:84%
this little handy function
here, set_random_seed.

00:32:30.730 --> 00:32:34.810 align:middle line:84%
And of course, I use 42,
just like everybody should.

00:32:34.810 --> 00:32:36.890 align:middle line:90%
So that's that.

00:32:36.890 --> 00:32:39.090 align:middle line:84%
By the way, that's just
a pop culture reference

00:32:39.090 --> 00:32:40.840 align:middle line:84%
to this book called
The Hitchhiker's Guide

00:32:40.840 --> 00:32:42.057 align:middle line:90%
to the Galaxy.

00:32:42.057 --> 00:32:43.890 align:middle line:84%
So when I say something
about the number 42,

00:32:43.890 --> 00:32:45.370 align:middle line:90%
and you'll know what I mean.

00:32:45.370 --> 00:32:49.310 align:middle line:84%
So by the way, the question
inevitably comes at this point,

00:32:49.310 --> 00:32:51.250 align:middle line:84%
OK, if we do exactly
this, will we actually

00:32:51.250 --> 00:32:52.708 align:middle line:84%
get the exact same
numbers that you

00:32:52.708 --> 00:32:55.150 align:middle line:84%
have in your version
of the notebook?

00:32:55.150 --> 00:32:57.510 align:middle line:84%
And the answer is,
hopefully, most of the time,

00:32:57.510 --> 00:32:58.790 align:middle line:90%
but it's not guaranteed.

00:32:58.790 --> 00:33:01.070 align:middle line:84%
So this is called
bitwise reproducibility.

00:33:01.070 --> 00:33:03.770 align:middle line:84%
It's not guaranteed due
to certain hardware things

00:33:03.770 --> 00:33:05.950 align:middle line:84%
and device drivers
and stuff like that,

00:33:05.950 --> 00:33:08.090 align:middle line:84%
so we won't get into
all that stuff, which

00:33:08.090 --> 00:33:10.330 align:middle line:84%
is why, as you see
here, I have a bit

00:33:10.330 --> 00:33:13.050 align:middle line:90%
of a fingers-crossed thing.

00:33:13.050 --> 00:33:13.550 align:middle line:90%
All right.

00:33:13.550 --> 00:33:14.060 align:middle line:90%
Cool.

00:33:14.060 --> 00:33:15.740 align:middle line:90%
So that's what we have.

00:33:15.740 --> 00:33:18.757 align:middle line:84%
So as it turns out, Francois
Chollet, who wrote the book,

00:33:18.757 --> 00:33:21.340 align:middle line:84%
the textbook, he actually made
this data available in a Pandas

00:33:21.340 --> 00:33:22.180 align:middle line:90%
dataframe.

00:33:22.180 --> 00:33:26.020 align:middle line:84%
So we read the CSV file into
this dataframe right there.

00:33:26.020 --> 00:33:30.420 align:middle line:84%
And then it's 303
rows, 14 columns.

00:33:30.420 --> 00:33:33.860 align:middle line:84%
And you can see here, we'll take
a look at the first few rows.

00:33:33.860 --> 00:33:37.020 align:middle line:84%
And these are all the rows--
age, gender, cholesterol, blah,

00:33:37.020 --> 00:33:38.140 align:middle line:90%
blah, blah, blah, blah.

00:33:38.140 --> 00:33:40.980 align:middle line:84%
And then this is the target
variable right there.

00:33:40.980 --> 00:33:42.607 align:middle line:84%
And one of the first
things I always

00:33:42.607 --> 00:33:44.940 align:middle line:84%
do when I'm working with a
binary classification problem

00:33:44.940 --> 00:33:47.580 align:middle line:84%
is to quickly check whether the
positive and negative classes

00:33:47.580 --> 00:33:48.983 align:middle line:90%
are balanced or not.

00:33:48.983 --> 00:33:50.900 align:middle line:84%
And so what you can do
is you can just quickly

00:33:50.900 --> 00:33:53.880 align:middle line:84%
check to see what percent of
the data points is 0 versus 1.

00:33:53.880 --> 00:33:58.318 align:middle line:84%
And you can see here,
72.6% of the patients

00:33:58.318 --> 00:33:59.360 align:middle line:90%
don't have heart disease.

00:33:59.360 --> 00:34:01.100 align:middle line:90%
That's a good thing, of course.

00:34:01.100 --> 00:34:03.760 align:middle line:84%
And then 27.4 have
heart disease.

00:34:03.760 --> 00:34:06.040 align:middle line:84%
So it's not 50/50
or roughly 50/50.

00:34:06.040 --> 00:34:07.260 align:middle line:90%
It's a little thin.

00:34:07.260 --> 00:34:09.520 align:middle line:90%
So by the way, quick question.

00:34:09.520 --> 00:34:12.915 align:middle line:84%
What is a good baseline
model for this problem?

00:34:12.915 --> 00:34:15.290 align:middle line:84%
Suppose you couldn't use
anything, any complicated thing.

00:34:15.290 --> 00:34:18.126 align:middle line:90%
What's a good baseline model?

00:34:18.126 --> 00:34:21.886 align:middle line:90%
Let's go to this other side.

00:34:21.886 --> 00:34:22.469 align:middle line:90%
AUDIENCE: Yes.

00:34:22.469 --> 00:34:23.922 align:middle line:90%
Just predict zero every time.

00:34:23.922 --> 00:34:25.630 align:middle line:84%
RAMA RAMAKRISHNAN:
Why would you do that?

00:34:25.630 --> 00:34:28.185 align:middle line:84%
AUDIENCE: It would give
you a 72.6% accuracy.

00:34:28.185 --> 00:34:29.310 align:middle line:90%
RAMA RAMAKRISHNAN: Exactly.

00:34:29.310 --> 00:34:32.810 align:middle line:84%
Because 72.6% is
the higher class,

00:34:32.810 --> 00:34:35.370 align:middle line:84%
higher class of the higher
percentage, you just predict it.

00:34:35.370 --> 00:34:37.373 align:middle line:84%
You'll be right on those
72.6% of the cases,

00:34:37.373 --> 00:34:38.790 align:middle line:84%
you'll be wrong
on the rest, which

00:34:38.790 --> 00:34:43.750 align:middle line:84%
means that your accuracy of
this model is going to be 72.6%.

00:34:43.750 --> 00:34:48.330 align:middle line:84%
And so any fancy model we build,
it's got to do better than this.

00:34:48.330 --> 00:34:51.250 align:middle line:84%
Otherwise, it's not worth
its weight in layers.

00:34:51.250 --> 00:34:51.750 align:middle line:90%
All right.

00:34:51.750 --> 00:34:53.389 align:middle line:84%
So we'll come back
to this later.

00:34:53.389 --> 00:34:55.931 align:middle line:84%
So the first thing we want to
do is we want to preprocess it,

00:34:55.931 --> 00:34:58.190 align:middle line:84%
because this data set has
both categorical variables

00:34:58.190 --> 00:35:00.030 align:middle line:90%
and numeric variables.

00:35:00.030 --> 00:35:02.750 align:middle line:84%
And so it's usually
convenient just to group them

00:35:02.750 --> 00:35:04.250 align:middle line:90%
into two different groups.

00:35:04.250 --> 00:35:06.430 align:middle line:84%
So I have listed all the
categorical variables here

00:35:06.430 --> 00:35:08.190 align:middle line:90%
and the numerics here.

00:35:08.190 --> 00:35:10.500 align:middle line:84%
And then we have the
preprocessing here.

00:35:10.500 --> 00:35:12.250 align:middle line:84%
We have to take the
categorical variables,

00:35:12.250 --> 00:35:14.410 align:middle line:84%
and we have to
one-hot encode them.

00:35:14.410 --> 00:35:17.750 align:middle line:84%
And the reason is that unlike,
say, a decision tree model,

00:35:17.750 --> 00:35:22.170 align:middle line:84%
a neural network cannot handle
categorical inputs directly.

00:35:22.170 --> 00:35:24.190 align:middle line:84%
It can only handle
numeric inputs,

00:35:24.190 --> 00:35:26.050 align:middle line:84%
which means that we
have to numericalize

00:35:26.050 --> 00:35:27.910 align:middle line:84%
every categorical
thing that comes in.

00:35:27.910 --> 00:35:29.430 align:middle line:84%
And there are many
ways to do it,

00:35:29.430 --> 00:35:32.570 align:middle line:84%
but the standard way to
do it is one-hot encoding.

00:35:32.570 --> 00:35:35.230 align:middle line:84%
And for the numeric variables,
we need to normalize them.

00:35:35.230 --> 00:35:36.730 align:middle line:84%
And I'll come to
that in a second.

00:35:36.730 --> 00:35:40.008 align:middle line:84%
So Pandas has this
get_dummies function here.

00:35:40.008 --> 00:35:42.050 align:middle line:84%
And you can just run this
thing, and it will just

00:35:42.050 --> 00:35:43.550 align:middle line:90%
one-hot encode the whole thing.

00:35:43.550 --> 00:35:45.510 align:middle line:84%
So once you do that,
this is what you have.

00:35:45.510 --> 00:35:50.650 align:middle line:84%
So you can see here,
previously, let's say, thal

00:35:50.650 --> 00:35:52.650 align:middle line:84%
had three values-- fixed,
normal, reversible,

00:35:52.650 --> 00:35:53.510 align:middle line:90%
or something.

00:35:53.510 --> 00:35:56.490 align:middle line:84%
And then you go to the
one-hot encoded version.

00:35:56.490 --> 00:35:59.770 align:middle line:84%
And now we can see here,
thal_fixed, thal_normal,

00:35:59.770 --> 00:36:00.590 align:middle line:90%
thal_reversible.

00:36:00.590 --> 00:36:02.130 align:middle line:90%
That's three columns.

00:36:02.130 --> 00:36:04.610 align:middle line:84%
That's the one-hot
encoding in action.

00:36:04.610 --> 00:36:08.690 align:middle line:84%
Now, the other thing to
remember is that neural networks

00:36:08.690 --> 00:36:10.700 align:middle line:84%
work best when
the numeric inputs

00:36:10.700 --> 00:36:13.560 align:middle line:84%
you send them are all in
a relatively small range.

00:36:13.560 --> 00:36:16.300 align:middle line:84%
They shouldn't have a
wide range of variation.

00:36:16.300 --> 00:36:19.160 align:middle line:84%
And so the standard
practice is to standardize

00:36:19.160 --> 00:36:20.160 align:middle line:90%
the numerical variables.

00:36:20.160 --> 00:36:22.660 align:middle line:84%
By standardize, I mean typically
subtract the mean, divide

00:36:22.660 --> 00:36:24.580 align:middle line:90%
by the standard deviation.

00:36:24.580 --> 00:36:25.800 align:middle line:90%
We should do that.

00:36:25.800 --> 00:36:28.140 align:middle line:84%
But before we do so, we
should split the data

00:36:28.140 --> 00:36:30.740 align:middle line:84%
into a training
set and a test set.

00:36:30.740 --> 00:36:32.800 align:middle line:84%
And why do we want to
split into a test set?

00:36:32.800 --> 00:36:34.420 align:middle line:84%
Because at the very end,
once we have built the model

00:36:34.420 --> 00:36:36.337 align:middle line:84%
and done all the things
we want to do with it,

00:36:36.337 --> 00:36:38.380 align:middle line:84%
we finally want to
take out the test set

00:36:38.380 --> 00:36:43.140 align:middle line:84%
and evaluate it once so that
we get this true measure of how

00:36:43.140 --> 00:36:45.980 align:middle line:84%
it's going to perform in the
wild after you deploy it.

00:36:45.980 --> 00:36:50.620 align:middle line:84%
So you want to divide it, let's
say, 80% training and 20% test

00:36:50.620 --> 00:36:51.200 align:middle line:90%
set.

00:36:51.200 --> 00:36:53.860 align:middle line:84%
So the question is, why
should we do the splitting now

00:36:53.860 --> 00:36:55.900 align:middle line:90%
before we do the normalization?

00:36:55.900 --> 00:36:57.860 align:middle line:84%
Why can't we just
do the normalization

00:36:57.860 --> 00:36:59.040 align:middle line:90%
and then do splitting?

00:36:59.040 --> 00:37:04.020 align:middle line:90%


00:37:04.020 --> 00:37:06.340 align:middle line:90%
All right.

00:37:06.340 --> 00:37:09.430 align:middle line:84%
AUDIENCE: Because then
your validation set

00:37:09.430 --> 00:37:11.470 align:middle line:84%
is also somewhat
dependent on your test

00:37:11.470 --> 00:37:13.770 align:middle line:84%
set results, as well as
the mean of the test set.

00:37:13.770 --> 00:37:14.895 align:middle line:90%
RAMA RAMAKRISHNAN: Correct.

00:37:14.895 --> 00:37:17.790 align:middle line:84%
Because the test set
has now essentially

00:37:17.790 --> 00:37:21.710 align:middle line:84%
been influenced by
the training set.

00:37:21.710 --> 00:37:24.090 align:middle line:84%
The modeling process-- part
of the modeling process

00:37:24.090 --> 00:37:24.830 align:middle line:90%
is the splitting.

00:37:24.830 --> 00:37:26.390 align:middle line:90%
And the splitting also--

00:37:26.390 --> 00:37:30.510 align:middle line:84%
the standardization-- if
the standardization, which

00:37:30.510 --> 00:37:33.730 align:middle line:84%
is part of the process, uses
information about the test set,

00:37:33.730 --> 00:37:37.510 align:middle line:84%
well, the test set is not really
kept away from anything, is it?

00:37:37.510 --> 00:37:39.790 align:middle line:84%
That's why we want to
split it, lock away

00:37:39.790 --> 00:37:42.250 align:middle line:84%
the test set somewhere, and
then proceed with the modeling.

00:37:42.250 --> 00:37:43.750 align:middle line:84%
Again, this is like
machine learning

00:37:43.750 --> 00:37:46.950 align:middle line:84%
101, which is why I'm going
through it pretty fast.

00:37:46.950 --> 00:37:52.550 align:middle line:84%
So we do this sampling
function, take 20% of the data,

00:37:52.550 --> 00:37:53.890 align:middle line:90%
and make it the test set.

00:37:53.890 --> 00:37:56.090 align:middle line:84%
And the remaining is going
to be the training set.

00:37:56.090 --> 00:37:58.750 align:middle line:84%
And when we do that, you
can see the training set

00:37:58.750 --> 00:38:05.110 align:middle line:84%
is now 242 rows, while
the test is 61 rows.

00:38:05.110 --> 00:38:08.640 align:middle line:84%
And any of these data frames,
you'll know the shape attribute

00:38:08.640 --> 00:38:11.540 align:middle line:84%
gives you the dimensions of the
number of rows and the columns.

00:38:11.540 --> 00:38:12.780 align:middle line:90%
That's what we're doing here.

00:38:12.780 --> 00:38:14.580 align:middle line:84%
And now that we have done
that, we have done the split,

00:38:14.580 --> 00:38:17.220 align:middle line:84%
we can calculate the mean
and the standard deviation.

00:38:17.220 --> 00:38:18.440 align:middle line:90%
So I calculate the mean here.

00:38:18.440 --> 00:38:20.060 align:middle line:84%
I calculate the
standard deviation.

00:38:20.060 --> 00:38:21.220 align:middle line:90%
These are all the means.

00:38:21.220 --> 00:38:25.420 align:middle line:84%
And once I do that, I just do
each column minus the mean,

00:38:25.420 --> 00:38:26.900 align:middle line:90%
divide the standard deviation.

00:38:26.900 --> 00:38:28.920 align:middle line:84%
And then once I do
that, I save them

00:38:28.920 --> 00:38:31.300 align:middle line:84%
in the train and
the test dataframes.

00:38:31.300 --> 00:38:34.920 align:middle line:84%
And you can see here now all the
numbers are all very smallish--

00:38:34.920 --> 00:38:37.340 align:middle line:84%
0, 1, minus 1,
around that range.

00:38:37.340 --> 00:38:39.840 align:middle line:84%
And that's kind of ideal
for neural network training.

00:38:39.840 --> 00:38:40.800 align:middle line:90%
OK?

00:38:40.800 --> 00:38:41.300 align:middle line:90%
All right.

00:38:41.300 --> 00:38:43.520 align:middle line:84%
So at this point, the
data is entirely numeric.

00:38:43.520 --> 00:38:46.640 align:middle line:84%
And then we are almost
ready to feed it into Keras.

00:38:46.640 --> 00:38:49.760 align:middle line:84%
And the way you do it is
you take a NumPy array--

00:38:49.760 --> 00:38:51.960 align:middle line:84%
you take a Pandas
dataframe, and then you

00:38:51.960 --> 00:38:53.620 align:middle line:90%
convert it into a NumPy array.

00:38:53.620 --> 00:38:56.340 align:middle line:84%
And then Keras is happy to
take it, happy to receive it.

00:38:56.340 --> 00:39:00.320 align:middle line:84%
So we use this thing called
to_numpy, which I think

00:39:00.320 --> 00:39:02.920 align:middle line:84%
is as descriptive as
it gets in programming.

00:39:02.920 --> 00:39:04.780 align:middle line:84%
And then you save it
as train and test.

00:39:04.780 --> 00:39:07.180 align:middle line:84%
Now, train and test
are two NumPy arrays

00:39:07.180 --> 00:39:08.880 align:middle line:84%
with exactly the
same information,

00:39:08.880 --> 00:39:10.780 align:middle line:84%
and now we can
feed it into Keras.

00:39:10.780 --> 00:39:11.280 align:middle line:90%
All right.

00:39:11.280 --> 00:39:13.363 align:middle line:84%
Now I guess there's one
other thing we need to do,

00:39:13.363 --> 00:39:17.300 align:middle line:84%
which is that in this
dataframe, train and test are

00:39:17.300 --> 00:39:18.385 align:middle line:90%
independent variables.

00:39:18.385 --> 00:39:20.760 align:middle line:84%
All the features, as well as
the target, the 0, 1 target,

00:39:20.760 --> 00:39:23.620 align:middle line:90%
they're all in this.

00:39:23.620 --> 00:39:25.420 align:middle line:84%
And we need to now
take it and just

00:39:25.420 --> 00:39:29.220 align:middle line:84%
take the dependent variable, the
0, 1 column, and split it out,

00:39:29.220 --> 00:39:31.540 align:middle line:84%
and keep the x and
the y separately.

00:39:31.540 --> 00:39:32.800 align:middle line:90%
That's the whole point of it.

00:39:32.800 --> 00:39:35.260 align:middle line:84%
Because you need to feed
the x, do the prediction,

00:39:35.260 --> 00:39:36.980 align:middle line:84%
and then compare
it to the actual y,

00:39:36.980 --> 00:39:39.180 align:middle line:84%
and calculate the loss,
and so on and so forth.

00:39:39.180 --> 00:39:42.900 align:middle line:84%
So the target column
is our y variable.

00:39:42.900 --> 00:39:44.800 align:middle line:84%
And it's column number
6 from the left.

00:39:44.800 --> 00:39:46.400 align:middle line:90%
If you count it, you can see it.

00:39:46.400 --> 00:39:51.900 align:middle line:84%
So we delete it from
the train and test.

00:39:51.900 --> 00:39:56.720 align:middle line:84%
And now we have 242 rows
and 29 columns, 29 features.

00:39:56.720 --> 00:39:59.160 align:middle line:84%
You will recall from the
network that we made way back,

00:39:59.160 --> 00:40:03.040 align:middle line:84%
it had 29 inputs, 29
nodes in the input layer.

00:40:03.040 --> 00:40:05.110 align:middle line:84%
And that's where the
29 is coming from.

00:40:05.110 --> 00:40:07.647 align:middle line:84%
And so now we just
select the sixth column,

00:40:07.647 --> 00:40:09.230 align:middle line:84%
which is the target,
and make it the y

00:40:09.230 --> 00:40:11.670 align:middle line:90%
variable, train_y and test_y.

00:40:11.670 --> 00:40:13.170 align:middle line:84%
And that is, of
course, a vector,

00:40:13.170 --> 00:40:15.590 align:middle line:84%
which is 242 long in the
training set and 61 long

00:40:15.590 --> 00:40:16.510 align:middle line:90%
in the thing.

00:40:16.510 --> 00:40:18.510 align:middle line:84%
So at this point,
all we have done

00:40:18.510 --> 00:40:21.495 align:middle line:84%
is, to be honest,
boring preprocessing.

00:40:21.495 --> 00:40:23.370 align:middle line:84%
We haven't actually
gotten to the action yet.

00:40:23.370 --> 00:40:26.170 align:middle line:90%
Finally, let's do something.

00:40:26.170 --> 00:40:28.818 align:middle line:84%
And we start with a
single hidden layer.

00:40:28.818 --> 00:40:30.610 align:middle line:84%
Since it's a binary
classification problem,

00:40:30.610 --> 00:40:32.530 align:middle line:84%
we'll use sigmoids,
as we saw earlier.

00:40:32.530 --> 00:40:36.490 align:middle line:84%
And this is the model we
created in class, last class.

00:40:36.490 --> 00:40:39.150 align:middle line:90%
This is the model we created.

00:40:39.150 --> 00:40:41.390 align:middle line:84%
The only difference between
that model and this model

00:40:41.390 --> 00:40:44.710 align:middle line:84%
is that I've actually given
names to these layers.

00:40:44.710 --> 00:40:46.743 align:middle line:84%
And this name thing
is totally optional.

00:40:46.743 --> 00:40:48.410 align:middle line:84%
If you want to give
a name, give a name.

00:40:48.410 --> 00:40:50.410 align:middle line:84%
It's just a little easier
to interpret later on.

00:40:50.410 --> 00:40:52.910 align:middle line:90%
It's just cosmetic.

00:40:52.910 --> 00:40:54.990 align:middle line:90%
But I've just put it here.

00:40:54.990 --> 00:40:57.550 align:middle line:84%
And once you build
the model, you

00:40:57.550 --> 00:41:00.470 align:middle line:84%
should immediately run
the model.summary command

00:41:00.470 --> 00:41:03.705 align:middle line:84%
because it gives you a
nice overview of the model.

00:41:03.705 --> 00:41:05.080 align:middle line:84%
For each layer,
it tells you what

00:41:05.080 --> 00:41:08.020 align:middle line:84%
the layer is, it tells you
what's coming into the layer,

00:41:08.020 --> 00:41:10.200 align:middle line:84%
meaning the shape of the
tensor that's coming in

00:41:10.200 --> 00:41:13.460 align:middle line:84%
and what's going out, and how
many parameters the layer has.

00:41:13.460 --> 00:41:15.880 align:middle line:84%
And it turns out,
this layer has--

00:41:15.880 --> 00:41:20.640 align:middle line:84%
sorry, this network
has 497 parameters.

00:41:20.640 --> 00:41:22.887 align:middle line:84%
And I have told you repeatedly,
the first few times,

00:41:22.887 --> 00:41:24.720 align:middle line:84%
just hand-calculate the
number of parameters

00:41:24.720 --> 00:41:26.140 align:middle line:90%
to make sure it verifies.

00:41:26.140 --> 00:41:29.340 align:middle line:84%
So we should just make sure
that it is, in fact, 497.

00:41:29.340 --> 00:41:30.900 align:middle line:90%
So let's hand-calculate it.

00:41:30.900 --> 00:41:34.000 align:middle line:84%
And you do basically-- it's
basically what's going on here.

00:41:34.000 --> 00:41:36.300 align:middle line:90%
29 inputs times 16.

00:41:36.300 --> 00:41:37.240 align:middle line:90%
All the arrows.

00:41:37.240 --> 00:41:39.080 align:middle line:90%
29 times 16 arrows.

00:41:39.080 --> 00:41:41.300 align:middle line:84%
And then you have a
bias of another 16.

00:41:41.300 --> 00:41:42.880 align:middle line:84%
That's why you have
this expression.

00:41:42.880 --> 00:41:44.560 align:middle line:84%
And then the next
one is 16 times

00:41:44.560 --> 00:41:46.740 align:middle line:84%
1 plus 1 bias for
the output sigmoid.

00:41:46.740 --> 00:41:48.480 align:middle line:90%
And you get to 497.

00:41:48.480 --> 00:41:48.980 align:middle line:90%
OK.

00:41:48.980 --> 00:41:50.840 align:middle line:84%
Just make sure you
follow this later on when

00:41:50.840 --> 00:41:52.160 align:middle line:90%
you work with the Colab.

00:41:52.160 --> 00:41:54.392 align:middle line:90%
We did this in class last week.

00:41:54.392 --> 00:41:56.600 align:middle line:84%
And you can visualize the
network graphically as well

00:41:56.600 --> 00:41:58.600 align:middle line:84%
by using the
plot_model function.

00:41:58.600 --> 00:42:00.908 align:middle line:90%
So we do that here.

00:42:00.908 --> 00:42:02.950 align:middle line:84%
And let's say, it gives
you the same information,

00:42:02.950 --> 00:42:05.390 align:middle line:84%
but in a slightly
easier form to consume.

00:42:05.390 --> 00:42:08.302 align:middle line:84%
And when we work with larger
networks, starting on Wednesday,

00:42:08.302 --> 00:42:10.010 align:middle line:84%
you will see that
being able to visualize

00:42:10.010 --> 00:42:13.010 align:middle line:84%
the topology of the network
is actually quite handy.

00:42:13.010 --> 00:42:13.650 align:middle line:90%
OK.

00:42:13.650 --> 00:42:17.410 align:middle line:84%
We finally come to actually
trying to train this thing.

00:42:17.410 --> 00:42:20.050 align:middle line:84%
And so what loss
function should we use?

00:42:20.050 --> 00:42:23.090 align:middle line:84%
We need to use
binary_crossentropy,

00:42:23.090 --> 00:42:24.090 align:middle line:90%
right there.

00:42:24.090 --> 00:42:25.590 align:middle line:90%
What optimizer to use?

00:42:25.590 --> 00:42:29.930 align:middle line:84%
Well, as I mentioned
earlier, we will use Adam.

00:42:29.930 --> 00:42:32.490 align:middle line:90%
Adam.

00:42:32.490 --> 00:42:33.030 align:middle line:90%
All right.

00:42:33.030 --> 00:42:35.370 align:middle line:90%
Adam.

00:42:35.370 --> 00:42:37.210 align:middle line:84%
And then the final
thing is, you can

00:42:37.210 --> 00:42:40.230 align:middle line:84%
ask Keras to report out
whatever metrics you care about.

00:42:40.230 --> 00:42:42.730 align:middle line:84%
These metrics are not going to
be used in any optimization.

00:42:42.730 --> 00:42:44.330 align:middle line:90%
It's just reporting it to you.

00:42:44.330 --> 00:42:46.130 align:middle line:84%
And the most common
thing people report

00:42:46.130 --> 00:42:49.070 align:middle line:84%
out for binary
classification is accuracy.

00:42:49.070 --> 00:42:51.130 align:middle line:84%
So we'll just go
with that metric.

00:42:51.130 --> 00:42:55.050 align:middle line:84%
And so what we do is we tell
Keras, take the model we just

00:42:55.050 --> 00:42:58.510 align:middle line:84%
built and compile it with
this choice of optimizer,

00:42:58.510 --> 00:43:00.710 align:middle line:84%
this choice of loss
function, and these metrics.

00:43:00.710 --> 00:43:02.470 align:middle line:84%
And this compilation
step, what it does

00:43:02.470 --> 00:43:05.630 align:middle line:84%
is, essentially, Keras
will take this information

00:43:05.630 --> 00:43:08.350 align:middle line:84%
and take the model you've
built, and it will reorganize

00:43:08.350 --> 00:43:12.750 align:middle line:84%
the model in such a way
that the parallel computing,

00:43:12.750 --> 00:43:14.710 align:middle line:84%
distribution of computing
across many servers,

00:43:14.710 --> 00:43:18.350 align:middle line:84%
and so on, that's what's
happening in the compile step.

00:43:18.350 --> 00:43:20.390 align:middle line:84%
Organizing it so that--
reorganizing the model

00:43:20.390 --> 00:43:22.670 align:middle line:84%
so that it becomes
amenable to parallelization

00:43:22.670 --> 00:43:24.490 align:middle line:84%
and distribution,
that's what's going on.

00:43:24.490 --> 00:43:26.323 align:middle line:84%
That's why you actually
have to do something

00:43:26.323 --> 00:43:27.460 align:middle line:90%
called a compile step.

00:43:27.460 --> 00:43:30.190 align:middle line:84%
And once we do
that, we are finally

00:43:30.190 --> 00:43:31.790 align:middle line:90%
ready to train the model.

00:43:31.790 --> 00:43:35.370 align:middle line:84%
And to do that, we have to
decide what the batch size is

00:43:35.370 --> 00:43:36.370 align:middle line:90%
that we're going to use.

00:43:36.370 --> 00:43:37.390 align:middle line:84%
Remember, we're
using some flavor

00:43:37.390 --> 00:43:40.270 align:middle line:84%
of SGD, which means we have to
choose what is the batch size.

00:43:40.270 --> 00:43:43.470 align:middle line:84%
And typically what
people do is that 32 is

00:43:43.470 --> 00:43:45.430 align:middle line:90%
a good default for batch size.

00:43:45.430 --> 00:43:48.390 align:middle line:84%
If you're just getting started
with something, just use 32.

00:43:48.390 --> 00:43:50.172 align:middle line:84%
And there's a whole
bunch of literature

00:43:50.172 --> 00:43:51.630 align:middle line:84%
on what the right
batch size should

00:43:51.630 --> 00:43:53.930 align:middle line:84%
be for the number of
data points you have,

00:43:53.930 --> 00:43:56.050 align:middle line:84%
the size of the network,
and so on and so forth.

00:43:56.050 --> 00:43:58.410 align:middle line:90%
My philosophy is start with 32.

00:43:58.410 --> 00:44:01.020 align:middle line:84%
And you can always
try 32, 64, 128.

00:44:01.020 --> 00:44:03.480 align:middle line:84%
It's kind of like--
oftentimes what people

00:44:03.480 --> 00:44:06.400 align:middle line:84%
tell me, researchers tell me, is
that just use the biggest batch

00:44:06.400 --> 00:44:09.782 align:middle line:84%
size that doesn't
make your machine die.

00:44:09.782 --> 00:44:11.740 align:middle line:84%
If you can fit into
memory, it's probably good.

00:44:11.740 --> 00:44:12.900 align:middle line:90%
Just try the bigger sizes.

00:44:12.900 --> 00:44:13.700 align:middle line:90%
We'll just start with 32.

00:44:13.700 --> 00:44:14.533 align:middle line:90%
It's a tiny problem.

00:44:14.533 --> 00:44:15.400 align:middle line:90%
It's not a big deal.

00:44:15.400 --> 00:44:18.740 align:middle line:84%
And then we also have to decide,
how many epochs through the data

00:44:18.740 --> 00:44:20.440 align:middle line:90%
do we want to go through?

00:44:20.440 --> 00:44:22.180 align:middle line:90%
How many epochs?

00:44:22.180 --> 00:44:26.365 align:middle line:84%
And usually, 20 to 30 epochs
is a good starting point.

00:44:26.365 --> 00:44:28.740 align:middle line:84%
And then, because this is a
tiny problem, just for kicks,

00:44:28.740 --> 00:44:31.080 align:middle line:84%
I just had to run it
for 300 epochs, just

00:44:31.080 --> 00:44:33.498 align:middle line:84%
to see if any overfitting
is going to happen.

00:44:33.498 --> 00:44:35.540 align:middle line:84%
And then whether we want
to use a validation set.

00:44:35.540 --> 00:44:38.220 align:middle line:84%
Of course, we want to
use a validation set.

00:44:38.220 --> 00:44:40.158 align:middle line:84%
So we will use 20%
of the data points

00:44:40.158 --> 00:44:42.450 align:middle line:84%
as a validation set so that
we can look for overfitting

00:44:42.450 --> 00:44:44.320 align:middle line:90%
and underfitting.

00:44:44.320 --> 00:44:44.980 align:middle line:90%
All right.

00:44:44.980 --> 00:44:47.840 align:middle line:84%
So with these decisions
made, we finally--

00:44:47.840 --> 00:44:50.060 align:middle line:90%
we use the model.fit command.

00:44:50.060 --> 00:44:54.320 align:middle line:84%
model.fit is what actually
trains the neural network.

00:44:54.320 --> 00:44:59.330 align:middle line:84%
And you have to tell it
what the x tensor is.

00:44:59.330 --> 00:45:02.970 align:middle line:84%
You have to tell it what the
dependent variable y tensor is.

00:45:02.970 --> 00:45:05.270 align:middle line:84%
We need to tell it how
many epochs to do this,

00:45:05.270 --> 00:45:06.690 align:middle line:90%
what the batch size to use.

00:45:06.690 --> 00:45:10.770 align:middle line:84%
verbose=1 just means just put a
lot of descriptive output as you

00:45:10.770 --> 00:45:11.670 align:middle line:90%
do this thing.

00:45:11.670 --> 00:45:16.370 align:middle line:84%
And then validation_split means,
take 20% of the training data

00:45:16.370 --> 00:45:18.285 align:middle line:84%
and set it aside as your
validation data set.

00:45:18.285 --> 00:45:20.410 align:middle line:84%
Don't use it for training
because I want to measure

00:45:20.410 --> 00:45:22.210 align:middle line:90%
overfitting using that.

00:45:22.210 --> 00:45:22.910 align:middle line:90%
So that's it.

00:45:22.910 --> 00:45:24.150 align:middle line:90%
So you do that thing.

00:45:24.150 --> 00:45:25.750 align:middle line:90%
It will run for 300 epochs.

00:45:25.750 --> 00:45:28.650 align:middle line:84%
And this is the reason why I
decided to just not actually

00:45:28.650 --> 00:45:31.050 align:middle line:90%
run it in class.

00:45:31.050 --> 00:45:32.730 align:middle line:90%
And so you keep on doing it.

00:45:32.730 --> 00:45:34.310 align:middle line:90%
It gives you a lot of output.

00:45:34.310 --> 00:45:37.540 align:middle line:90%
And finally, we reach the end.

00:45:37.540 --> 00:45:41.550 align:middle line:90%


00:45:41.550 --> 00:45:42.050 align:middle line:90%
OK.

00:45:42.050 --> 00:45:43.610 align:middle line:84%
Now, let's take a
moment to understand

00:45:43.610 --> 00:45:44.527 align:middle line:90%
what's being reported.

00:45:44.527 --> 00:45:46.350 align:middle line:84%
So I'll just take
this one line here.

00:45:46.350 --> 00:45:48.890 align:middle line:90%
So these two--

00:45:48.890 --> 00:45:51.010 align:middle line:84%
There is a pair of
lines for each epoch.

00:45:51.010 --> 00:45:53.834 align:middle line:90%
And then here it's telling you--

00:45:53.834 --> 00:46:00.020 align:middle line:84%
it actually uses-- in this
300th epoch, it used 7 batches.

00:46:00.020 --> 00:46:01.980 align:middle line:90%
7 out of 7 batches.

00:46:01.980 --> 00:46:03.520 align:middle line:90%
So it used 7 batches.

00:46:03.520 --> 00:46:06.378 align:middle line:84%
And you will recall, from
the math we did in the class,

00:46:06.378 --> 00:46:08.420 align:middle line:84%
that it's actually 7
batches, where the first six

00:46:08.420 --> 00:46:10.300 align:middle line:84%
batches are 32,
and the last batch

00:46:10.300 --> 00:46:11.720 align:middle line:90%
is just a couple of examples.

00:46:11.720 --> 00:46:13.680 align:middle line:90%
But we have 7 batches.

00:46:13.680 --> 00:46:18.100 align:middle line:84%
This is the 193
by 32 rounded up.

00:46:18.100 --> 00:46:19.820 align:middle line:90%
So that's why you have 7 here.

00:46:19.820 --> 00:46:22.000 align:middle line:84%
And then it tells you how
long it took it for that.

00:46:22.000 --> 00:46:23.920 align:middle line:90%
And then this is the loss value.

00:46:23.920 --> 00:46:26.180 align:middle line:84%
This is the binary
cross-entropy loss value

00:46:26.180 --> 00:46:30.300 align:middle line:84%
on the training set on
that particular batch

00:46:30.300 --> 00:46:32.060 align:middle line:90%
that it calculated.

00:46:32.060 --> 00:46:35.040 align:middle line:84%
This is the accuracy that
you asked it to report out.

00:46:35.040 --> 00:46:37.700 align:middle line:90%
98.5% accuracy on that batch.

00:46:37.700 --> 00:46:41.260 align:middle line:84%
And then, at the
end of this epoch,

00:46:41.260 --> 00:46:43.765 align:middle line:84%
using whatever weights were
available in that network,

00:46:43.765 --> 00:46:45.140 align:middle line:84%
you'd actually
calculate the loss

00:46:45.140 --> 00:46:47.820 align:middle line:84%
on the validation set,
which is the 20% the data

00:46:47.820 --> 00:46:49.100 align:middle line:90%
we have set aside.

00:46:49.100 --> 00:46:52.620 align:middle line:84%
And then this is the accuracy
on the validation set.

00:46:52.620 --> 00:46:55.480 align:middle line:84%
So that's what each
of these numbers mean.

00:46:55.480 --> 00:46:58.380 align:middle line:84%
Now, looking at this wall of
numbers is kind of painful.

00:46:58.380 --> 00:47:00.940 align:middle line:90%
So usually you just plot it.

00:47:00.940 --> 00:47:03.883 align:middle line:84%
And the way you do that
is if you notice here--

00:47:03.883 --> 00:47:05.300 align:middle line:84%
OK, I'm not going
to go back here.

00:47:05.300 --> 00:47:08.320 align:middle line:84%
So I said history equals
model.fit, blah, blah, blah,

00:47:08.320 --> 00:47:08.940 align:middle line:90%
blah, blah.

00:47:08.940 --> 00:47:11.080 align:middle line:84%
And that history object
has a lot of information

00:47:11.080 --> 00:47:14.400 align:middle line:84%
that we can use for plotting
and diagnostics and so on.

00:47:14.400 --> 00:47:17.520 align:middle line:84%
And that history
thing, history object,

00:47:17.520 --> 00:47:20.620 align:middle line:84%
has another object called
history, history.history,

00:47:20.620 --> 00:47:22.580 align:middle line:84%
which is a dictionary
with all these values.

00:47:22.580 --> 00:47:23.820 align:middle line:84%
And that's what
we're going to plot.

00:47:23.820 --> 00:47:24.903 align:middle line:90%
Was there a question here?

00:47:24.903 --> 00:47:25.840 align:middle line:90%
Yeah.

00:47:25.840 --> 00:47:29.640 align:middle line:84%
AUDIENCE: So you prompted it to
keep 20% aside for validation.

00:47:29.640 --> 00:47:32.780 align:middle line:84%
But didn't we already
do a test set?

00:47:32.780 --> 00:47:34.680 align:middle line:84%
So that's going to be
a secondary validation.

00:47:34.680 --> 00:47:35.722 align:middle line:90%
RAMA RAMAKRISHNAN: Right.

00:47:35.722 --> 00:47:38.040 align:middle line:84%
So basically, we have
a training and then

00:47:38.040 --> 00:47:39.700 align:middle line:90%
a validation and a test.

00:47:39.700 --> 00:47:41.880 align:middle line:84%
The role of the validation
set is to figure out

00:47:41.880 --> 00:47:43.267 align:middle line:90%
things like early stopping.

00:47:43.267 --> 00:47:44.100 align:middle line:90%
Should we stop here?

00:47:44.100 --> 00:47:45.060 align:middle line:90%
Should we go back?

00:47:45.060 --> 00:47:48.200 align:middle line:84%
And as you will see later on,
if we use hyperparameters,

00:47:48.200 --> 00:47:50.325 align:middle line:84%
we will try different values
of the hyperparameters

00:47:50.325 --> 00:47:52.533 align:middle line:84%
and figure out-- use the
validation set to figure out

00:47:52.533 --> 00:47:53.650 align:middle line:90%
which one is the best one.

00:47:53.650 --> 00:47:55.270 align:middle line:84%
But once we are
done with all that,

00:47:55.270 --> 00:47:57.090 align:middle line:90%
we will finally have a model.

00:47:57.090 --> 00:48:00.290 align:middle line:84%
At that point, we open the
safe, take out the test set,

00:48:00.290 --> 00:48:03.240 align:middle line:84%
and use it just once with
your final, final model.

00:48:03.240 --> 00:48:04.990 align:middle line:84%
Not because you want
to improve the model,

00:48:04.990 --> 00:48:07.157 align:middle line:84%
but because you want to
have a realistic idea of how

00:48:07.157 --> 00:48:11.770 align:middle line:84%
it will do when you actually
deploy it out in the real world.

00:48:11.770 --> 00:48:14.370 align:middle line:90%
Yeah.

00:48:14.370 --> 00:48:16.410 align:middle line:90%
AUDIENCE: Can we use--

00:48:16.410 --> 00:48:19.730 align:middle line:84%
instead of accuracy, can
be used other metrics

00:48:19.730 --> 00:48:21.090 align:middle line:90%
to evaluate whether to--

00:48:21.090 --> 00:48:21.370 align:middle line:90%
RAMA RAMAKRISHNAN: Absolutely.

00:48:21.370 --> 00:48:23.745 align:middle line:84%
AUDIENCE: --actually [INAUDIBLE]
on the confusion matrix,

00:48:23.745 --> 00:48:24.250 align:middle line:90%
let's say?

00:48:24.250 --> 00:48:24.850 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:48:24.850 --> 00:48:26.058 align:middle line:90%
You can do whatever you want.

00:48:26.058 --> 00:48:28.450 align:middle line:84%
You can use-- like I said,
it's not used for training.

00:48:28.450 --> 00:48:31.470 align:middle line:84%
So there is no mathematical
implication of what you choose.

00:48:31.470 --> 00:48:34.302 align:middle line:84%
You can choose error
rates, accuracy, F1 F beta.

00:48:34.302 --> 00:48:35.510 align:middle line:90%
You can do whatever you want.

00:48:35.510 --> 00:48:38.570 align:middle line:84%
And Keras, as you will
see, has this dizzying list

00:48:38.570 --> 00:48:40.975 align:middle line:84%
of possible metrics you
can use for reporting.

00:48:40.975 --> 00:48:42.850 align:middle line:84%
The key thing to remember
is that you're just

00:48:42.850 --> 00:48:43.870 align:middle line:90%
reporting these metrics.

00:48:43.870 --> 00:48:47.490 align:middle line:84%
You're not actually using
them for any training.

00:48:47.490 --> 00:48:49.103 align:middle line:90%
Yeah.

00:48:49.103 --> 00:48:51.270 align:middle line:84%
AUDIENCE: My question is
with respect to validation.

00:48:51.270 --> 00:48:53.740 align:middle line:84%
Like, we've got a
training data set.

00:48:53.740 --> 00:48:58.660 align:middle line:84%
So when we take out 20%
of data from validation,

00:48:58.660 --> 00:49:00.820 align:middle line:84%
are we taking out from
the training set or--

00:49:00.820 --> 00:49:01.140 align:middle line:90%
RAMA RAMAKRISHNAN: Correct.

00:49:01.140 --> 00:49:02.515 align:middle line:84%
AUDIENCE: From
there, that level?

00:49:02.515 --> 00:49:04.980 align:middle line:84%
Or we go to each batch and
take out 20% from each?

00:49:04.980 --> 00:49:06.360 align:middle line:84%
RAMA RAMAKRISHNAN: We're taking
out from the training set.

00:49:06.360 --> 00:49:07.943 align:middle line:84%
AUDIENCE: So it means
the batch size--

00:49:07.943 --> 00:49:10.733 align:middle line:84%
the number of data would be
available for calculating

00:49:10.733 --> 00:49:11.900 align:middle line:90%
the batch size would reduce?

00:49:11.900 --> 00:49:13.025 align:middle line:90%
RAMA RAMAKRISHNAN: Correct.

00:49:13.025 --> 00:49:15.140 align:middle line:84%
And in fact, once we take
out the validation set,

00:49:15.140 --> 00:49:17.100 align:middle line:90%
whatever remaining is 193.

00:49:17.100 --> 00:49:19.240 align:middle line:84%
AUDIENCE: And then we
divide that into batches.

00:49:19.240 --> 00:49:22.700 align:middle line:84%
And then every epoch, the
validation and the data

00:49:22.700 --> 00:49:23.703 align:middle line:90%
gets different.

00:49:23.703 --> 00:49:24.620 align:middle line:90%
RAMA RAMAKRISHNAN: No.

00:49:24.620 --> 00:49:27.260 align:middle line:84%
Once you take out the validation
set at the very beginning,

00:49:27.260 --> 00:49:29.340 align:middle line:84%
you keep it aside,
and then you only

00:49:29.340 --> 00:49:32.580 align:middle line:84%
evaluate at the
end of each epoch

00:49:32.580 --> 00:49:35.865 align:middle line:84%
what your loss and accuracy
is on that validation set.

00:49:35.865 --> 00:49:37.740 align:middle line:84%
AUDIENCE: So you would
have cross-validation.

00:49:37.740 --> 00:49:39.400 align:middle line:84%
RAMA RAMAKRISHNAN: No, no, we're
not doing any of that stuff.

00:49:39.400 --> 00:49:41.733 align:middle line:84%
We're just taking it out once,
and we're just evaluating

00:49:41.733 --> 00:49:43.460 align:middle line:90%
the end of every epoch.

00:49:43.460 --> 00:49:44.340 align:middle line:90%
OK.

00:49:44.340 --> 00:49:47.700 align:middle line:90%
So yeah.

00:49:47.700 --> 00:49:51.910 align:middle line:84%
AUDIENCE: So I know we both
asked similar questions- but--

00:49:51.910 --> 00:49:53.910 align:middle line:84%
RAMA RAMAKRISHNAN: Use
the microphone, please.

00:49:53.910 --> 00:49:55.370 align:middle line:84%
AUDIENCE: I know both of
us asked similar questions,

00:49:55.370 --> 00:49:56.610 align:middle line:90%
but just to reconfirm.

00:49:56.610 --> 00:49:59.350 align:middle line:84%
So here, my training
model is giving me, say,

00:49:59.350 --> 00:50:01.710 align:middle line:90%
a loss of 0.0860.

00:50:01.710 --> 00:50:04.750 align:middle line:84%
My validation model
is giving me 0.660.

00:50:04.750 --> 00:50:07.590 align:middle line:84%
That means I have
already crossed the int.

00:50:07.590 --> 00:50:11.950 align:middle line:84%
So when I have to actually
test the model, that

00:50:11.950 --> 00:50:13.630 align:middle line:84%
is the midpoint
which I did, that

00:50:13.630 --> 00:50:16.705 align:middle line:84%
will be the model which will
get deployed in production.

00:50:16.705 --> 00:50:17.830 align:middle line:90%
RAMA RAMAKRISHNAN: Correct.

00:50:17.830 --> 00:50:19.930 align:middle line:84%
And as to, OK, what do
we do to get that model--

00:50:19.930 --> 00:50:22.430 align:middle line:84%
do we actually have to go back
to the beginning and run it

00:50:22.430 --> 00:50:25.570 align:middle line:84%
for a few epochs, or can we do
something smarter than that?--

00:50:25.570 --> 00:50:26.870 align:middle line:90%
we'll get to that.

00:50:26.870 --> 00:50:27.870 align:middle line:90%
Yeah.

00:50:27.870 --> 00:50:30.850 align:middle line:84%
AUDIENCE: Isn't that validation
set different for each epoch,

00:50:30.850 --> 00:50:31.600 align:middle line:90%
or is it the same?

00:50:31.600 --> 00:50:32.975 align:middle line:84%
RAMA RAMAKRISHNAN:
It's the same.

00:50:32.975 --> 00:50:34.950 align:middle line:84%
So what you do is you
have a training set.

00:50:34.950 --> 00:50:36.950 align:middle line:84%
Before you do any
training, you take out 20%.

00:50:36.950 --> 00:50:38.130 align:middle line:90%
Keep it aside.

00:50:38.130 --> 00:50:39.930 align:middle line:90%
You take whatever is left over.

00:50:39.930 --> 00:50:42.430 align:middle line:84%
You divide that into
minibatches and then start

00:50:42.430 --> 00:50:43.930 align:middle line:90%
running it through each epochs.

00:50:43.930 --> 00:50:46.030 align:middle line:84%
But at the end of
each epoch, you just

00:50:46.030 --> 00:50:47.870 align:middle line:84%
evaluate the quality
of that resulting

00:50:47.870 --> 00:50:49.810 align:middle line:90%
model using the validation set.

00:50:49.810 --> 00:50:51.772 align:middle line:84%
AUDIENCE: What's different
between each epoch?

00:50:51.772 --> 00:50:52.730 align:middle line:90%
Is it just the weight--

00:50:52.730 --> 00:50:54.563 align:middle line:84%
RAMA RAMAKRISHNAN: The
weights have changed.

00:50:54.563 --> 00:50:57.753 align:middle line:84%
AUDIENCE: It's the division
into the different--

00:50:57.753 --> 00:50:58.670 align:middle line:90%
RAMA RAMAKRISHNAN: No.

00:50:58.670 --> 00:51:02.090 align:middle line:84%
So the difference in each epoch
is the weights have changed.

00:51:02.090 --> 00:51:04.230 align:middle line:84%
So after every minibatch,
the weights have changed.

00:51:04.230 --> 00:51:05.730 align:middle line:84%
At the end of one
epoch, you've gone

00:51:05.730 --> 00:51:09.138 align:middle line:84%
through all the data points you
ever had in the training set.

00:51:09.138 --> 00:51:11.680 align:middle line:84%
And then you come back to the
beginning, and you do it again.

00:51:11.680 --> 00:51:17.690 align:middle line:90%


00:51:17.690 --> 00:51:20.770 align:middle line:84%
AUDIENCE: How do you
identify the sweet spot?

00:51:20.770 --> 00:51:22.490 align:middle line:90%
RAMA RAMAKRISHNAN: It's coming.

00:51:22.490 --> 00:51:23.050 align:middle line:90%
Yeah.

00:51:23.050 --> 00:51:23.550 align:middle line:90%
All right.

00:51:23.550 --> 00:51:24.970 align:middle line:90%
So I'm going to keep going.

00:51:24.970 --> 00:51:26.530 align:middle line:90%
So we have this here.

00:51:26.530 --> 00:51:27.930 align:middle line:90%
And so you just--

00:51:27.930 --> 00:51:30.310 align:middle line:84%
I mean, there's a little
bit of Matplotlib code.

00:51:30.310 --> 00:51:32.410 align:middle line:84%
So what we do is we
just plot the training

00:51:32.410 --> 00:51:34.890 align:middle line:84%
loss and the validation
loss as a function

00:51:34.890 --> 00:51:36.770 align:middle line:90%
of the number of epochs.

00:51:36.770 --> 00:51:39.450 align:middle line:84%
And as you can see
here, the training loss

00:51:39.450 --> 00:51:40.730 align:middle line:90%
is these things here.

00:51:40.730 --> 00:51:44.530 align:middle line:84%
And it's steadily going
down, as you would expect.

00:51:44.530 --> 00:51:46.910 align:middle line:84%
The validation loss
goes down here.

00:51:46.910 --> 00:51:49.280 align:middle line:84%
And then, at some point,
it kind of flattens out,

00:51:49.280 --> 00:51:51.460 align:middle line:84%
and then maybe gently
starts to rise.

00:51:51.460 --> 00:51:56.140 align:middle line:84%
So do you think
there's overfitting?

00:51:56.140 --> 00:51:58.320 align:middle line:84%
There seems to be some
level of overfitting here.

00:51:58.320 --> 00:52:00.180 align:middle line:84%
But the thing you have
to always remember

00:52:00.180 --> 00:52:04.980 align:middle line:84%
is that the binary cross-entropy
loss is a loss function that

00:52:04.980 --> 00:52:07.420 align:middle line:84%
is convenient for you
because it sort of captures

00:52:07.420 --> 00:52:09.980 align:middle line:84%
the thing you want to
capture, the discrepancy,

00:52:09.980 --> 00:52:13.140 align:middle line:84%
but also because it's
mathematically convenient.

00:52:13.140 --> 00:52:15.540 align:middle line:84%
But what you may actually
care about in practice

00:52:15.540 --> 00:52:17.755 align:middle line:90%
is something like accuracy.

00:52:17.755 --> 00:52:20.380 align:middle line:84%
So I always-- that's why you're
reporting out the accuracy when

00:52:20.380 --> 00:52:21.320 align:middle line:90%
we do these things.

00:52:21.320 --> 00:52:24.180 align:middle line:84%
So you should also plot the
accuracy to see what's going on.

00:52:24.180 --> 00:52:26.100 align:middle line:84%
And really, you should
look at the accuracy

00:52:26.100 --> 00:52:28.725 align:middle line:84%
and figure out overfitting and
underfitting and all that stuff.

00:52:28.725 --> 00:52:29.740 align:middle line:90%
So let's just do that.

00:52:29.740 --> 00:52:34.300 align:middle line:90%
So I have here overfitting.

00:52:34.300 --> 00:52:36.060 align:middle line:84%
So this is how it looks
like for accuracy.

00:52:36.060 --> 00:52:37.727 align:middle line:84%
Accuracy, of course,
as the model gets--

00:52:37.727 --> 00:52:39.560 align:middle line:84%
as you do more and more
epochs, hopefully it

00:52:39.560 --> 00:52:41.100 align:middle line:84%
gets better and
better for training.

00:52:41.100 --> 00:52:44.380 align:middle line:84%
So you can see here accuracy
actually climbs all the way up

00:52:44.380 --> 00:52:47.270 align:middle line:90%
to the mid 90s right there.

00:52:47.270 --> 00:52:48.630 align:middle line:90%
The low 90s here.

00:52:48.630 --> 00:52:51.190 align:middle line:84%
The validation
gets to this point

00:52:51.190 --> 00:52:53.730 align:middle line:84%
after, I don't know,
50 epochs maybe.

00:52:53.730 --> 00:52:55.190 align:middle line:84%
And then it kind
of flattens out.

00:52:55.190 --> 00:52:57.960 align:middle line:84%
And then, strangely, it
climbs up again a bit later.

00:52:57.960 --> 00:53:00.470 align:middle line:90%


00:53:00.470 --> 00:53:04.030 align:middle line:84%
Now, the fact that the
accuracy actually got better

00:53:04.030 --> 00:53:07.150 align:middle line:84%
at the very end
suggests that maybe we

00:53:07.150 --> 00:53:09.630 align:middle line:90%
can live with this overfitting.

00:53:09.630 --> 00:53:10.670 align:middle line:90%
It's OK.

00:53:10.670 --> 00:53:11.970 align:middle line:90%
It's not the end of the world.

00:53:11.970 --> 00:53:16.332 align:middle line:84%
So you can certainly-- what
you can do is you can go back

00:53:16.332 --> 00:53:17.290 align:middle line:90%
and say, you know what?

00:53:17.290 --> 00:53:19.210 align:middle line:84%
No, I'm going to be
a purist about this.

00:53:19.210 --> 00:53:20.770 align:middle line:90%
Around 50 epochs or so.

00:53:20.770 --> 00:53:23.410 align:middle line:84%
I think that's when it actually
flattened out for loss.

00:53:23.410 --> 00:53:25.790 align:middle line:84%
So you can just go back
and just restart the model

00:53:25.790 --> 00:53:28.303 align:middle line:84%
and run it only for
50 epochs, not 300.

00:53:28.303 --> 00:53:30.470 align:middle line:84%
And then stop and just use
that model for everything

00:53:30.470 --> 00:53:31.520 align:middle line:90%
from that point on.

00:53:31.520 --> 00:53:32.770 align:middle line:90%
Or you can say, you know what?

00:53:32.770 --> 00:53:33.270 align:middle line:90%
It's OK.

00:53:33.270 --> 00:53:34.500 align:middle line:90%
I can live with this thing.

00:53:34.500 --> 00:53:36.250 align:middle line:84%
And so that's what
we're going to do here.

00:53:36.250 --> 00:53:37.500 align:middle line:90%
Let me just stop for a second.

00:53:37.500 --> 00:53:39.790 align:middle line:90%
There was the question.

00:53:39.790 --> 00:53:40.370 align:middle line:90%
Yeah.

00:53:40.370 --> 00:53:42.370 align:middle line:84%
AUDIENCE: Originally,
when we were starting out,

00:53:42.370 --> 00:53:45.520 align:middle line:84%
we were saying 20 to 30 epochs,
but we're going to be doing 300.

00:53:45.520 --> 00:53:47.580 align:middle line:90%
50 is over 20 to 30.

00:53:47.580 --> 00:53:50.480 align:middle line:84%
So when it comes to validation
of if you've run enough epochs,

00:53:50.480 --> 00:53:52.620 align:middle line:84%
are you doing a
derivative calculation?

00:53:52.620 --> 00:53:53.180 align:middle line:90%
RAMA RAMAKRISHNAN: Oh, I see.

00:53:53.180 --> 00:53:54.440 align:middle line:90%
No, that's a great question.

00:53:54.440 --> 00:53:57.128 align:middle line:84%
So the question is, I said
start with 20 and 30 epochs

00:53:57.128 --> 00:53:57.920 align:middle line:90%
as a rule of thumb.

00:53:57.920 --> 00:53:59.345 align:middle line:90%
Here, I'm just going with 300.

00:53:59.345 --> 00:54:00.720 align:middle line:84%
And because I'm
going with 300, I

00:54:00.720 --> 00:54:03.100 align:middle line:84%
can actually see some potential
evidence of overfitting.

00:54:03.100 --> 00:54:04.812 align:middle line:84%
But if I had done
only 20 to 30, maybe I

00:54:04.812 --> 00:54:06.020 align:middle line:90%
wouldn't have even seen that.

00:54:06.020 --> 00:54:06.860 align:middle line:90%
What happens next?

00:54:06.860 --> 00:54:07.740 align:middle line:90%
Is that the question?

00:54:07.740 --> 00:54:08.700 align:middle line:90%
Great question.

00:54:08.700 --> 00:54:11.440 align:middle line:84%
So what you should do is when
you look at these curves,

00:54:11.440 --> 00:54:14.600 align:middle line:84%
if, at the end of 30 epochs,
you find that the validation

00:54:14.600 --> 00:54:17.960 align:middle line:84%
loss continues to drop,
then you know maybe there

00:54:17.960 --> 00:54:19.540 align:middle line:90%
is more room for it to drop.

00:54:19.540 --> 00:54:21.420 align:middle line:84%
So you continue
from that point on.

00:54:21.420 --> 00:54:23.560 align:middle line:84%
The thing about Keras
is that you can actually

00:54:23.560 --> 00:54:27.260 align:middle line:84%
run the fit command
at that point,

00:54:27.260 --> 00:54:29.020 align:middle line:84%
and it will continue
where it left off.

00:54:29.020 --> 00:54:31.680 align:middle line:84%
It won't go to the
beginning again.

00:54:31.680 --> 00:54:32.960 align:middle line:90%
So you can run 10.

00:54:32.960 --> 00:54:33.460 align:middle line:90%
OK.

00:54:33.460 --> 00:54:35.220 align:middle line:84%
The validation is still
getting better and better.

00:54:35.220 --> 00:54:35.400 align:middle line:90%
OK.

00:54:35.400 --> 00:54:36.340 align:middle line:90%
Run for another 10.

00:54:36.340 --> 00:54:37.800 align:middle line:90%
It's getting better and better.

00:54:37.800 --> 00:54:38.620 align:middle line:90%
Run for another 10.

00:54:38.620 --> 00:54:39.380 align:middle line:90%
Getting better and better.

00:54:39.380 --> 00:54:40.140 align:middle line:90%
Run for another 10.

00:54:40.140 --> 00:54:41.100 align:middle line:90%
Oh, it starts to climb up again.

00:54:41.100 --> 00:54:42.350 align:middle line:90%
OK, now I'm going to back off.

00:54:42.350 --> 00:54:44.730 align:middle line:90%
That's what you do.

00:54:44.730 --> 00:54:45.470 align:middle line:90%
All right.

00:54:45.470 --> 00:54:48.610 align:middle line:84%
Now, all this manual stuff, I'm
going through it just because--

00:54:48.610 --> 00:54:49.985 align:middle line:90%
to build intuition.

00:54:49.985 --> 00:54:51.610 align:middle line:84%
There are these things
called callbacks

00:54:51.610 --> 00:54:54.170 align:middle line:84%
in Keras, which we'll get to
later on, in which you can

00:54:54.170 --> 00:54:59.310 align:middle line:84%
actually tell it, hey, when the
validation loss stops improving,

00:54:59.310 --> 00:55:01.990 align:middle line:84%
stop everything, or
when it stops improving,

00:55:01.990 --> 00:55:04.365 align:middle line:84%
save that model
for me somewhere.

00:55:04.365 --> 00:55:06.490 align:middle line:84%
So they don't have to go
back and rerun everything.

00:55:06.490 --> 00:55:08.448 align:middle line:84%
It will have saved it
for you, and you can just

00:55:08.448 --> 00:55:09.530 align:middle line:90%
pick it up and use it.

00:55:09.530 --> 00:55:12.290 align:middle line:90%
Yeah.

00:55:12.290 --> 00:55:16.490 align:middle line:84%
AUDIENCE: What's the intuition
behind the accuracy continuing

00:55:16.490 --> 00:55:19.135 align:middle line:84%
to improve when the
loss is getting higher?

00:55:19.135 --> 00:55:21.510 align:middle line:84%
RAMA RAMAKRISHNAN: Because
accuracy and loss are related,

00:55:21.510 --> 00:55:24.310 align:middle line:90%
but they're not the same thing.

00:55:24.310 --> 00:55:26.030 align:middle line:90%
So it's a really good question.

00:55:26.030 --> 00:55:27.690 align:middle line:84%
It's also kind of
a profound question

00:55:27.690 --> 00:55:29.953 align:middle line:84%
because accuracy is a
very discrete measure.

00:55:29.953 --> 00:55:32.370 align:middle line:84%
So if a particular point, we're
predicting its probability

00:55:32.370 --> 00:55:35.270 align:middle line:84%
to be, say, 0.49,
we're going to say, OK,

00:55:35.270 --> 00:55:37.610 align:middle line:90%
that's a 0, no heart disease.

00:55:37.610 --> 00:55:39.290 align:middle line:84%
But if it goes to
0.51, we're going

00:55:39.290 --> 00:55:40.930 align:middle line:84%
to be like, oh,
that's heart disease.

00:55:40.930 --> 00:55:43.810 align:middle line:84%
So when you go
from 0.49 to 0.51,

00:55:43.810 --> 00:55:46.950 align:middle line:84%
the binary cross-entropy loss
will change very, very slightly.

00:55:46.950 --> 00:55:49.270 align:middle line:84%
Well, the accuracy
will go from 0 to 1.

00:55:49.270 --> 00:55:50.290 align:middle line:90%
Dramatic jump.

00:55:50.290 --> 00:55:52.230 align:middle line:90%
So it's very jumpy and discrete.

00:55:52.230 --> 00:55:54.310 align:middle line:84%
And that's why it
tends to be a proxy

00:55:54.310 --> 00:55:57.082 align:middle line:90%
but a crude proxy for loss.

00:55:57.082 --> 00:55:58.290 align:middle line:90%
So that's part of the reason.

00:55:58.290 --> 00:56:01.183 align:middle line:90%
And I can talk more offline.

00:56:01.183 --> 00:56:01.683 align:middle line:90%
OK.

00:56:01.683 --> 00:56:04.550 align:middle line:90%
So-- yeah.

00:56:04.550 --> 00:56:06.610 align:middle line:84%
AUDIENCE: You mentioned
that if you are curious,

00:56:06.610 --> 00:56:10.350 align:middle line:84%
you could stop at
epoch 50 in this case

00:56:10.350 --> 00:56:11.930 align:middle line:90%
and run it and stop it there.

00:56:11.930 --> 00:56:14.730 align:middle line:84%
I was wondering, if you could
see the history of the model,

00:56:14.730 --> 00:56:17.750 align:middle line:84%
take the weights at epoch 50,
and include it in your model,

00:56:17.750 --> 00:56:20.510 align:middle line:84%
if it would be roughly
the same, or it

00:56:20.510 --> 00:56:22.130 align:middle line:90%
would be certain differences?

00:56:22.130 --> 00:56:23.630 align:middle line:84%
RAMA RAMAKRISHNAN:
You could try it.

00:56:23.630 --> 00:56:24.610 align:middle line:90%
Yeah, you should just try it.

00:56:24.610 --> 00:56:27.110 align:middle line:84%
Because what happens is that,
ultimately, what we care about

00:56:27.110 --> 00:56:29.213 align:middle line:84%
is how it performs on
the validation set.

00:56:29.213 --> 00:56:30.630 align:middle line:84%
Here, it appears
to perform better

00:56:30.630 --> 00:56:33.488 align:middle line:84%
on the validation set if
you stop at 50, but only

00:56:33.488 --> 00:56:34.030 align:middle line:90%
for the loss.

00:56:34.030 --> 00:56:36.280 align:middle line:84%
For accuracy, actually, if
you wait till the very end,

00:56:36.280 --> 00:56:37.790 align:middle line:90%
it gets better.

00:56:37.790 --> 00:56:40.310 align:middle line:84%
So my thrust tends
to be, what is

00:56:40.310 --> 00:56:44.080 align:middle line:84%
the measure that's closest
to the real-world deployment?

00:56:44.080 --> 00:56:44.940 align:middle line:90%
It's accuracy.

00:56:44.940 --> 00:56:48.640 align:middle line:90%
So I tend to go with accuracy.

00:56:48.640 --> 00:56:51.440 align:middle line:84%
Binary cross-entropy
is a beautiful proxy,

00:56:51.440 --> 00:56:53.442 align:middle line:84%
but an imperfect
proxy for the thing

00:56:53.442 --> 00:56:55.400 align:middle line:84%
we actually care about
in the real world, which

00:56:55.400 --> 00:56:57.440 align:middle line:90%
is error rate and accuracy.

00:56:57.440 --> 00:56:58.993 align:middle line:90%
That's why I tend to want both.

00:56:58.993 --> 00:57:00.660 align:middle line:84%
And if accuracy is
telling me one thing,

00:57:00.660 --> 00:57:03.040 align:middle line:90%
I kind of tend to believe that.

00:57:03.040 --> 00:57:03.540 align:middle line:90%
All right.

00:57:03.540 --> 00:57:08.000 align:middle line:90%
So here is what we have.

00:57:08.000 --> 00:57:09.740 align:middle line:84%
So once we do all
this we have a model.

00:57:09.740 --> 00:57:11.572 align:middle line:84%
And we know we want to
evaluate to see, OK,

00:57:11.572 --> 00:57:13.780 align:middle line:84%
if we actually deploy it,
how good is it going to be?

00:57:13.780 --> 00:57:16.600 align:middle line:84%
So you use this thing called
the model.evaluate function.

00:57:16.600 --> 00:57:18.540 align:middle line:84%
So you take the
model.evaluate function.

00:57:18.540 --> 00:57:21.000 align:middle line:84%
Now you use the
test and the test_x

00:57:21.000 --> 00:57:22.505 align:middle line:84%
and the test_y
data set, which we

00:57:22.505 --> 00:57:23.880 align:middle line:84%
split at the very,
very beginning

00:57:23.880 --> 00:57:25.840 align:middle line:84%
and never used
from that point on.

00:57:25.840 --> 00:57:26.460 align:middle line:90%
We run it.

00:57:26.460 --> 00:57:29.240 align:middle line:84%
And when I ran it
last night, it came up

00:57:29.240 --> 00:57:32.680 align:middle line:84%
with an 83.6% accuracy
for the model.

00:57:32.680 --> 00:57:34.740 align:middle line:84%
And remember, our
baseline model,

00:57:34.740 --> 00:57:36.380 align:middle line:84%
which just predicts
everybody's a 0,

00:57:36.380 --> 00:57:38.663 align:middle line:84%
is going to have
a 72.6% accuracy.

00:57:38.663 --> 00:57:40.330 align:middle line:84%
And this little neural
network gives you

00:57:40.330 --> 00:57:43.270 align:middle line:90%
83.6%, which is pretty good.

00:57:43.270 --> 00:57:46.290 align:middle line:84%
So it's actually--
phew, it's beating

00:57:46.290 --> 00:57:48.847 align:middle line:84%
the model, the baseline
model, which is nice.

00:57:48.847 --> 00:57:50.930 align:middle line:84%
And I guess there is
something here about the fact

00:57:50.930 --> 00:57:53.493 align:middle line:84%
that we did a bunch of
preprocessing outside Keras,

00:57:53.493 --> 00:57:54.910 align:middle line:84%
and then we send
stuff into Keras.

00:57:54.910 --> 00:57:57.202 align:middle line:84%
You can actually do all this
preprocessing inside Keras

00:57:57.202 --> 00:57:57.790 align:middle line:90%
automatically.

00:57:57.790 --> 00:57:59.040 align:middle line:90%
And there are layers for that.

00:57:59.040 --> 00:58:00.850 align:middle line:84%
And I have linked to
a bunch of stuff here.

00:58:00.850 --> 00:58:03.127 align:middle line:84%
So that's it as far as
this model is concerned.

00:58:03.127 --> 00:58:04.710 align:middle line:84%
I know we went through
it really fast,

00:58:04.710 --> 00:58:06.570 align:middle line:84%
but please go
through it afterwards

00:58:06.570 --> 00:58:08.590 align:middle line:84%
and make sure you understand
every single line.

00:58:08.590 --> 00:58:10.585 align:middle line:84%
Change each of these
lines, rerun it,

00:58:10.585 --> 00:58:11.710 align:middle line:90%
see how the output changes.

00:58:11.710 --> 00:58:13.410 align:middle line:84%
That's how you build
some intuition.

00:58:13.410 --> 00:58:14.690 align:middle line:90%
OK.

00:58:14.690 --> 00:58:15.190 align:middle line:90%
All right.

00:58:15.190 --> 00:58:17.970 align:middle line:90%
Computer vision.

00:58:17.970 --> 00:58:20.170 align:middle line:90%
As I do.

00:58:20.170 --> 00:58:22.010 align:middle line:90%
AUDIENCE: Just one question.

00:58:22.010 --> 00:58:23.730 align:middle line:84%
Is there a way to
build the model just

00:58:23.730 --> 00:58:26.490 align:middle line:84%
to have less false positives
or less false negative?

00:58:26.490 --> 00:58:27.670 align:middle line:90%
Or we don't know that?

00:58:27.670 --> 00:58:29.545 align:middle line:84%
RAMA RAMAKRISHNAN: Oh,
yeah, you can do that.

00:58:29.545 --> 00:58:31.770 align:middle line:84%
But there are-- so you can
report on all those things

00:58:31.770 --> 00:58:32.710 align:middle line:90%
very easily.

00:58:32.710 --> 00:58:35.010 align:middle line:84%
But there are more
complex loss functions

00:58:35.010 --> 00:58:38.860 align:middle line:84%
which will take the asymmetry
between the false positive/false

00:58:38.860 --> 00:58:41.320 align:middle line:90%
negative into account.

00:58:41.320 --> 00:58:41.820 align:middle line:90%
Yeah.

00:58:41.820 --> 00:58:43.540 align:middle line:84%
So the short answer--
it's possible.

00:58:43.540 --> 00:58:45.140 align:middle line:90%
Yeah.

00:58:45.140 --> 00:58:45.680 align:middle line:90%
All right.

00:58:45.680 --> 00:58:47.180 align:middle line:84%
So first, let's
just talk about, how

00:58:47.180 --> 00:58:50.540 align:middle line:84%
do you represent
an image digitally?

00:58:50.540 --> 00:58:54.100 align:middle line:84%
And so these are our
grayscale images represented,

00:58:54.100 --> 00:58:55.280 align:middle line:90%
black and white images.

00:58:55.280 --> 00:58:57.040 align:middle line:84%
So the basic idea
is very simple.

00:58:57.040 --> 00:59:00.020 align:middle line:84%
Every picture you have, every
location in that picture

00:59:00.020 --> 00:59:01.000 align:middle line:90%
is a pixel.

00:59:01.000 --> 00:59:04.020 align:middle line:84%
And a pixel basically has a
light intensity-- the amount

00:59:04.020 --> 00:59:05.780 align:middle line:90%
of light at that location.

00:59:05.780 --> 00:59:11.300 align:middle line:84%
And that light level is measured
from 0, no light, to blinding

00:59:11.300 --> 00:59:13.100 align:middle line:90%
white light, which is 255.

00:59:13.100 --> 00:59:15.780 align:middle line:90%
And so all the numbers here--

00:59:15.780 --> 00:59:20.100 align:middle line:84%
if you take this 5, for example,
you can see a lot of no light.

00:59:20.100 --> 00:59:23.323 align:middle line:84%
All the black regions,
those are all zeros.

00:59:23.323 --> 00:59:25.740 align:middle line:84%
And then wherever there is
white light, there is a number.

00:59:25.740 --> 00:59:29.197 align:middle line:84%
And more the amount of light,
the closer it gets to 255.

00:59:29.197 --> 00:59:31.280 align:middle line:84%
In fact, if you just step
back and squint at this,

00:59:31.280 --> 00:59:34.258 align:middle line:90%
you can actually see the 5.

00:59:34.258 --> 00:59:34.800 align:middle line:90%
So that's it.

00:59:34.800 --> 00:59:37.460 align:middle line:84%
That's how black and white
images are represented.

00:59:37.460 --> 00:59:39.013 align:middle line:90%
Very simple.

00:59:39.013 --> 00:59:40.680 align:middle line:90%
Now-- yeah.

00:59:40.680 --> 00:59:42.240 align:middle line:84%
AUDIENCE: Can you
see [INAUDIBLE]--

00:59:42.240 --> 00:59:43.880 align:middle line:84%
RAMA RAMAKRISHNAN:
Microphone, please.

00:59:43.880 --> 00:59:45.880 align:middle line:84%
AUDIENCE: Just when you
say amount of light, what's

00:59:45.880 --> 00:59:47.172 align:middle line:90%
the unit that's being measured?

00:59:47.172 --> 00:59:48.260 align:middle line:90%
What do you mean?

00:59:48.260 --> 00:59:50.135 align:middle line:84%
RAMA RAMAKRISHNAN: So
here, basically what we

00:59:50.135 --> 00:59:54.120 align:middle line:84%
have is the computer
takes whatever--

00:59:54.120 --> 00:59:56.867 align:middle line:84%
so when you send an analog--
you take an analog picture.

00:59:56.867 --> 00:59:59.200 align:middle line:84%
There is a process by which
you take that analog picture

00:59:59.200 --> 01:00:01.920 align:middle line:84%
and read it in, and it
gets mapped to a scale

01:00:01.920 --> 01:00:03.220 align:middle line:90%
between 0 and 255.

01:00:03.220 --> 01:00:04.080 align:middle line:90%
That's it.

01:00:04.080 --> 01:00:04.580 align:middle line:90%
That's all.

01:00:04.580 --> 01:00:06.580 align:middle line:84%
So you can think of it
as like a relative scale,

01:00:06.580 --> 01:00:08.600 align:middle line:84%
a normalized scale,
between 0 and 255.

01:00:08.600 --> 01:00:11.840 align:middle line:84%
And so it just roughly
maps to the amount of light

01:00:11.840 --> 01:00:12.900 align:middle line:90%
in that location.

01:00:12.900 --> 01:00:15.413 align:middle line:84%
The exact lumens to
the number mapping,

01:00:15.413 --> 01:00:16.580 align:middle line:90%
I don't know how they do it.

01:00:16.580 --> 01:00:19.480 align:middle line:84%
My guess is there are a dizzying
number of variations on that.

01:00:19.480 --> 01:00:21.680 align:middle line:84%
But for our purposes,
just think of it

01:00:21.680 --> 01:00:26.200 align:middle line:84%
as, it's a normalized scale
which runs from 0 to 255.

01:00:26.200 --> 01:00:26.700 align:middle line:90%
All right.

01:00:26.700 --> 01:00:28.780 align:middle line:90%
So if you look at--

01:00:28.780 --> 01:00:29.960 align:middle line:90%
so that's what's happening.

01:00:29.960 --> 01:00:32.160 align:middle line:84%
Every is a number
between 0 to 255.

01:00:32.160 --> 01:00:33.240 align:middle line:90%
Boom, boom, boom.

01:00:33.240 --> 01:00:37.810 align:middle line:84%
So if you have a color image,
each pixel of a color image

01:00:37.810 --> 01:00:41.730 align:middle line:84%
is represented by three
numbers, and these numbers

01:00:41.730 --> 01:00:44.270 align:middle line:84%
measure the intensity of
red light, blue light,

01:00:44.270 --> 01:00:45.055 align:middle line:90%
and green light.

01:00:45.055 --> 01:00:46.930 align:middle line:84%
Because red, blue, and
green, if you mix them

01:00:46.930 --> 01:00:50.530 align:middle line:84%
in the right proportion, you
can get whatever you want.

01:00:50.530 --> 01:00:52.410 align:middle line:84%
And so each light
density is still

01:00:52.410 --> 01:00:55.188 align:middle line:84%
a number between 0 and 255,
and that's what you have,

01:00:55.188 --> 01:00:57.730 align:middle line:84%
which means that now you have
three tables of numbers instead

01:00:57.730 --> 01:00:59.130 align:middle line:90%
of one table of numbers.

01:00:59.130 --> 01:01:01.370 align:middle line:84%
And by the way, just
some lingo here,

01:01:01.370 --> 01:01:05.310 align:middle line:84%
in the deep-learning world,
these colors, RGB, Red, Blue,

01:01:05.310 --> 01:01:09.010 align:middle line:84%
Green, are sometimes
referred to as channels.

01:01:09.010 --> 01:01:09.630 align:middle line:90%
All right.

01:01:09.630 --> 01:01:10.870 align:middle line:90%
So this is what we have here.

01:01:10.870 --> 01:01:12.530 align:middle line:84%
This is a picture
of Killian Court.

01:01:12.530 --> 01:01:14.113 align:middle line:84%
And then if you take
that little thing

01:01:14.113 --> 01:01:18.250 align:middle line:84%
here, the red table, the green
table, and the blue table.

01:01:18.250 --> 01:01:23.730 align:middle line:84%
So for this picture, these three
tables is a tensor of rank what?

01:01:23.730 --> 01:01:26.090 align:middle line:90%
Good.

01:01:26.090 --> 01:01:26.790 align:middle line:90%
All right.

01:01:26.790 --> 01:01:27.870 align:middle line:90%
Any questions on this?

01:01:27.870 --> 01:01:33.512 align:middle line:90%


01:01:33.512 --> 01:01:35.740 align:middle line:84%
So the key task in
computer vision--

01:01:35.740 --> 01:01:39.700 align:middle line:84%
obviously, the important
thing is image classification.

01:01:39.700 --> 01:01:41.613 align:middle line:84%
The most basic
task, if you will,

01:01:41.613 --> 01:01:43.780 align:middle line:84%
when you're working with
images is you have an image

01:01:43.780 --> 01:01:45.740 align:middle line:84%
and you want to take
whatever-- you take the image

01:01:45.740 --> 01:01:47.780 align:middle line:84%
and figure out, OK, you have
a list of possible objects

01:01:47.780 --> 01:01:48.960 align:middle line:90%
the image could contain.

01:01:48.960 --> 01:01:51.460 align:middle line:84%
And you're figuring out, OK,
which of these possible objects

01:01:51.460 --> 01:01:52.460 align:middle line:90%
exists in that image?

01:01:52.460 --> 01:01:55.460 align:middle line:84%
The Dog/Cat classification
is the canonical example

01:01:55.460 --> 01:01:57.820 align:middle line:90%
that we all know and love.

01:01:57.820 --> 01:02:01.040 align:middle line:84%
And that's what we will solve
later today and on Wednesday.

01:02:01.040 --> 01:02:03.620 align:middle line:84%
But there are many other tasks
that you need to be aware of.

01:02:03.620 --> 01:02:06.640 align:middle line:84%
So then you actually not
just classify an image,

01:02:06.640 --> 01:02:10.583 align:middle line:84%
but you also localize
where in the image is it.

01:02:10.583 --> 01:02:12.000 align:middle line:84%
It's not just
enough to say sheep.

01:02:12.000 --> 01:02:14.060 align:middle line:84%
You want to figure out,
where is the sheep?

01:02:14.060 --> 01:02:15.437 align:middle line:90%
And that's called localization.

01:02:15.437 --> 01:02:17.020 align:middle line:84%
And the way you do
localization is you

01:02:17.020 --> 01:02:19.240 align:middle line:90%
put this little box around it.

01:02:19.240 --> 01:02:23.740 align:middle line:84%
And then you output not just
whether it's a sheep, yes or no,

01:02:23.740 --> 01:02:25.800 align:middle line:90%
but the coordinates of this box.

01:02:25.800 --> 01:02:28.147 align:middle line:84%
The top left and the
bottom right, for example.

01:02:28.147 --> 01:02:29.980 align:middle line:84%
If you put the coordinates,
you can actually

01:02:29.980 --> 01:02:31.280 align:middle line:90%
draw a box around it.

01:02:31.280 --> 01:02:34.030 align:middle line:84%
So you output the
numbers, the coordinates,

01:02:34.030 --> 01:02:37.510 align:middle line:84%
of where this box
is in the picture.

01:02:37.510 --> 01:02:39.670 align:middle line:90%
This is called localization.

01:02:39.670 --> 01:02:42.470 align:middle line:84%
Now, this is object
detection where

01:02:42.470 --> 01:02:44.890 align:middle line:84%
you may have lots
of objects going on,

01:02:44.890 --> 01:02:47.070 align:middle line:84%
and you want to pick
up every one of them,

01:02:47.070 --> 01:02:49.928 align:middle line:90%
and you want to localize it.

01:02:49.928 --> 01:02:50.970 align:middle line:90%
This is object detection.

01:02:50.970 --> 01:02:52.850 align:middle line:84%
So here, we have gone
in there and said,

01:02:52.850 --> 01:02:55.470 align:middle line:90%
OK, sheep 1, sheep 2, sheep 3.

01:02:55.470 --> 01:02:59.510 align:middle line:84%
And each of these sheep
has a little box around it.

01:02:59.510 --> 01:03:03.692 align:middle line:84%
By the way, self-driving
cars, the camera vision system

01:03:03.692 --> 01:03:06.150 align:middle line:84%
is constantly scanning what's
coming in through the cameras

01:03:06.150 --> 01:03:09.070 align:middle line:84%
and doing object detection
constantly, many times a second.

01:03:09.070 --> 01:03:13.390 align:middle line:84%
Pedestrian box, zebra crossing
box, doggy box, stroller box,

01:03:13.390 --> 01:03:16.310 align:middle line:90%
and so on and so forth.

01:03:16.310 --> 01:03:19.070 align:middle line:84%
And then we have this thing
called semantic segmentation,

01:03:19.070 --> 01:03:22.110 align:middle line:84%
where we take every
pixel in the picture

01:03:22.110 --> 01:03:23.623 align:middle line:90%
and classify every pixel.

01:03:23.623 --> 01:03:25.290 align:middle line:84%
We're not classifying
the whole picture.

01:03:25.290 --> 01:03:26.790 align:middle line:90%
We're classifying every pixel.

01:03:26.790 --> 01:03:31.050 align:middle line:84%
So we are saying, OK, all
these gray pixels, road.

01:03:31.050 --> 01:03:34.730 align:middle line:84%
All these pixels are sheep,
and all these pixels are grass.

01:03:34.730 --> 01:03:37.730 align:middle line:90%
Every pixel is being classified.

01:03:37.730 --> 01:03:39.690 align:middle line:84%
So we are taking an
image and, instead

01:03:39.690 --> 01:03:42.550 align:middle line:84%
of giving one classification,
for every pixel,

01:03:42.550 --> 01:03:44.740 align:middle line:84%
we are solving a multi-class
classification problem.

01:03:44.740 --> 01:03:48.630 align:middle line:90%


01:03:48.630 --> 01:03:49.750 align:middle line:90%
Every pixel is classified.

01:03:49.750 --> 01:03:53.250 align:middle line:84%
And just when you think it can't
get more complicated than this,

01:03:53.250 --> 01:03:56.010 align:middle line:84%
we have something called
instance segmentation, where

01:03:56.010 --> 01:03:58.130 align:middle line:84%
not only are we
classifying every pixel,

01:03:58.130 --> 01:04:01.890 align:middle line:84%
we are distinguishing
between the different sheep.

01:04:01.890 --> 01:04:05.130 align:middle line:84%
So every pixel is classified,
and different instances

01:04:05.130 --> 01:04:07.550 align:middle line:84%
of the same category
need to be identified.

01:04:07.550 --> 01:04:10.490 align:middle line:90%


01:04:10.490 --> 01:04:13.250 align:middle line:84%
So these are all
some of the most--

01:04:13.250 --> 01:04:16.230 align:middle line:84%
I won't say popular, most
prevalently and useful--

01:04:16.230 --> 01:04:18.290 align:middle line:84%
most prevalent and
useful categories

01:04:18.290 --> 01:04:19.850 align:middle line:84%
of image-processing
problems that

01:04:19.850 --> 01:04:23.810 align:middle line:84%
are amenable to a
deep-learning system.

01:04:23.810 --> 01:04:24.590 align:middle line:90%
All right.

01:04:24.590 --> 01:04:26.450 align:middle line:84%
So let's go to image
classification.

01:04:26.450 --> 01:04:28.617 align:middle line:84%
And we're going to work
with this application called

01:04:28.617 --> 01:04:29.230 align:middle line:90%
fashion-mnist.

01:04:29.230 --> 01:04:33.060 align:middle line:90%


01:04:33.060 --> 01:04:37.660 align:middle line:84%
So the idea here is that you
have 70,000 images of clothing

01:04:37.660 --> 01:04:42.592 align:middle line:84%
items across 10 categories, like
boots and sweaters and T-shirts.

01:04:42.592 --> 01:04:43.300 align:middle line:90%
You get the idea.

01:04:43.300 --> 01:04:45.260 align:middle line:90%
10 categories of clothing.

01:04:45.260 --> 01:04:48.220 align:middle line:90%
We have 70,000 images like this.

01:04:48.220 --> 01:04:50.260 align:middle line:84%
And then we'll build
a network from scratch

01:04:50.260 --> 01:04:53.720 align:middle line:84%
to classify all these things,
with pretty high accuracy.

01:04:53.720 --> 01:04:55.580 align:middle line:84%
So these classes--
by the way, this

01:04:55.580 --> 01:04:57.160 align:middle line:90%
is a very balanced data set.

01:04:57.160 --> 01:04:59.740 align:middle line:84%
So 10% of the data is
sweaters, 10% is boots,

01:04:59.740 --> 01:05:00.960 align:middle line:90%
and so on and so forth.

01:05:00.960 --> 01:05:04.200 align:middle line:84%
So a naive baseline model
would give you what accuracy?

01:05:04.200 --> 01:05:07.660 align:middle line:90%


01:05:07.660 --> 01:05:08.580 align:middle line:90%
10%.

01:05:08.580 --> 01:05:09.512 align:middle line:90%
Exactly.

01:05:09.512 --> 01:05:11.720 align:middle line:84%
So we need to build something
that's better than 10%.

01:05:11.720 --> 01:05:14.012 align:middle line:84%
And I'm glad to report that
a simple neural network can

01:05:14.012 --> 01:05:18.460 align:middle line:84%
actually get you close to
90% on this [INAUDIBLE].

01:05:18.460 --> 01:05:19.000 align:middle line:90%
All right.

01:05:19.000 --> 01:05:23.380 align:middle line:84%
So this is the simple
network that we have.

01:05:23.380 --> 01:05:28.141 align:middle line:84%
The input in this case
is a 28 by 28 picture.

01:05:28.141 --> 01:05:31.310 align:middle line:90%
It's a 28 by 28 picture.

01:05:31.310 --> 01:05:37.690 align:middle line:84%
And so far, we have been feeding
vectors into our neural network.

01:05:37.690 --> 01:05:39.810 align:middle line:84%
Now we have a picture,
which is 28 by 28.

01:05:39.810 --> 01:05:42.430 align:middle line:90%
It's a tensor of rank 2.

01:05:42.430 --> 01:05:44.707 align:middle line:90%
It's a table of numbers.

01:05:44.707 --> 01:05:45.290 align:middle line:90%
What do we do?

01:05:45.290 --> 01:05:46.530 align:middle line:90%
How do we feed that in?

01:05:46.530 --> 01:05:51.230 align:middle line:90%


01:05:51.230 --> 01:05:52.350 align:middle line:90%
It's a-- no.

01:05:52.350 --> 01:05:54.170 align:middle line:84%
Each image is a
table of numbers.

01:05:54.170 --> 01:05:57.590 align:middle line:90%
Let's just take a single image.

01:05:57.590 --> 01:05:58.970 align:middle line:90%
What do we do?

01:05:58.970 --> 01:06:01.710 align:middle line:90%
What do we do with this table?

01:06:01.710 --> 01:06:03.290 align:middle line:90%
Convert it into a vector.

01:06:03.290 --> 01:06:03.790 align:middle line:90%
Exactly.

01:06:03.790 --> 01:06:05.775 align:middle line:90%
And that's called flattening.

01:06:05.775 --> 01:06:07.150 align:middle line:84%
So we take this
table of numbers,

01:06:07.150 --> 01:06:09.230 align:middle line:90%
and we flatten it into a vector.

01:06:09.230 --> 01:06:13.510 align:middle line:84%
And so what we do
is-- let me just--

01:06:13.510 --> 01:06:14.550 align:middle line:90%
OK.

01:06:14.550 --> 01:06:20.230 align:middle line:90%
So we have 28 by 28.

01:06:20.230 --> 01:06:24.310 align:middle line:84%
So what we can do is
we can take each row.

01:06:24.310 --> 01:06:27.960 align:middle line:84%
We can take this row and
then write it like that.

01:06:27.960 --> 01:06:29.340 align:middle line:90%
We take the second row.

01:06:29.340 --> 01:06:29.840 align:middle line:90%
Oops.

01:06:29.840 --> 01:06:32.917 align:middle line:90%


01:06:32.917 --> 01:06:34.250 align:middle line:90%
I'm going to write it like that.

01:06:34.250 --> 01:06:38.120 align:middle line:90%


01:06:38.120 --> 01:06:40.600 align:middle line:90%
Third row is here.

01:06:40.600 --> 01:06:42.180 align:middle line:90%
I'm going to write it like that.

01:06:42.180 --> 01:06:43.160 align:middle line:90%
You get the idea.

01:06:43.160 --> 01:06:45.620 align:middle line:84%
So you take each row, just
rotate it, and stack it all up.

01:06:45.620 --> 01:06:46.860 align:middle line:90%
And string them up.

01:06:46.860 --> 01:06:48.500 align:middle line:90%
It becomes one long vector.

01:06:48.500 --> 01:06:50.120 align:middle line:90%
So this is called flattening.

01:06:50.120 --> 01:06:51.815 align:middle line:84%
So that's how you
take this thing

01:06:51.815 --> 01:06:53.190 align:middle line:84%
and make it into
one long vector.

01:06:53.190 --> 01:06:56.400 align:middle line:90%


01:06:56.400 --> 01:06:59.740 align:middle line:84%
So when you do
that, 28 by 28 is?

01:06:59.740 --> 01:07:00.240 align:middle line:90%
What is it?

01:07:00.240 --> 01:07:03.440 align:middle line:90%


01:07:03.440 --> 01:07:05.000 align:middle line:90%
784.

01:07:05.000 --> 01:07:06.060 align:middle line:90%
So we get 7--

01:07:06.060 --> 01:07:07.400 align:middle line:90%
so we get a vector.

01:07:07.400 --> 01:07:08.900 align:middle line:90%
This is the flattened input.

01:07:08.900 --> 01:07:11.480 align:middle line:90%
And you get 784.

01:07:11.480 --> 01:07:16.790 align:middle line:84%
It's a vector that's 784
long after the flattening.

01:07:16.790 --> 01:07:18.540 align:middle line:84%
We have not done
anything complicated yet.

01:07:18.540 --> 01:07:19.915 align:middle line:84%
We've literally
taken the numbers

01:07:19.915 --> 01:07:22.320 align:middle line:84%
and just reorganized
them in a different way.

01:07:22.320 --> 01:07:24.560 align:middle line:84%
And once we do that,
now we are back

01:07:24.560 --> 01:07:27.140 align:middle line:84%
in our familiar neural
network territory.

01:07:27.140 --> 01:07:28.740 align:middle line:84%
We know how to
work with vectors.

01:07:28.740 --> 01:07:31.980 align:middle line:84%
So we just need to pass
it through a hidden layer.

01:07:31.980 --> 01:07:36.380 align:middle line:84%
And this hidden layer, we're
going to use ReLU neurons.

01:07:36.380 --> 01:07:37.880 align:middle line:84%
And I tried a few
different values.

01:07:37.880 --> 01:07:42.340 align:middle line:84%
And it turns out that 256
neurons does a really good job.

01:07:42.340 --> 01:07:45.100 align:middle line:84%
And so I'm going to
use 256 neurons here.

01:07:45.100 --> 01:07:46.940 align:middle line:84%
And then we need to
now think about what

01:07:46.940 --> 01:07:49.940 align:middle line:90%
the output layer should be.

01:07:49.940 --> 01:07:53.348 align:middle line:84%
Now we run into a problem
because the output layer--

01:07:53.348 --> 01:07:55.140 align:middle line:84%
before we saw for the
heart disease example

01:07:55.140 --> 01:07:56.440 align:middle line:90%
it's just 0 or 1.

01:07:56.440 --> 01:07:59.620 align:middle line:84%
Here, there are 10
possible outputs.

01:07:59.620 --> 01:08:02.940 align:middle line:84%
It could be a boot, a sweater,
a shirt, and so on and so forth.

01:08:02.940 --> 01:08:04.340 align:middle line:90%
10 possible categories.

01:08:04.340 --> 01:08:06.300 align:middle line:84%
So we need some way
to handle something

01:08:06.300 --> 01:08:11.660 align:middle line:84%
with more than one binary
output, many possible outputs.

01:08:11.660 --> 01:08:15.380 align:middle line:90%
So the way we do that--

01:08:15.380 --> 01:08:18.060 align:middle line:84%
by the way, pay attention to
this, because this is actually

01:08:18.060 --> 01:08:19.979 align:middle line:90%
how GPT-4 works.

01:08:19.979 --> 01:08:24.670 align:middle line:84%
So what we do is,
here's what we have.

01:08:24.670 --> 01:08:27.310 align:middle line:84%
We know how to
output 10 numbers.

01:08:27.310 --> 01:08:29.670 align:middle line:84%
If you want to output
10 numbers, no problem.

01:08:29.670 --> 01:08:31.430 align:middle line:84%
We can easily output
10 numbers by just

01:08:31.430 --> 01:08:32.870 align:middle line:90%
using a linear activation.

01:08:32.870 --> 01:08:36.630 align:middle line:84%
We also know how to
output 10 probabilities.

01:08:36.630 --> 01:08:38.490 align:middle line:84%
Each one just needs
to be a sigmoid.

01:08:38.490 --> 01:08:41.609 align:middle line:84%
But here, we can't use 10
sigmoids as the output.

01:08:41.609 --> 01:08:44.031 align:middle line:90%
Why is that?

01:08:44.031 --> 01:08:45.239 align:middle line:90%
Why can't we use 10 sigmoids?

01:08:45.239 --> 01:08:47.640 align:middle line:90%


01:08:47.640 --> 01:08:49.890 align:middle line:84%
AUDIENCE: Because the
probabilities wouldn't sum to 1.

01:08:49.890 --> 01:08:51.015 align:middle line:90%
RAMA RAMAKRISHNAN: Correct.

01:08:51.015 --> 01:08:53.330 align:middle line:84%
So here, when the output
comes, we need to figure out,

01:08:53.330 --> 01:08:56.210 align:middle line:84%
OK, is it a boot, a sweater, a
shirt, and so on and so forth?

01:08:56.210 --> 01:08:58.832 align:middle line:90%
There is only one right answer.

01:08:58.832 --> 01:09:00.790 align:middle line:84%
Which means that we need
to actually figure out

01:09:00.790 --> 01:09:03.415 align:middle line:84%
which of these 10 is the right
answer, which means that we need

01:09:03.415 --> 01:09:05.149 align:middle line:84%
to produce
probabilities, but they

01:09:05.149 --> 01:09:09.350 align:middle line:84%
have to add up to 1 because
only one of them can be true.

01:09:09.350 --> 01:09:10.350 align:middle line:90%
So that's the key thing.

01:09:10.350 --> 01:09:11.290 align:middle line:90%
They have to add up to 1.

01:09:11.290 --> 01:09:12.082 align:middle line:90%
That's the wrinkle.

01:09:12.082 --> 01:09:14.750 align:middle line:84%
If not for that, we can
just use 10 sigmoids.

01:09:14.750 --> 01:09:17.189 align:middle line:84%
And the way we do that
is using something

01:09:17.189 --> 01:09:20.029 align:middle line:84%
called the softmax function
or the softmax layer.

01:09:20.029 --> 01:09:22.029 align:middle line:84%
And the idea is
actually very simple.

01:09:22.029 --> 01:09:25.600 align:middle line:84%
We have these 10 outputs in
the very final layer, which

01:09:25.600 --> 01:09:27.720 align:middle line:90%
is just linear activations.

01:09:27.720 --> 01:09:30.439 align:middle line:84%
And then we take each
one of these numbers,

01:09:30.439 --> 01:09:33.880 align:middle line:84%
and then run it through
the exponential function,

01:09:33.880 --> 01:09:36.160 align:middle line:90%
and then divide by the total.

01:09:36.160 --> 01:09:38.165 align:middle line:84%
So when you do that,
two things happen.

01:09:38.165 --> 01:09:40.540 align:middle line:84%
The first one is when you take
these numbers and run it--

01:09:40.540 --> 01:09:43.340 align:middle line:84%
say you take a1 and
do e raised to a1,

01:09:43.340 --> 01:09:45.583 align:middle line:90%
you now get a positive number.

01:09:45.583 --> 01:09:47.000 align:middle line:84%
And now you have
a positive number

01:09:47.000 --> 01:09:49.100 align:middle line:84%
divided by the sum of a
bunch of positive numbers.

01:09:49.100 --> 01:09:50.220 align:middle line:84%
And they're all--
you can see here,

01:09:50.220 --> 01:09:52.240 align:middle line:84%
you can confirm visually
that they will add up

01:09:52.240 --> 01:09:54.720 align:middle line:84%
to 1 because you're
literally taking each number,

01:09:54.720 --> 01:09:55.698 align:middle line:90%
dividing by the total.

01:09:55.698 --> 01:09:56.740 align:middle line:90%
So they will add up to 1.

01:09:56.740 --> 01:09:58.880 align:middle line:90%
There's no other option.

01:09:58.880 --> 01:10:00.540 align:middle line:84%
So this is called
the softmax function,

01:10:00.540 --> 01:10:02.680 align:middle line:84%
which means that you can take
any set of 10 numbers that's

01:10:02.680 --> 01:10:05.222 align:middle line:84%
coming out of the network and
convert them into probabilities

01:10:05.222 --> 01:10:07.160 align:middle line:90%
that add up to 1.

01:10:07.160 --> 01:10:09.840 align:middle line:84%
And so, by the way,
the GPT-4 reference,

01:10:09.840 --> 01:10:13.440 align:middle line:84%
when you actually put a prompt
into GPT-4 and it starts giving

01:10:13.440 --> 01:10:17.480 align:middle line:84%
you the output, every
word it's emitting--

01:10:17.480 --> 01:10:19.780 align:middle line:84%
it's actually a token, but
we'll get to that later.

01:10:19.780 --> 01:10:20.900 align:middle line:90%
Imagine it's a word.

01:10:20.900 --> 01:10:27.650 align:middle line:84%
Every word it's emitting, it's
doing a 52,000-way softmax.

01:10:27.650 --> 01:10:31.390 align:middle line:84%
Think of it as every word in the
language is a possible output.

01:10:31.390 --> 01:10:33.730 align:middle line:84%
So it's a vector,
which is 52,000 long,

01:10:33.730 --> 01:10:35.570 align:middle line:90%
but it's actually a softmax.

01:10:35.570 --> 01:10:39.690 align:middle line:84%
And it just picks the most
probable word and emits that.

01:10:39.690 --> 01:10:43.770 align:middle line:84%
So this notion of a softmax
is actually very powerful.

01:10:43.770 --> 01:10:45.890 align:middle line:84%
But we'll come
back to that later.

01:10:45.890 --> 01:10:49.810 align:middle line:84%
So to summarize, if you
have a single number,

01:10:49.810 --> 01:10:53.330 align:middle line:84%
you can use a simple output
layer, a single probability,

01:10:53.330 --> 01:10:54.268 align:middle line:90%
a sigmoid.

01:10:54.268 --> 01:10:55.310 align:middle line:90%
You have lots of numbers.

01:10:55.310 --> 01:10:56.753 align:middle line:84%
Just have a stack
of these things.

01:10:56.753 --> 01:10:58.170 align:middle line:84%
And when you have
a lot of numbers

01:10:58.170 --> 01:11:00.430 align:middle line:84%
that have to add up to 1 that
have to be probabilities,

01:11:00.430 --> 01:11:00.940 align:middle line:90%
use softmax.

01:11:00.940 --> 01:11:03.610 align:middle line:90%


01:11:03.610 --> 01:11:04.450 align:middle line:90%
OK.

01:11:04.450 --> 01:11:05.090 align:middle line:90%
So.

01:11:05.090 --> 01:11:06.423 align:middle line:90%
Yeah?

01:11:06.423 --> 01:11:08.090 align:middle line:84%
AUDIENCE: Why do we
choose probabilities

01:11:08.090 --> 01:11:11.488 align:middle line:84%
instead of just number when we
know it's going to be only 1?

01:11:11.488 --> 01:11:12.530 align:middle line:90%
RAMA RAMAKRISHNAN: Sorry?

01:11:12.530 --> 01:11:14.090 align:middle line:84%
AUDIENCE: When we know
it's only going to be 1?

01:11:14.090 --> 01:11:15.810 align:middle line:84%
RAMA RAMAKRISHNAN: Because you
can't force the network to give

01:11:15.810 --> 01:11:16.660 align:middle line:90%
you 1's or 0's.

01:11:16.660 --> 01:11:19.530 align:middle line:90%


01:11:19.530 --> 01:11:22.670 align:middle line:84%
It's going to produce what
it's going to produce.

01:11:22.670 --> 01:11:26.230 align:middle line:84%
You can't force it
to be exactly 1 or 0.

01:11:26.230 --> 01:11:27.530 align:middle line:90%
It will give you some number.

01:11:27.530 --> 01:11:29.697 align:middle line:84%
The best you can do
is to tame that number

01:11:29.697 --> 01:11:31.530 align:middle line:84%
so that it comes into
a range that you like,

01:11:31.530 --> 01:11:34.670 align:middle line:90%
like between 0 and 1.

01:11:34.670 --> 01:11:39.990 align:middle line:84%
So here, very quickly, we have
a binary classification example

01:11:39.990 --> 01:11:43.010 align:middle line:84%
like yes or no, this is the
one-hot-encoded version, 1 or 0.

01:11:43.010 --> 01:11:45.390 align:middle line:84%
This is what we saw in
the heart disease example.

01:11:45.390 --> 01:11:47.765 align:middle line:84%
When you have something like
this example, fashion-mnist,

01:11:47.765 --> 01:11:50.030 align:middle line:84%
where you have all these
different possibilities,

01:11:50.030 --> 01:11:52.290 align:middle line:84%
then you can encode
it in one of two ways.

01:11:52.290 --> 01:11:55.450 align:middle line:84%
You can encode it just
using integers, like 0 to 9.

01:11:55.450 --> 01:11:58.270 align:middle line:84%
This is called the
sparse encoded version.

01:11:58.270 --> 01:12:01.510 align:middle line:84%
Or you can do a one-hot-encoded
version of the output.

01:12:01.510 --> 01:12:04.630 align:middle line:84%
You can have a one-hot-encoded
version of the output.

01:12:04.630 --> 01:12:08.330 align:middle line:84%
And depending on how your
data comes in to you,

01:12:08.330 --> 01:12:11.850 align:middle line:84%
comes into your Colab,
just pay attention to this.

01:12:11.850 --> 01:12:14.910 align:middle line:84%
And depending on what it is,
you have to pick the right Keras

01:12:14.910 --> 01:12:15.950 align:middle line:90%
loss function.

01:12:15.950 --> 01:12:19.387 align:middle line:84%
So if our data
comes like a 1 and 0

01:12:19.387 --> 01:12:21.720 align:middle line:84%
thing, which is exactly what
we had in the heart disease

01:12:21.720 --> 01:12:23.880 align:middle line:84%
example, we use
binary_crossentropy.

01:12:23.880 --> 01:12:27.500 align:middle line:84%
If your data comes in this
form where it's sparse encoded,

01:12:27.500 --> 01:12:30.600 align:middle line:84%
you use
sparse_categorical_crossentropy.

01:12:30.600 --> 01:12:32.240 align:middle line:84%
And then if it
comes in this form,

01:12:32.240 --> 01:12:34.727 align:middle line:84%
you use
categorical_crossentropy.

01:12:34.727 --> 01:12:36.060 align:middle line:90%
These are all equivalent things.

01:12:36.060 --> 01:12:37.780 align:middle line:84%
It just depends on
the data that you get,

01:12:37.780 --> 01:12:41.092 align:middle line:84%
how it happens to be encoded by
the people who sent it to you.

01:12:41.092 --> 01:12:43.300 align:middle line:84%
And if they send it this
way, use this loss function.

01:12:43.300 --> 01:12:46.060 align:middle line:84%
If they send it that way,
use that loss function.

01:12:46.060 --> 01:12:47.560 align:middle line:84%
Now, as it turns
out, in our example

01:12:47.560 --> 01:12:49.623 align:middle line:84%
here, the data is actually
coming in this form.

01:12:49.623 --> 01:12:51.040 align:middle line:84%
So we'll use this
thing called the

01:12:51.040 --> 01:12:53.360 align:middle line:90%
sparse_categorical_crossentropy.

01:12:53.360 --> 01:12:55.640 align:middle line:84%
And categorical_crossentropy
is a generalization

01:12:55.640 --> 01:12:57.348 align:middle line:84%
of binary_crossentropy,
which I'm not

01:12:57.348 --> 01:12:59.140 align:middle line:84%
going to get into the
mathematical details,

01:12:59.140 --> 01:13:01.660 align:middle line:84%
but the intuition is
basically roughly the same.

01:13:01.660 --> 01:13:04.280 align:middle line:90%


01:13:04.280 --> 01:13:07.080 align:middle line:90%
So this is what we have.

01:13:07.080 --> 01:13:09.680 align:middle line:84%
If this is your output layer,
use mean squared error.

01:13:09.680 --> 01:13:13.182 align:middle line:84%
If this is your output layer,
use binary cross-entropy.

01:13:13.182 --> 01:13:15.140 align:middle line:84%
And if you still have a
stack of these numbers,

01:13:15.140 --> 01:13:16.682 align:middle line:84%
you can still use
mean squared error.

01:13:16.682 --> 01:13:18.850 align:middle line:84%
And if your output
is a softmax, use

01:13:18.850 --> 01:13:21.970 align:middle line:84%
categorical cross-entropy
or sparse categorical

01:13:21.970 --> 01:13:24.450 align:middle line:90%
cross-entropy.

01:13:24.450 --> 01:13:25.090 align:middle line:90%
OK.

01:13:25.090 --> 01:13:26.740 align:middle line:84%
So let's actually
run this in Colab.

01:13:26.740 --> 01:13:31.970 align:middle line:90%


01:13:31.970 --> 01:13:33.550 align:middle line:90%
So this is what we have.

01:13:33.550 --> 01:13:35.370 align:middle line:90%
Can folks see this OK?

01:13:35.370 --> 01:13:36.650 align:middle line:90%
All right.

01:13:36.650 --> 01:13:40.730 align:middle line:84%
So this is the data
set we saw earlier.

01:13:40.730 --> 01:13:41.650 align:middle line:90%
Here.

01:13:41.650 --> 01:13:45.470 align:middle line:84%
As usual, we load
TensorFlow and Keras.

01:13:45.470 --> 01:13:48.330 align:middle line:84%
We load our usual
three packages.

01:13:48.330 --> 01:13:50.830 align:middle line:84%
And then we set the random_seed
for reproducibility.

01:13:50.830 --> 01:13:53.370 align:middle line:84%
And it turns out that the
fashion-mnist data is actually

01:13:53.370 --> 01:13:54.370 align:middle line:90%
available in Keras.

01:13:54.370 --> 01:13:56.310 align:middle line:84%
You don't have to go find it
somewhere and bring it in.

01:13:56.310 --> 01:13:57.390 align:middle line:84%
It's actually
available in Keras.

01:13:57.390 --> 01:13:59.050 align:middle line:84%
It's one of the
standard data sets.

01:13:59.050 --> 01:13:59.790 align:middle line:90%
We luck out.

01:13:59.790 --> 01:14:02.130 align:middle line:84%
So we just actually
load the data

01:14:02.130 --> 01:14:04.190 align:middle line:90%
using this load_data command.

01:14:04.190 --> 01:14:06.690 align:middle line:84%
And then, if you do that,
and conveniently for us,

01:14:06.690 --> 01:14:08.950 align:middle line:84%
Keras has not only made
the data available,

01:14:08.950 --> 01:14:11.568 align:middle line:84%
it has already split it into
a training and test set.

01:14:11.568 --> 01:14:13.110 align:middle line:84%
So we don't have to
do the splitting.

01:14:13.110 --> 01:14:15.530 align:middle line:84%
And the reason they do that--
why would they do that?

01:14:15.530 --> 01:14:18.660 align:middle line:90%


01:14:18.660 --> 01:14:20.247 align:middle line:84%
They do that so that
different people

01:14:20.247 --> 01:14:22.580 align:middle line:84%
who are building algorithms
for that particular data set

01:14:22.580 --> 01:14:26.572 align:middle line:84%
can all be evaluated
using the same test set.

01:14:26.572 --> 01:14:28.280 align:middle line:84%
Otherwise, if I split
it one way and say,

01:14:28.280 --> 01:14:30.363 align:middle line:84%
hey, look how well I did
that, well, I don't know.

01:14:30.363 --> 01:14:31.380 align:middle line:90%
How did you split it?

01:14:31.380 --> 01:14:34.000 align:middle line:90%
That's the reason.

01:14:34.000 --> 01:14:38.220 align:middle line:84%
And you can see
here that we have--

01:14:38.220 --> 01:14:45.860 align:middle line:84%
the input data is
a tensor of rank 3.

01:14:45.860 --> 01:14:48.580 align:middle line:84%
And basically, another way to
think about a tensor of rank 3

01:14:48.580 --> 01:14:50.970 align:middle line:84%
is just a list of
rank 2 tensors.

01:14:50.970 --> 01:14:54.060 align:middle line:90%
So here you have 60,000 images.

01:14:54.060 --> 01:14:55.580 align:middle line:90%
60,000 images.

01:14:55.580 --> 01:15:01.620 align:middle line:84%
And each image is a 28
by 28 square of numbers.

01:15:01.620 --> 01:15:04.060 align:middle line:90%
Each image is a 28 by 28 table.

01:15:04.060 --> 01:15:06.940 align:middle line:84%
And then, of course,
the output is just

01:15:06.940 --> 01:15:10.000 align:middle line:84%
what category it is, a
number between 0 and 9.

01:15:10.000 --> 01:15:11.400 align:middle line:90%
So you just have 60,000 numbers.

01:15:11.400 --> 01:15:13.160 align:middle line:84%
It's just a vector
of 60,000 numbers.

01:15:13.160 --> 01:15:16.220 align:middle line:84%
So there are 60,000
in the training set.

01:15:16.220 --> 01:15:17.280 align:middle line:90%
Oops.

01:15:17.280 --> 01:15:20.020 align:middle line:84%
And then there are
10,000 in the test set.

01:15:20.020 --> 01:15:22.280 align:middle line:90%
Same structure, 28 by 28.

01:15:22.280 --> 01:15:23.180 align:middle line:90%
That's what we have.

01:15:23.180 --> 01:15:24.720 align:middle line:84%
So if you look at
the first 10 rows

01:15:24.720 --> 01:15:28.280 align:middle line:84%
of the dependent variable y, you
get these numbers, 9, 0, 3, 3,

01:15:28.280 --> 01:15:28.780 align:middle line:90%
like that.

01:15:28.780 --> 01:15:30.400 align:middle line:90%
There are numbers from 0 to 9.

01:15:30.400 --> 01:15:33.140 align:middle line:84%
So if you look at the
fashion-mnist GitHub site,

01:15:33.140 --> 01:15:34.320 align:middle line:90%
this is what it refers to.

01:15:34.320 --> 01:15:38.480 align:middle line:84%
0 is a T-shirt, 1 is a trouser,
and so on and so forth.

01:15:38.480 --> 01:15:41.520 align:middle line:90%
And 9 is an ankle boot.

01:15:41.520 --> 01:15:42.520 align:middle line:90%
All right.

01:15:42.520 --> 01:15:45.200 align:middle line:84%
So whenever I'm working with
multi-class classification

01:15:45.200 --> 01:15:47.840 align:middle line:84%
problems, I always
do a little thing

01:15:47.840 --> 01:15:50.902 align:middle line:84%
here to help me figure out that
9 corresponds to an ankle boot

01:15:50.902 --> 01:15:51.860 align:middle line:90%
and so on and so forth.

01:15:51.860 --> 01:15:54.235 align:middle line:84%
It just makes it a little
easier to work with this stuff.

01:15:54.235 --> 01:15:56.560 align:middle line:90%
So I create this little list.

01:15:56.560 --> 01:15:58.930 align:middle line:90%
And then it turns out, if you--

01:15:58.930 --> 01:16:00.760 align:middle line:84%
OK, what is the very
first data point?

01:16:00.760 --> 01:16:01.260 align:middle line:90%
What is it?

01:16:01.260 --> 01:16:02.540 align:middle line:90%
What is its y value?

01:16:02.540 --> 01:16:04.400 align:middle line:84%
It turns out to
be an ankle boot.

01:16:04.400 --> 01:16:07.840 align:middle line:84%
So you can actually look at the
raw data for that image, which

01:16:07.840 --> 01:16:09.780 align:middle line:90%
is just a 28 by 28 thing.

01:16:09.780 --> 01:16:12.701 align:middle line:84%
And these are the
numbers you have.

01:16:12.701 --> 01:16:13.710 align:middle line:90%
You can see all these.

01:16:13.710 --> 01:16:18.142 align:middle line:84%
250, 233, lots of zeros,
and so on and so forth.

01:16:18.142 --> 01:16:20.350 align:middle line:84%
And so you can actually look
at the first-- visualize

01:16:20.350 --> 01:16:21.210 align:middle line:90%
the first 25 images.

01:16:21.210 --> 01:16:23.610 align:middle line:84%
I have a little bit of code
here which visualizes that.

01:16:23.610 --> 01:16:24.722 align:middle line:90%
Just Matplotlib code.

01:16:24.722 --> 01:16:26.430 align:middle line:84%
And you can see these
are all the images.

01:16:26.430 --> 01:16:27.530 align:middle line:90%
They're kind of smallish.

01:16:27.530 --> 01:16:30.000 align:middle line:84%
This, my friends,
is an ankle boot.

01:16:30.000 --> 01:16:32.870 align:middle line:90%


01:16:32.870 --> 01:16:35.210 align:middle line:84%
It's like, OK, can the network
really make any sense out

01:16:35.210 --> 01:16:35.793 align:middle line:90%
of this thing?

01:16:35.793 --> 01:16:37.590 align:middle line:84%
It looks very blurry
and, I don't know.

01:16:37.590 --> 01:16:41.586 align:middle line:90%


01:16:41.586 --> 01:16:43.630 align:middle line:84%
Oh, this is actually
a better ankle boot.

01:16:43.630 --> 01:16:44.230 align:middle line:90%
Look at that.

01:16:44.230 --> 01:16:44.650 align:middle line:90%
OK.

01:16:44.650 --> 01:16:44.990 align:middle line:90%
Sorry.

01:16:44.990 --> 01:16:45.950 align:middle line:90%
I'm getting distracted.

01:16:45.950 --> 01:16:49.810 align:middle line:90%
So this is what we have here.

01:16:49.810 --> 01:16:51.450 align:middle line:90%
OK, we're at 9:55.

01:16:51.450 --> 01:16:53.410 align:middle line:84%
I'm going to stop
so you folks are not

01:16:53.410 --> 01:16:54.470 align:middle line:90%
late for your next class.

01:16:54.470 --> 01:16:56.270 align:middle line:84%
So we'll continue this
journey on Wednesday,

01:16:56.270 --> 01:16:58.410 align:middle line:84%
and then we'll go on to color
images in the next class

01:16:58.410 --> 01:16:58.910 align:middle line:90%
as well.

01:16:58.910 --> 01:16:59.618 align:middle line:90%
Thank you, folks.

01:16:59.618 --> 01:17:01.440 align:middle line:90%
Have a good one.

01:17:01.440 --> 01:17:12.000 align:middle line:90%