WEBVTT

00:00:00.000 --> 00:00:01.964 align:middle line:90%
[SQUEAKING]

00:00:01.964 --> 00:00:03.928 align:middle line:90%
[RUSTLING]

00:00:03.928 --> 00:00:07.856 align:middle line:90%
[CLICKING]

00:00:07.856 --> 00:00:17.020 align:middle line:90%


00:00:17.020 --> 00:00:19.920 align:middle line:84%
RAMA RAMAKRISHNAN: All
right, folks, good morning.

00:00:19.920 --> 00:00:20.740 align:middle line:90%
Welcome back.

00:00:20.740 --> 00:00:23.240 align:middle line:84%
I hope you all
had a nice weekend

00:00:23.240 --> 00:00:26.440 align:middle line:84%
and I hope you had a chance to
watch the video walkthrough I

00:00:26.440 --> 00:00:28.320 align:middle line:90%
posted yesterday.

00:00:28.320 --> 00:00:31.000 align:middle line:84%
It's going to save us some time
today, so let's get right in.

00:00:31.000 --> 00:00:33.360 align:middle line:84%
Today is going to
be super packed.

00:00:33.360 --> 00:00:35.440 align:middle line:84%
You're going to go from
not knowing anything

00:00:35.440 --> 00:00:37.353 align:middle line:84%
about convolutions
perhaps for some of you,

00:00:37.353 --> 00:00:39.520 align:middle line:84%
to actually knowing how
convolutional networks work,

00:00:39.520 --> 00:00:42.200 align:middle line:84%
and actually to build
one, and demo it in class.

00:00:42.200 --> 00:00:44.600 align:middle line:84%
And this demo has actually
worked pretty well

00:00:44.600 --> 00:00:46.860 align:middle line:84%
for the last few years
that I've taught the class.

00:00:46.860 --> 00:00:48.380 align:middle line:84%
But you never know
because it's a live demo.

00:00:48.380 --> 00:00:49.100 align:middle line:90%
It may not work.

00:00:49.100 --> 00:00:51.880 align:middle line:90%
We'll see.

00:00:51.880 --> 00:00:54.720 align:middle line:84%
Valentine's Day gods,
may they be with us.

00:00:54.720 --> 00:00:56.260 align:middle line:90%
OK, so let's get going.

00:00:56.260 --> 00:01:00.070 align:middle line:84%
So Fashion MNIST, we
saw previously, i.e.

00:01:00.070 --> 00:01:02.950 align:middle line:84%
as in, in the walkthrough,
the video walkthrough,

00:01:02.950 --> 00:01:07.110 align:middle line:84%
that a neural network with a
single hidden layer can get us

00:01:07.110 --> 00:01:10.670 align:middle line:90%
to an accuracy in the high 80s.

00:01:10.670 --> 00:01:13.590 align:middle line:84%
And that thing, that
network actually

00:01:13.590 --> 00:01:16.270 align:middle line:84%
didn't know what was
coming in was an image.

00:01:16.270 --> 00:01:19.413 align:middle line:84%
It literally took this table of
numbers, and just took each row,

00:01:19.413 --> 00:01:20.830 align:middle line:84%
and then concatenated
all the rows

00:01:20.830 --> 00:01:23.530 align:middle line:84%
into one giant, long
vector, and then sent it in.

00:01:23.530 --> 00:01:26.230 align:middle line:84%
So the neural network did
exploit the fact that the input

00:01:26.230 --> 00:01:29.950 align:middle line:84%
data was known to be of
a certain type, which

00:01:29.950 --> 00:01:32.750 align:middle line:84%
is the clue for, how
can we do better?

00:01:32.750 --> 00:01:37.150 align:middle line:84%
So let's just spend
a few minutes on,

00:01:37.150 --> 00:01:41.070 align:middle line:84%
what is it about images that we
have to really pay attention to,

00:01:41.070 --> 00:01:43.350 align:middle line:84%
as opposed to any arbitrary
vector of numbers that's

00:01:43.350 --> 00:01:44.390 align:middle line:90%
coming in?

00:01:44.390 --> 00:01:48.590 align:middle line:84%
So when we flatten the
image into a long vector

00:01:48.590 --> 00:01:52.230 align:middle line:84%
and feed it into a dense layer,
several undesirable things

00:01:52.230 --> 00:01:55.133 align:middle line:90%
can actually happen.

00:01:55.133 --> 00:01:56.050 align:middle line:90%
What are some of them.

00:01:56.050 --> 00:01:56.760 align:middle line:90%
Any guesses?

00:01:56.760 --> 00:02:00.630 align:middle line:90%


00:02:00.630 --> 00:02:02.710 align:middle line:90%
Yeah?

00:02:02.710 --> 00:02:04.750 align:middle line:84%
AUDIENCE: Did you lose
the proximity of one

00:02:04.750 --> 00:02:06.583 align:middle line:84%
pixel to other ones
that would be around it?

00:02:06.583 --> 00:02:07.625 align:middle line:90%
RAMA RAMAKRISHNAN: Right.

00:02:07.625 --> 00:02:09.490 align:middle line:84%
So if you take a
particular pixel,

00:02:09.490 --> 00:02:12.790 align:middle line:84%
then let's say that the
picture shows a t-shirt.

00:02:12.790 --> 00:02:15.810 align:middle line:84%
If there's a little pixel in
the center of the t-shirt,

00:02:15.810 --> 00:02:17.790 align:middle line:84%
knowing that the surrounding
pixels are related

00:02:17.790 --> 00:02:19.470 align:middle line:84%
to the pixel in a
way, because they

00:02:19.470 --> 00:02:22.190 align:middle line:84%
are all part of this
concept called a t-shirt,

00:02:22.190 --> 00:02:24.950 align:middle line:90%
would certainly be helpful.

00:02:24.950 --> 00:02:28.750 align:middle line:84%
So to put it more technically,
spatial adjacency information

00:02:28.750 --> 00:02:30.150 align:middle line:90%
is very important.

00:02:30.150 --> 00:02:33.710 align:middle line:84%
And we need to somehow
take that into account.

00:02:33.710 --> 00:02:34.210 align:middle line:90%
All right.

00:02:34.210 --> 00:02:34.690 align:middle line:90%
What else?

00:02:34.690 --> 00:02:36.065 align:middle line:84%
What else might
be going on here?

00:02:36.065 --> 00:02:40.955 align:middle line:90%


00:02:40.955 --> 00:02:42.830 align:middle line:84%
AUDIENCE: Yeah, doing
some metadata about it,

00:02:42.830 --> 00:02:44.830 align:middle line:84%
like read the dimensions
or the resolution

00:02:44.830 --> 00:02:45.872 align:middle line:90%
can help you [INAUDIBLE].

00:02:45.872 --> 00:02:47.080 align:middle line:90%
RAMA RAMAKRISHNAN: Oh, I see.

00:02:47.080 --> 00:02:49.590 align:middle line:84%
So if you actually had
structured data about the image,

00:02:49.590 --> 00:02:52.650 align:middle line:84%
such as various characters
about, that might be helpful.

00:02:52.650 --> 00:02:53.665 align:middle line:90%
True.

00:02:53.665 --> 00:02:55.790 align:middle line:84%
But let's just focus on
the case where we only have

00:02:55.790 --> 00:02:57.860 align:middle line:90%
the raw image and nothing else.

00:02:57.860 --> 00:03:02.500 align:middle line:84%
And under that constraint,
what else might go wrong

00:03:02.500 --> 00:03:04.320 align:middle line:84%
or what else might
be suboptimal?

00:03:04.320 --> 00:03:08.700 align:middle line:90%


00:03:08.700 --> 00:03:10.860 align:middle line:84%
Well, the first thing
that might happen

00:03:10.860 --> 00:03:14.880 align:middle line:84%
is that we may have
too many parameters.

00:03:14.880 --> 00:03:17.500 align:middle line:90%


00:03:17.500 --> 00:03:20.620 align:middle line:84%
So these numbers are
from my older iPhone.

00:03:20.620 --> 00:03:23.720 align:middle line:84%
I noticed that when I take a
color picture with my phone,

00:03:23.720 --> 00:03:28.800 align:middle line:84%
it's a 3,000-times-3,000,
roughly, grid.

00:03:28.800 --> 00:03:32.960 align:middle line:84%
So the picture is actually
3,024 pixels on this axis,

00:03:32.960 --> 00:03:34.940 align:middle line:90%
3,024 on that axis.

00:03:34.940 --> 00:03:39.540 align:middle line:84%
So that gets us to
roughly 9 million pixels.

00:03:39.540 --> 00:03:41.280 align:middle line:84%
But remember, there's
a color picture,

00:03:41.280 --> 00:03:43.660 align:middle line:84%
which means there are
three channels, which

00:03:43.660 --> 00:03:47.380 align:middle line:84%
means there are 27 million
numbers, each of which

00:03:47.380 --> 00:03:50.580 align:middle line:84%
is between 0 and 255
from that little picture.

00:03:50.580 --> 00:03:55.650 align:middle line:84%
And now let's say we connect
it to a single 100-neuron dense

00:03:55.650 --> 00:03:58.875 align:middle line:84%
layer, a single
100-neuron dense layer.

00:03:58.875 --> 00:04:01.250 align:middle line:84%
How many parameters are we
going to have just in that one

00:04:01.250 --> 00:04:02.375 align:middle line:90%
little part of the network?

00:04:02.375 --> 00:04:07.010 align:middle line:90%


00:04:07.010 --> 00:04:10.290 align:middle line:90%
Could the mumbling be louder?

00:04:10.290 --> 00:04:12.590 align:middle line:90%
Yes, roughly 2.7 billion.

00:04:12.590 --> 00:04:17.070 align:middle line:84%
Because 27 million parameters
times 100, roughly, of course.

00:04:17.070 --> 00:04:18.829 align:middle line:84%
Forget about the
biases for a moment.

00:04:18.829 --> 00:04:20.970 align:middle line:90%
It's 2.7 billion.

00:04:20.970 --> 00:04:23.490 align:middle line:90%
2.7 billion parameters.

00:04:23.490 --> 00:04:27.210 align:middle line:84%
Do you think we can actually
get 2.7 billion images

00:04:27.210 --> 00:04:32.530 align:middle line:84%
to train any of these things
so that we don't overfit?

00:04:32.530 --> 00:04:34.130 align:middle line:90%
Too many parameters.

00:04:34.130 --> 00:04:35.510 align:middle line:84%
We have to be
smarter about this.

00:04:35.510 --> 00:04:36.770 align:middle line:90%
It's not going to work.

00:04:36.770 --> 00:04:39.810 align:middle line:90%
That's the first problem.

00:04:39.810 --> 00:04:41.870 align:middle line:84%
So this clearly is
computationally demanding,

00:04:41.870 --> 00:04:46.370 align:middle line:84%
very data hungry, and increase
the risk of overfitting.

00:04:46.370 --> 00:04:51.610 align:middle line:90%
Next, we lose spatial adjacency.

00:04:51.610 --> 00:04:55.450 align:middle line:84%
We literally are
ignoring what's nearby.

00:04:55.450 --> 00:04:57.330 align:middle line:90%
So that's a huge, huge factor.

00:04:57.330 --> 00:05:01.010 align:middle line:84%
There is a third factor
that we have to worry about,

00:05:01.010 --> 00:05:06.170 align:middle line:84%
which is that, let's say that
the picture has a vertical line

00:05:06.170 --> 00:05:07.790 align:middle line:90%
on the top left side.

00:05:07.790 --> 00:05:12.250 align:middle line:84%
And it has some other vertical
line on the bottom right side.

00:05:12.250 --> 00:05:15.210 align:middle line:84%
What this sort of dumb
approach is going to do,

00:05:15.210 --> 00:05:17.730 align:middle line:84%
it's going to learn to
detect that vertical line

00:05:17.730 --> 00:05:18.770 align:middle line:90%
on the top left.

00:05:18.770 --> 00:05:20.610 align:middle line:84%
And independent of
that, it's going

00:05:20.610 --> 00:05:24.690 align:middle line:84%
to learn to detect the vertical
line on the bottom right, which

00:05:24.690 --> 00:05:26.570 align:middle line:90%
doesn't make any sense.

00:05:26.570 --> 00:05:28.290 align:middle line:84%
A vertical line is
a vertical line.

00:05:28.290 --> 00:05:31.370 align:middle line:84%
So you want to be able to
detect it wherever it happens.

00:05:31.370 --> 00:05:33.890 align:middle line:90%
Detect once, reuse everywhere.

00:05:33.890 --> 00:05:35.490 align:middle line:90%
That's what you need to do.

00:05:35.490 --> 00:05:38.650 align:middle line:84%
So this, by the way, is
called translation invariance.

00:05:38.650 --> 00:05:41.970 align:middle line:84%
Translation is math speak
for move stuff around.

00:05:41.970 --> 00:05:43.960 align:middle line:84%
You take a line and
it moves around.

00:05:43.960 --> 00:05:44.710 align:middle line:90%
It doesn't matter.

00:05:44.710 --> 00:05:45.460 align:middle line:90%
It's still a line.

00:05:45.460 --> 00:05:47.210 align:middle line:90%
Let's figure it out.

00:05:47.210 --> 00:05:50.410 align:middle line:84%
So these are the three things
we need to worry about.

00:05:50.410 --> 00:05:53.280 align:middle line:84%
So we want to learn once
and use all over the place.

00:05:53.280 --> 00:05:55.000 align:middle line:84%
We want to take
spatial adjacency

00:05:55.000 --> 00:05:56.185 align:middle line:90%
into account, number two.

00:05:56.185 --> 00:05:57.560 align:middle line:84%
And number three,
let's just find

00:05:57.560 --> 00:06:00.060 align:middle line:84%
a way to make sure that we don't
have billions of parameters

00:06:00.060 --> 00:06:02.320 align:middle line:90%
for simple toy problems.

00:06:02.320 --> 00:06:05.520 align:middle line:90%
Any questions?

00:06:05.520 --> 00:06:07.800 align:middle line:90%
Yeah?

00:06:07.800 --> 00:06:10.840 align:middle line:84%
AUDIENCE: Is this the
problem of just because we

00:06:10.840 --> 00:06:12.840 align:middle line:84%
are compressing
the image, or that

00:06:12.840 --> 00:06:14.120 align:middle line:90%
would've happened nonetheless?

00:06:14.120 --> 00:06:14.940 align:middle line:84%
RAMA RAMAKRISHNAN: It
would have happened.

00:06:14.940 --> 00:06:16.440 align:middle line:84%
So the question
was, is it a problem

00:06:16.440 --> 00:06:18.480 align:middle line:84%
because we're compressing
the image or would

00:06:18.480 --> 00:06:19.800 align:middle line:90%
it have happened anyway?

00:06:19.800 --> 00:06:20.880 align:middle line:84%
The answer is it
would've happened anyway.

00:06:20.880 --> 00:06:22.922 align:middle line:84%
You can take any picture,
that's going to happen.

00:06:22.922 --> 00:06:25.440 align:middle line:84%
Because I'm not
making any assumptions

00:06:25.440 --> 00:06:28.240 align:middle line:84%
about how the image is coming in
to me, whether it's compressed

00:06:28.240 --> 00:06:29.660 align:middle line:90%
or not, and so on, and so forth.

00:06:29.660 --> 00:06:32.320 align:middle line:90%


00:06:32.320 --> 00:06:33.600 align:middle line:90%
All right.

00:06:33.600 --> 00:06:38.640 align:middle line:84%
So convolutional layers were
developed to precisely address

00:06:38.640 --> 00:06:41.113 align:middle line:84%
these shortcomings, and
they're an amazing solution,

00:06:41.113 --> 00:06:41.780 align:middle line:90%
as you will see.

00:06:41.780 --> 00:06:42.322 align:middle line:90%
Very elegant.

00:06:42.322 --> 00:06:45.040 align:middle line:90%


00:06:45.040 --> 00:06:45.840 align:middle line:90%
All right.

00:06:45.840 --> 00:06:49.120 align:middle line:84%
So the next, I don't
know, half an hour

00:06:49.120 --> 00:06:52.400 align:middle line:84%
is going to be me defining
a whole bunch of stuff

00:06:52.400 --> 00:06:54.120 align:middle line:84%
before we actually
get to the fun Colabs,

00:06:54.120 --> 00:06:55.885 align:middle line:90%
and so on, and so forth.

00:06:55.885 --> 00:06:57.260 align:middle line:84%
So just to put it
in perspective,

00:06:57.260 --> 00:07:02.480 align:middle line:84%
I have a PowerPoint, two Colabs,
and an Excel spreadsheet,

00:07:02.480 --> 00:07:06.080 align:middle line:84%
and maybe even a Notability
file to cover today.

00:07:06.080 --> 00:07:08.480 align:middle line:84%
But hang on for
the next 30 minutes

00:07:08.480 --> 00:07:10.880 align:middle line:84%
because it's going to be a
little concept heavy before we

00:07:10.880 --> 00:07:12.120 align:middle line:90%
get to the fun stuff.

00:07:12.120 --> 00:07:12.740 align:middle line:90%
Stop me.

00:07:12.740 --> 00:07:15.120 align:middle line:84%
Ask me questions
because we do have time.

00:07:15.120 --> 00:07:15.740 align:middle line:90%
All right.

00:07:15.740 --> 00:07:18.040 align:middle line:84%
A convolutional layer
is made up of something

00:07:18.040 --> 00:07:20.000 align:middle line:90%
called a convolutional filter.

00:07:20.000 --> 00:07:22.200 align:middle line:84%
That's the atomic
building block.

00:07:22.200 --> 00:07:26.240 align:middle line:84%
A convolutional filter is
nothing but a small matrix

00:07:26.240 --> 00:07:28.160 align:middle line:90%
of numbers, like this.

00:07:28.160 --> 00:07:30.220 align:middle line:84%
It's just a small square
matrix of numbers.

00:07:30.220 --> 00:07:32.560 align:middle line:90%
That's the convolutional filter.

00:07:32.560 --> 00:07:38.160 align:middle line:84%
Now a layer is just composed of
one or more of these filters.

00:07:38.160 --> 00:07:41.440 align:middle line:90%
All right, filters and layers.

00:07:41.440 --> 00:07:44.840 align:middle line:84%
Now the thing about the
convolutional filter that

00:07:44.840 --> 00:07:47.840 align:middle line:84%
makes it really magical
is that if you choose

00:07:47.840 --> 00:07:51.230 align:middle line:84%
the numbers in a filter
carefully and then

00:07:51.230 --> 00:07:52.930 align:middle line:84%
you apply the
filter to an image,

00:07:52.930 --> 00:07:55.932 align:middle line:84%
and I'll get to what I mean
by applying the filter.

00:07:55.932 --> 00:07:57.390 align:middle line:84%
If you choose the
numbers carefully

00:07:57.390 --> 00:08:01.150 align:middle line:84%
and you apply it to that image,
this little, humble thing

00:08:01.150 --> 00:08:04.750 align:middle line:84%
has the ability to detect
features in your image.

00:08:04.750 --> 00:08:08.230 align:middle line:84%
It can detect lines,
curves, gradations in color,

00:08:08.230 --> 00:08:09.910 align:middle line:90%
circles, things like that.

00:08:09.910 --> 00:08:11.350 align:middle line:90%
It's pretty cool.

00:08:11.350 --> 00:08:13.910 align:middle line:84%
And so I'm going
to claim and I'm

00:08:13.910 --> 00:08:16.230 align:middle line:84%
going to prove shortly that
this little humble filter

00:08:16.230 --> 00:08:19.030 align:middle line:84%
with the 1's and 0's, it
can detect horizontal lines

00:08:19.030 --> 00:08:22.110 align:middle line:90%
in any picture you give it.

00:08:22.110 --> 00:08:27.630 align:middle line:84%
This thing here has the ability
to detect vertical lines.

00:08:27.630 --> 00:08:29.910 align:middle line:84%
So I will demonstrate
how this thing actually

00:08:29.910 --> 00:08:31.350 align:middle line:90%
detects all these things.

00:08:31.350 --> 00:08:33.669 align:middle line:84%
And then we will ask
the big question that's

00:08:33.669 --> 00:08:35.230 align:middle line:90%
probably in your minds already.

00:08:35.230 --> 00:08:37.787 align:middle line:84%
Where are we going to
get these numbers from?

00:08:37.787 --> 00:08:38.870 align:middle line:90%
It all sounds great, Rama.

00:08:38.870 --> 00:08:40.662 align:middle line:84%
Where are we going to
get the numbers from?

00:08:40.662 --> 00:08:43.870 align:middle line:84%
And we have a beautiful
answer to that question.

00:08:43.870 --> 00:08:46.110 align:middle line:90%
All right, so let's go.

00:08:46.110 --> 00:08:47.710 align:middle line:84%
Now I'm going to
first explain to you

00:08:47.710 --> 00:08:50.980 align:middle line:84%
what I mean by applying
a filter to an image.

00:08:50.980 --> 00:08:52.678 align:middle line:84%
And then I'm going
to give you examples

00:08:52.678 --> 00:08:54.220 align:middle line:84%
of how the filter
works for detecting

00:08:54.220 --> 00:08:56.680 align:middle line:90%
vertical and horizontal lines.

00:08:56.680 --> 00:09:00.900 align:middle line:84%
So let's say that this
is the image we have.

00:09:00.900 --> 00:09:03.580 align:middle line:84%
Again, an image, I assume
it's a grayscale image.

00:09:03.580 --> 00:09:06.680 align:middle line:84%
So you just have a bunch of
numbers between 0 and 255.

00:09:06.680 --> 00:09:08.160 align:middle line:90%
So this is the image we have.

00:09:08.160 --> 00:09:09.617 align:middle line:90%
It's a little, tiny image.

00:09:09.617 --> 00:09:11.700 align:middle line:84%
And this is the filter
that's been magically given

00:09:11.700 --> 00:09:13.180 align:middle line:90%
to us by somebody.

00:09:13.180 --> 00:09:15.220 align:middle line:84%
And what we're trying to
do now is to apply it.

00:09:15.220 --> 00:09:17.660 align:middle line:84%
So what we do is
that we literally

00:09:17.660 --> 00:09:21.780 align:middle line:84%
take this filter, the
little one, and then

00:09:21.780 --> 00:09:24.780 align:middle line:84%
we superimpose it on the
top left part of the image.

00:09:24.780 --> 00:09:26.580 align:middle line:90%
So you have the image here.

00:09:26.580 --> 00:09:29.180 align:middle line:84%
You take this little filter
and then you move it to the top

00:09:29.180 --> 00:09:32.100 align:middle line:84%
left so that they're right
on top of each other.

00:09:32.100 --> 00:09:35.380 align:middle line:84%
Once you have it right
on top of each other,

00:09:35.380 --> 00:09:37.440 align:middle line:90%
you have these matching numbers.

00:09:37.440 --> 00:09:39.560 align:middle line:84%
You have three
numbers in the image.

00:09:39.560 --> 00:09:41.112 align:middle line:84%
There are three
numbers in the filter

00:09:41.112 --> 00:09:42.820 align:middle line:84%
and they're all matching
each other right

00:09:42.820 --> 00:09:43.695 align:middle line:90%
on top of each other.

00:09:43.695 --> 00:09:46.420 align:middle line:84%
So you have nine
pairs of numbers.

00:09:46.420 --> 00:09:48.980 align:middle line:84%
And then what we do
once we overlay it,

00:09:48.980 --> 00:09:51.660 align:middle line:84%
we literally just multiply
all the matching numbers

00:09:51.660 --> 00:09:54.193 align:middle line:90%
and add them up.

00:09:54.193 --> 00:09:56.360 align:middle line:84%
You just multiply all the
numbers and match them up.

00:09:56.360 --> 00:09:59.380 align:middle line:84%
And you can confirm later on
that the arithmetic I'm doing

00:09:59.380 --> 00:10:00.540 align:middle line:90%
is actually accurate.

00:10:00.540 --> 00:10:05.540 align:middle line:84%
And once you do that,
you'll get some number.

00:10:05.540 --> 00:10:09.700 align:middle line:84%
And once you get that
number, what we do

00:10:09.700 --> 00:10:12.300 align:middle line:84%
is we go to our good,
old friend, the ReLU,

00:10:12.300 --> 00:10:15.163 align:middle line:84%
and then we just
run it through ReLU.

00:10:15.163 --> 00:10:16.580 align:middle line:84%
Now, in this case,
all that effort

00:10:16.580 --> 00:10:18.180 align:middle line:90%
came to nothing because it's 0.

00:10:18.180 --> 00:10:19.540 align:middle line:90%
That's OK.

00:10:19.540 --> 00:10:23.140 align:middle line:84%
So 0, and this number
becomes the top left cell

00:10:23.140 --> 00:10:23.870 align:middle line:90%
of your output.

00:10:23.870 --> 00:10:26.620 align:middle line:90%


00:10:26.620 --> 00:10:29.798 align:middle line:84%
So this is called the
convolution operation.

00:10:29.798 --> 00:10:32.340 align:middle line:84%
And we won't get into why it's
called that, and so on, and so

00:10:32.340 --> 00:10:32.840 align:middle line:90%
forth.

00:10:32.840 --> 00:10:35.780 align:middle line:84%
There's a long, and rich, and
storied history of these things.

00:10:35.780 --> 00:10:38.180 align:middle line:84%
But this is the
convolution operation.

00:10:38.180 --> 00:10:42.060 align:middle line:84%
And once we do that, you can now
predict what's going to happen.

00:10:42.060 --> 00:10:44.380 align:middle line:84%
We take the same exact
operation and we just

00:10:44.380 --> 00:10:46.890 align:middle line:90%
move it to the right.

00:10:46.890 --> 00:10:49.410 align:middle line:84%
We move this little 3-by-3
thing to the right and repeat

00:10:49.410 --> 00:10:51.450 align:middle line:90%
the exact same process.

00:10:51.450 --> 00:10:55.750 align:middle line:84%
Matching numbers, multiply all
the matching numbers together.

00:10:55.750 --> 00:10:59.370 align:middle line:84%
Add them up, run
them through a ReLU.

00:10:59.370 --> 00:11:03.730 align:middle line:84%
And then boom, you get
the second number here.

00:11:03.730 --> 00:11:06.670 align:middle line:84%
And you keep doing that
until you reach the very end.

00:11:06.670 --> 00:11:08.210 align:middle line:84%
You fill up all
these numbers, then

00:11:08.210 --> 00:11:12.610 align:middle line:84%
you come to the top
of the second row.

00:11:12.610 --> 00:11:16.410 align:middle line:84%
And you keep on doing that
till you reach the very bottom.

00:11:16.410 --> 00:11:22.130 align:middle line:84%
So this is what I mean when I
say apply a filter to an image.

00:11:22.130 --> 00:11:22.990 align:middle line:90%
Any questions?

00:11:22.990 --> 00:11:29.490 align:middle line:90%


00:11:29.490 --> 00:11:31.050 align:middle line:90%
Microphone, please.

00:11:31.050 --> 00:11:32.170 align:middle line:90%
Microphone.

00:11:32.170 --> 00:11:35.050 align:middle line:90%


00:11:35.050 --> 00:11:39.410 align:middle line:84%
AUDIENCE: What happens when they
aren't particularly [INAUDIBLE]

00:11:39.410 --> 00:11:43.690 align:middle line:84%
the remaining but the filter
doesn't perfectly match?

00:11:43.690 --> 00:11:44.690 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:11:44.690 --> 00:11:47.100 align:middle line:84%
So you start from the left
and then you keep on going.

00:11:47.100 --> 00:11:49.200 align:middle line:84%
At some point, the
right edge of the filter

00:11:49.200 --> 00:11:52.800 align:middle line:84%
is going to match the right edge
of the image, and then you stop.

00:11:52.800 --> 00:11:55.360 align:middle line:90%
Now there are some nuances here.

00:11:55.360 --> 00:11:59.880 align:middle line:84%
So for example, you can
actually pad the whole image

00:11:59.880 --> 00:12:03.080 align:middle line:84%
on its borders so that you can
actually go outside the image

00:12:03.080 --> 00:12:05.560 align:middle line:90%
and it'll still work.

00:12:05.560 --> 00:12:06.520 align:middle line:90%
Number one.

00:12:06.520 --> 00:12:08.480 align:middle line:90%
Number two, nuance.

00:12:08.480 --> 00:12:10.960 align:middle line:84%
Instead of just moving one
step to the right every time

00:12:10.960 --> 00:12:13.880 align:middle line:84%
you finish, you can move
two steps to the right.

00:12:13.880 --> 00:12:16.520 align:middle line:84%
And that's something
called a stride.

00:12:16.520 --> 00:12:19.360 align:middle line:84%
So there are a bunch
of pesky details here,

00:12:19.360 --> 00:12:22.600 align:middle line:84%
but I'm just ignoring them
because this basic default

00:12:22.600 --> 00:12:25.860 align:middle line:84%
approach works well, amazingly
well, almost all the time.

00:12:25.860 --> 00:12:29.140 align:middle line:90%


00:12:29.140 --> 00:12:33.243 align:middle line:84%
So that's the mechanics of
how this operation works.

00:12:33.243 --> 00:12:35.160 align:middle line:84%
Now I'm going to switch
to a spreadsheet which

00:12:35.160 --> 00:12:41.340 align:middle line:84%
shows this really beautifully,
courtesy of the fast.ai people.

00:12:41.340 --> 00:12:43.240 align:middle line:90%
So what I'm going to do here--

00:12:43.240 --> 00:12:44.360 align:middle line:90%
this is a big spreadsheet.

00:12:44.360 --> 00:12:46.777 align:middle line:84%
I'll upload the spreadsheet
after class so you can see it.

00:12:46.777 --> 00:12:48.480 align:middle line:84%
So all I have done
here, or rather,

00:12:48.480 --> 00:12:51.067 align:middle line:84%
all they have done
here, thanks to them,

00:12:51.067 --> 00:12:53.400 align:middle line:84%
is that they have essentially
created a table of numbers

00:12:53.400 --> 00:12:55.360 align:middle line:90%
in Excel, as you can tell.

00:12:55.360 --> 00:12:57.500 align:middle line:84%
And they have just
put some numbers.

00:12:57.500 --> 00:13:00.840 align:middle line:84%
Most of the numbers are 0,
but some of these numbers

00:13:00.840 --> 00:13:02.060 align:middle line:90%
are all more than 0.

00:13:02.060 --> 00:13:03.333 align:middle line:90%
They are 0.8, 0.9, and so on.

00:13:03.333 --> 00:13:05.000 align:middle line:84%
Basically all they
have done is, instead

00:13:05.000 --> 00:13:06.858 align:middle line:84%
of working with numbers
between 0 and 255,

00:13:06.858 --> 00:13:08.400 align:middle line:84%
they're just dividing
all the numbers

00:13:08.400 --> 00:13:09.920 align:middle line:90%
by 255 so you get fractions.

00:13:09.920 --> 00:13:11.940 align:middle line:84%
And they just put the
fractions in the table.

00:13:11.940 --> 00:13:16.880 align:middle line:84%
And then they have used Excel's
very cool conditional formatting

00:13:16.880 --> 00:13:21.620 align:middle line:84%
to essentially mark in red
all the values that are high.

00:13:21.620 --> 00:13:24.680 align:middle line:84%
If the number is closer to
1, the more reddish it gets.

00:13:24.680 --> 00:13:28.560 align:middle line:84%
And when you do that,
the 3 obviously pops out.

00:13:28.560 --> 00:13:30.160 align:middle line:90%
So there is a 3 in the image.

00:13:30.160 --> 00:13:31.040 align:middle line:90%
Yes?

00:13:31.040 --> 00:13:32.000 align:middle line:90%
OK, good.

00:13:32.000 --> 00:13:35.080 align:middle line:84%
So now what we're
going to do is we're

00:13:35.080 --> 00:13:37.880 align:middle line:84%
going to move to our
little filter here.

00:13:37.880 --> 00:13:40.020 align:middle line:90%
You can see the filter.

00:13:40.020 --> 00:13:42.080 align:middle line:84%
And I'm claiming this
detects horizontal lines.

00:13:42.080 --> 00:13:47.130 align:middle line:90%


00:13:47.130 --> 00:13:47.630 align:middle line:90%
Sorry.

00:13:47.630 --> 00:13:51.350 align:middle line:90%


00:13:51.350 --> 00:13:54.710 align:middle line:84%
This table here is the result
of applying that filter

00:13:54.710 --> 00:13:57.790 align:middle line:84%
to that tree And
you can see here,

00:13:57.790 --> 00:13:59.400 align:middle line:84%
I'm looking at the
top left cell here.

00:13:59.400 --> 00:14:03.790 align:middle line:90%


00:14:03.790 --> 00:14:05.010 align:middle line:90%
Look at this top left cell.

00:14:05.010 --> 00:14:08.670 align:middle line:84%
The formula is nothing more
than multiply all those things

00:14:08.670 --> 00:14:09.630 align:middle line:90%
and add them up.

00:14:09.630 --> 00:14:13.010 align:middle line:84%
And then once you add it up,
run it through a max of 0 comma

00:14:13.010 --> 00:14:15.990 align:middle line:90%
that, which is just the ReLU.

00:14:15.990 --> 00:14:18.110 align:middle line:90%
Basic arithmetic.

00:14:18.110 --> 00:14:19.470 align:middle line:90%
So we do that.

00:14:19.470 --> 00:14:21.030 align:middle line:90%
And this is the output.

00:14:21.030 --> 00:14:23.830 align:middle line:84%
And the output is also
conditionally formatted

00:14:23.830 --> 00:14:26.670 align:middle line:84%
to show you where
things are lighting up.

00:14:26.670 --> 00:14:31.710 align:middle line:84%
And you can see, only the
horizontal lines of the 3

00:14:31.710 --> 00:14:34.430 align:middle line:90%
are lighting up.

00:14:34.430 --> 00:14:36.750 align:middle line:90%
Everyone see that?

00:14:36.750 --> 00:14:38.730 align:middle line:90%
So now you understand.

00:14:38.730 --> 00:14:43.270 align:middle line:84%
The filter, in fact, is living
up to the claim I made for it.

00:14:43.270 --> 00:14:46.250 align:middle line:84%
Similarly, if you look
at what's going on here,

00:14:46.250 --> 00:14:48.350 align:middle line:84%
this is a vertical
filter, the same thing.

00:14:48.350 --> 00:14:53.510 align:middle line:84%
You apply it, only the
vertical line is lighting up.

00:14:53.510 --> 00:14:58.470 align:middle line:84%
Now what you can do is, I
would encourage you to do this

00:14:58.470 --> 00:15:01.110 align:middle line:84%
after class, is you can
look at all these numbers

00:15:01.110 --> 00:15:02.490 align:middle line:90%
here, for example.

00:15:02.490 --> 00:15:06.510 align:middle line:84%
And then ask yourself, OK,
why is that lighting up?

00:15:06.510 --> 00:15:09.830 align:middle line:84%
And you will discover that
what's actually going on

00:15:09.830 --> 00:15:12.470 align:middle line:90%
is that it's looking for edges.

00:15:12.470 --> 00:15:16.150 align:middle line:84%
It's looking for
rows in the table

00:15:16.150 --> 00:15:18.990 align:middle line:84%
where there is some non-zero
thing in the first row

00:15:18.990 --> 00:15:21.750 align:middle line:90%
and zeros in the second row.

00:15:21.750 --> 00:15:23.650 align:middle line:84%
And by choosing the
numbers carefully,

00:15:23.650 --> 00:15:25.930 align:middle line:84%
you multiply the ones
with positive numbers

00:15:25.930 --> 00:15:28.072 align:middle line:84%
and you multiply the
zeros with zeros.

00:15:28.072 --> 00:15:30.030 align:middle line:84%
And then you'll come up
with a positive number,

00:15:30.030 --> 00:15:32.910 align:middle line:90%
and thereby you detect an edge.

00:15:32.910 --> 00:15:37.340 align:middle line:84%
So what I would encourage you to
do is use this Excel thing here.

00:15:37.340 --> 00:15:40.050 align:middle line:90%


00:15:40.050 --> 00:15:42.140 align:middle line:90%
So here is a cell we have.

00:15:42.140 --> 00:15:51.380 align:middle line:90%
So let's trace its incidence.

00:15:51.380 --> 00:15:57.340 align:middle line:84%
So you can see here,
these numbers, this

00:15:57.340 --> 00:15:59.440 align:middle line:90%
is what it's processing.

00:15:59.440 --> 00:16:01.540 align:middle line:84%
That is a grid that's
being processing to come up

00:16:01.540 --> 00:16:02.840 align:middle line:90%
with that big number.

00:16:02.840 --> 00:16:06.980 align:middle line:84%
And you can see here in this
grid, these numbers are here.

00:16:06.980 --> 00:16:10.620 align:middle line:84%
And then these numbers are a
lot lower than these numbers

00:16:10.620 --> 00:16:13.513 align:middle line:90%
because there is an edge.

00:16:13.513 --> 00:16:14.680 align:middle line:90%
The numbers are a lot lower.

00:16:14.680 --> 00:16:17.820 align:middle line:84%
That's why you can see the
horizontal part of the 3.

00:16:17.820 --> 00:16:21.180 align:middle line:84%
And so what this filter is
doing, it's basically saying,

00:16:21.180 --> 00:16:24.980 align:middle line:84%
well, the row that I'm
catching here has the 1's.

00:16:24.980 --> 00:16:25.960 align:middle line:90%
The middle has 0's.

00:16:25.960 --> 00:16:27.740 align:middle line:90%
The rest are all minus 1's.

00:16:27.740 --> 00:16:31.460 align:middle line:84%
So the small values are
going to get very small.

00:16:31.460 --> 00:16:33.240 align:middle line:84%
The big values are
going to get very big,

00:16:33.240 --> 00:16:35.580 align:middle line:84%
and the overall thing is
going to be emphasized.

00:16:35.580 --> 00:16:38.380 align:middle line:84%
So that's the basic
idea of edge detection.

00:16:38.380 --> 00:16:40.110 align:middle line:84%
Spend some time
with it, with Excel,

00:16:40.110 --> 00:16:43.130 align:middle line:84%
and it will become clear to you
what I'm talking about here.

00:16:43.130 --> 00:16:44.210 align:middle line:90%
All right, cool.

00:16:44.210 --> 00:16:46.710 align:middle line:90%
So that's that.

00:16:46.710 --> 00:16:48.450 align:middle line:90%
By the way, I also have--

00:16:48.450 --> 00:16:51.850 align:middle line:84%
there is a very cool site
here in which you can actually

00:16:51.850 --> 00:16:53.490 align:middle line:84%
go in, and punch in
your own numbers,

00:16:53.490 --> 00:16:55.432 align:middle line:90%
and see what it detects.

00:16:55.432 --> 00:16:57.390 align:middle line:84%
A lot of edges, and
curves, and this, and that.

00:16:57.390 --> 00:17:00.090 align:middle line:84%
It's very cool so I
encourage you to try it out.

00:17:00.090 --> 00:17:02.330 align:middle line:84%
So the key thing
here I want to say

00:17:02.330 --> 00:17:08.210 align:middle line:84%
is by choosing the
numbers in a filter

00:17:08.210 --> 00:17:10.310 align:middle line:84%
carefully and applying
this operation,

00:17:10.310 --> 00:17:13.730 align:middle line:84%
different features
can be detected.

00:17:13.730 --> 00:17:17.130 align:middle line:84%
Now I mentioned earlier that a
convolution layer is composed

00:17:17.130 --> 00:17:21.490 align:middle line:84%
of one or more of these filters,
so one or more of these filters.

00:17:21.490 --> 00:17:25.569 align:middle line:84%
And so you can think of
each filter as a specialist

00:17:25.569 --> 00:17:28.650 align:middle line:90%
for a particular feature.

00:17:28.650 --> 00:17:29.750 align:middle line:90%
So it's a specialist.

00:17:29.750 --> 00:17:31.410 align:middle line:84%
Maybe it specializes
in detecting

00:17:31.410 --> 00:17:34.863 align:middle line:84%
vertical lines, horizontal
lines, semicircles,

00:17:34.863 --> 00:17:35.530 align:middle line:90%
quarter circles.

00:17:35.530 --> 00:17:36.790 align:middle line:90%
You don't know.

00:17:36.790 --> 00:17:39.850 align:middle line:84%
You can imagine each of
them as being specialists.

00:17:39.850 --> 00:17:43.110 align:middle line:84%
And given that modern images
could be very complicated,

00:17:43.110 --> 00:17:45.310 align:middle line:84%
they may have lots of
interesting features going on,

00:17:45.310 --> 00:17:49.050 align:middle line:84%
you probably want to have
lots of these filters.

00:17:49.050 --> 00:17:54.050 align:middle line:84%
But the key is that you
don't have to decide upfront.

00:17:54.050 --> 00:17:56.250 align:middle line:84%
Hey, you, filter,
you better specialize

00:17:56.250 --> 00:17:57.590 align:middle line:90%
in detecting vertical lines.

00:17:57.590 --> 00:17:59.150 align:middle line:84%
And you, on the
other hand, do not.

00:17:59.150 --> 00:18:00.030 align:middle line:90%
Stay in your lane.

00:18:00.030 --> 00:18:01.030 align:middle line:90%
Do vertical lines.

00:18:01.030 --> 00:18:02.530 align:middle line:90%
You're not going to do that.

00:18:02.530 --> 00:18:07.030 align:middle line:84%
You will let the system figure
out what it wants to figure out.

00:18:07.030 --> 00:18:10.170 align:middle line:84%
So there is no human
bottleneck in doing this.

00:18:10.170 --> 00:18:11.810 align:middle line:84%
And I mention this
because there used

00:18:11.810 --> 00:18:15.000 align:middle line:84%
to be a human bottleneck
before deep learning happened.

00:18:15.000 --> 00:18:17.570 align:middle line:90%


00:18:17.570 --> 00:18:19.298 align:middle line:84%
Now let's just make
sure we understand

00:18:19.298 --> 00:18:21.090 align:middle line:84%
the mechanics of what
happens when you have

00:18:21.090 --> 00:18:22.850 align:middle line:90%
two of these filters, not one.

00:18:22.850 --> 00:18:25.170 align:middle line:84%
So this is the input
image as before.

00:18:25.170 --> 00:18:26.890 align:middle line:84%
This is the filter
we saw earlier.

00:18:26.890 --> 00:18:29.238 align:middle line:84%
And this is another
filter we have.

00:18:29.238 --> 00:18:31.030 align:middle line:84%
The thing is, we just
run them in parallel.

00:18:31.030 --> 00:18:32.570 align:middle line:84%
We take each filter,
do the operation,

00:18:32.570 --> 00:18:33.430 align:middle line:90%
come up with an output.

00:18:33.430 --> 00:18:35.388 align:middle line:84%
Take the other filter,
do the operation come up

00:18:35.388 --> 00:18:36.130 align:middle line:90%
with its output.

00:18:36.130 --> 00:18:38.463 align:middle line:84%
And then when you do that,
the first one gives you that,

00:18:38.463 --> 00:18:39.880 align:middle line:90%
the second one gives you that.

00:18:39.880 --> 00:18:42.460 align:middle line:84%
And this output is
a table of some--

00:18:42.460 --> 00:18:44.820 align:middle line:90%
it's actually not a table.

00:18:44.820 --> 00:18:45.500 align:middle line:90%
What is it?

00:18:45.500 --> 00:18:49.320 align:middle line:90%


00:18:49.320 --> 00:18:51.400 align:middle line:90%
Louder, please.

00:18:51.400 --> 00:18:52.300 align:middle line:90%
It's a tensor.

00:18:52.300 --> 00:18:52.860 align:middle line:90%
Thank you.

00:18:52.860 --> 00:18:54.040 align:middle line:90%
It's a tensor.

00:18:54.040 --> 00:18:57.320 align:middle line:84%
And so these two 5-by-5 matrices
can be represented as a tensor

00:18:57.320 --> 00:18:57.910 align:middle line:90%
of what shape?

00:18:57.910 --> 00:19:01.907 align:middle line:90%


00:19:01.907 --> 00:19:03.240 align:middle line:90%
And there are two right answers.

00:19:03.240 --> 00:19:04.710 align:middle line:90%
AUDIENCE: 5 by 5.

00:19:04.710 --> 00:19:05.880 align:middle line:90%
RAMA RAMAKRISHNAN: 5 by 5?

00:19:05.880 --> 00:19:06.440 align:middle line:90%
AUDIENCE: Into 2.

00:19:06.440 --> 00:19:07.523 align:middle line:90%
RAMA RAMAKRISHNAN: Into 2.

00:19:07.523 --> 00:19:08.120 align:middle line:90%
Correct.

00:19:08.120 --> 00:19:11.720 align:middle line:84%
So you can either think of it
as 5 by 5 times 2, or 2 times 5

00:19:11.720 --> 00:19:12.480 align:middle line:90%
by 5.

00:19:12.480 --> 00:19:14.377 align:middle line:90%
They're both fine.

00:19:14.377 --> 00:19:15.960 align:middle line:84%
Which one you go
with actually ends up

00:19:15.960 --> 00:19:18.720 align:middle line:90%
being a matter of convention.

00:19:18.720 --> 00:19:22.720 align:middle line:84%
So now you begin to see
why we care about tensors.

00:19:22.720 --> 00:19:27.960 align:middle line:84%
Imagine if, instead of having
2 filters, we have 103 filters.

00:19:27.960 --> 00:19:33.560 align:middle line:84%
The resulting tensor is
going to be 5 by 5 by 103.

00:19:33.560 --> 00:19:36.530 align:middle line:90%
OK, good.

00:19:36.530 --> 00:19:41.110 align:middle line:84%
Now let's look at the slightly
more complex situation,

00:19:41.110 --> 00:19:44.010 align:middle line:84%
where you have not a
black and white image,

00:19:44.010 --> 00:19:46.170 align:middle line:84%
a grayscale image with
just a little table,

00:19:46.170 --> 00:19:49.670 align:middle line:90%
but an actual color image.

00:19:49.670 --> 00:19:54.030 align:middle line:84%
So we know how to apply a filter
to a 2D tensor, like this,

00:19:54.030 --> 00:19:55.230 align:middle line:90%
and to get that.

00:19:55.230 --> 00:19:58.890 align:middle line:84%
But let's say we have something
like this, where it has three.

00:19:58.890 --> 00:20:01.650 align:middle line:84%
It's got three channels,
red, blue, green, RGB.

00:20:01.650 --> 00:20:03.510 align:middle line:84%
It's got three
tables of numbers.

00:20:03.510 --> 00:20:07.470 align:middle line:84%
So this is a tensor of shape
6 times 6 times 3, let's say.

00:20:07.470 --> 00:20:10.830 align:middle line:84%
And you want to apply this
3-by-3 filter, just like before,

00:20:10.830 --> 00:20:11.490 align:middle line:90%
to this thing.

00:20:11.490 --> 00:20:13.550 align:middle line:84%
You want to apply the
convolution operation.

00:20:13.550 --> 00:20:14.810 align:middle line:90%
How is that going to work?

00:20:14.810 --> 00:20:18.430 align:middle line:90%


00:20:18.430 --> 00:20:21.750 align:middle line:90%
Do we just apply this to each?

00:20:21.750 --> 00:20:25.210 align:middle line:84%
We first apply it to the red,
then we apply it to the green,

00:20:25.210 --> 00:20:26.790 align:middle line:90%
then we apply it to the blue.

00:20:26.790 --> 00:20:33.080 align:middle line:84%
Should we do that, or is there
a problem with that approach?

00:20:33.080 --> 00:20:36.070 align:middle line:90%


00:20:36.070 --> 00:20:37.492 align:middle line:90%
Yeah?

00:20:37.492 --> 00:20:40.030 align:middle line:90%
AUDIENCE: The problem--

00:20:40.030 --> 00:20:42.505 align:middle line:84%
RAMA RAMAKRISHNAN: Could you
use the microphone, please?

00:20:42.505 --> 00:20:44.130 align:middle line:84%
AUDIENCE: The problem
with the approach

00:20:44.130 --> 00:20:46.130 align:middle line:84%
would be the same as
what you said earlier,

00:20:46.130 --> 00:20:47.250 align:middle line:90%
that it wouldn't learn.

00:20:47.250 --> 00:20:50.430 align:middle line:84%
The lines are probably
the same in each channel.

00:20:50.430 --> 00:20:53.352 align:middle line:84%
The location of the lines are
probably the same each channel.

00:20:53.352 --> 00:20:54.310 align:middle line:90%
RAMA RAMAKRISHNAN: Yes.

00:20:54.310 --> 00:20:55.685 align:middle line:84%
The location of
the line is going

00:20:55.685 --> 00:20:58.470 align:middle line:84%
to be the same thing because
that line, if you will,

00:20:58.470 --> 00:21:01.110 align:middle line:84%
is the aggregation of
information from the three

00:21:01.110 --> 00:21:02.090 align:middle line:90%
different channels.

00:21:02.090 --> 00:21:02.590 align:middle line:90%
Right.

00:21:02.590 --> 00:21:07.190 align:middle line:84%
But the problem here
is slightly different,

00:21:07.190 --> 00:21:12.030 align:middle line:84%
which is that if you
do them independently,

00:21:12.030 --> 00:21:15.030 align:middle line:84%
the network has
not been informed

00:21:15.030 --> 00:21:19.737 align:middle line:84%
that these things are all part
of the same underlying concept.

00:21:19.737 --> 00:21:21.570 align:middle line:84%
As far as concerned,
it's just three things.

00:21:21.570 --> 00:21:23.487 align:middle line:84%
It's just going to process
them independently.

00:21:23.487 --> 00:21:26.750 align:middle line:84%
So we need to somehow change the
filter so that it understands,

00:21:26.750 --> 00:21:29.950 align:middle line:84%
what is at this pixel location,
the three numbers under it?

00:21:29.950 --> 00:21:30.770 align:middle line:90%
RGB.

00:21:30.770 --> 00:21:35.140 align:middle line:84%
They are actually part of the
same thing, underlying thing.

00:21:35.140 --> 00:21:38.020 align:middle line:84%
So what we do is
actually very simple.

00:21:38.020 --> 00:21:42.580 align:middle line:84%
We just take this
filter and make it 3D.

00:21:42.580 --> 00:21:43.900 align:middle line:90%
So we take this filter.

00:21:43.900 --> 00:21:45.440 align:middle line:84%
Instead of having
just one of them,

00:21:45.440 --> 00:21:49.220 align:middle line:84%
we just make it a cube,
like that, three times.

00:21:49.220 --> 00:21:53.260 align:middle line:84%
And once we do that, you can
imagine taking this thing here

00:21:53.260 --> 00:21:56.380 align:middle line:90%
and essentially doing that.

00:21:56.380 --> 00:22:00.460 align:middle line:84%
Now, instead of having 9 numbers
in the image and 9 numbers

00:22:00.460 --> 00:22:03.060 align:middle line:84%
in the filter, you
have 27 numbers

00:22:03.060 --> 00:22:05.620 align:middle line:84%
in the image, 27
numbers in the filter.

00:22:05.620 --> 00:22:08.268 align:middle line:84%
But you still match
them up, multiply them,

00:22:08.268 --> 00:22:09.810 align:middle line:84%
add them up, run
them through a ReLU.

00:22:09.810 --> 00:22:14.780 align:middle line:90%


00:22:14.780 --> 00:22:16.540 align:middle line:84%
By the way, I tried
to get ChatGPT

00:22:16.540 --> 00:22:19.092 align:middle line:90%
to give me a picture like that.

00:22:19.092 --> 00:22:21.565 align:middle line:90%
It just completely bombed.

00:22:21.565 --> 00:22:23.440 align:middle line:84%
I tried three, four,
five different variants.

00:22:23.440 --> 00:22:24.200 align:middle line:90%
It just gave up.

00:22:24.200 --> 00:22:26.700 align:middle line:84%
Then I found this nice
picture in DeepLearning.ai

00:22:26.700 --> 00:22:28.623 align:middle line:90%
and I used it.

00:22:28.623 --> 00:22:30.540 align:middle line:84%
AUDIENCE: So then if you
put different numbers

00:22:30.540 --> 00:22:33.120 align:middle line:84%
in each of the layers of
that color processing,

00:22:33.120 --> 00:22:35.245 align:middle line:84%
would you get a different
thing to green and blue?

00:22:35.245 --> 00:22:37.120 align:middle line:84%
RAMA RAMAKRISHNAN: I'm
sorry, say that again.

00:22:37.120 --> 00:22:38.740 align:middle line:84%
AUDIENCE: If you put
different numbers

00:22:38.740 --> 00:22:40.820 align:middle line:90%
in each of the layers of your--

00:22:40.820 --> 00:22:43.420 align:middle line:84%
not layers, in each of the
different depth dimensions

00:22:43.420 --> 00:22:45.260 align:middle line:84%
of your convolution
filter, would that

00:22:45.260 --> 00:22:47.360 align:middle line:90%
be like color processing?

00:22:47.360 --> 00:22:48.360 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:22:48.360 --> 00:22:49.300 align:middle line:90%
AUDIENCE: [INAUDIBLE].

00:22:49.300 --> 00:22:49.540 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:22:49.540 --> 00:22:50.832 align:middle line:90%
You will put different numbers.

00:22:50.832 --> 00:22:52.383 align:middle line:84%
In fact, you have
27 numbers now.

00:22:52.383 --> 00:22:54.300 align:middle line:84%
But we haven't gotten
to the question of where

00:22:54.300 --> 00:22:55.640 align:middle line:90%
these numbers are coming from.

00:22:55.640 --> 00:22:58.780 align:middle line:84%
So just hold the thought
till we get there.

00:22:58.780 --> 00:23:03.340 align:middle line:90%
So any questions on this?

00:23:03.340 --> 00:23:05.740 align:middle line:84%
You literally take the
2D thing and make it 3D.

00:23:05.740 --> 00:23:08.740 align:middle line:84%
You basically give it
depth, and the depth just

00:23:08.740 --> 00:23:11.300 align:middle line:90%
matches the depth of the input.

00:23:11.300 --> 00:23:15.560 align:middle line:84%
So if the input is 10 deep, your
filter is going to get 10 deep.

00:23:15.560 --> 00:23:18.260 align:middle line:90%


00:23:18.260 --> 00:23:20.100 align:middle line:90%
OK?

00:23:20.100 --> 00:23:22.740 align:middle line:90%
Yes?

00:23:22.740 --> 00:23:26.140 align:middle line:84%
AUDIENCE: Rather than increasing
the rank order of the tensor

00:23:26.140 --> 00:23:27.620 align:middle line:84%
by one, is there
any instance where

00:23:27.620 --> 00:23:29.828 align:middle line:84%
you would create an abstraction
layer where you would

00:23:29.828 --> 00:23:31.930 align:middle line:84%
run an operation across
the different layers

00:23:31.930 --> 00:23:34.770 align:middle line:84%
to come up with a
intermediary layer

00:23:34.770 --> 00:23:38.210 align:middle line:84%
that you would run a lower
rank tensor or filter upper?

00:23:38.210 --> 00:23:39.210 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:23:39.210 --> 00:23:41.970 align:middle line:84%
So there is a lot of stuff
in the research literature

00:23:41.970 --> 00:23:43.970 align:middle line:84%
which tries to do
things like that.

00:23:43.970 --> 00:23:49.090 align:middle line:84%
I'm just describing the most
basic approach to doing this.

00:23:49.090 --> 00:23:53.450 align:middle line:84%
And as it turns out, this basic
approach is extremely powerful.

00:23:53.450 --> 00:23:56.890 align:middle line:84%
And of course, researchers try
to go from the 95th percent

00:23:56.890 --> 00:23:59.510 align:middle line:90%
thing to the 95.1%.

00:23:59.510 --> 00:24:02.250 align:middle line:84%
So they invent all sorts of
crazy, complicated stuff, which

00:24:02.250 --> 00:24:03.958 align:middle line:90%
is all good for us, humanity.

00:24:03.958 --> 00:24:05.750 align:middle line:84%
But for practical use,
this is good enough.

00:24:05.750 --> 00:24:08.330 align:middle line:90%


00:24:08.330 --> 00:24:11.310 align:middle line:84%
AUDIENCE: How do you convert
the 3-by-3 layer into a single

00:24:11.310 --> 00:24:12.050 align:middle line:90%
4-by-4 matrix?

00:24:12.050 --> 00:24:15.170 align:middle line:84%
[INAUDIBLE], what with the
three layers part of it?

00:24:15.170 --> 00:24:16.170 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:24:16.170 --> 00:24:17.430 align:middle line:90%
So we are coming to that.

00:24:17.430 --> 00:24:18.678 align:middle line:90%
I think we have a slide here.

00:24:18.678 --> 00:24:19.470 align:middle line:90%
Actually, we don't.

00:24:19.470 --> 00:24:19.970 align:middle line:90%
Never mind.

00:24:19.970 --> 00:24:21.010 align:middle line:90%
I'll answer that.

00:24:21.010 --> 00:24:23.670 align:middle line:90%
So here you have one filter.

00:24:23.670 --> 00:24:27.450 align:middle line:84%
You have one 3-by-3-by-3 filter,
which plugs into this thing

00:24:27.450 --> 00:24:28.030 align:middle line:90%
here.

00:24:28.030 --> 00:24:31.000 align:middle line:84%
And then it gives you
the 4 by 4 at the end.

00:24:31.000 --> 00:24:37.160 align:middle line:84%
So for one filter, we know
that by doing this operation,

00:24:37.160 --> 00:24:38.720 align:middle line:90%
we get this 4 by 4.

00:24:38.720 --> 00:24:41.840 align:middle line:84%
Let's say that you have another
filter which is also 3D.

00:24:41.840 --> 00:24:45.080 align:middle line:84%
You do that thing, you'll
get another 4 by 4.

00:24:45.080 --> 00:24:47.700 align:middle line:84%
And if you have 10 filters,
you'll get 10 of these 4 by 4s,

00:24:47.700 --> 00:24:50.550 align:middle line:84%
which then gets packaged up
into a 4-by-4-by-10 tensor.

00:24:50.550 --> 00:24:54.560 align:middle line:90%


00:24:54.560 --> 00:24:57.880 align:middle line:84%
Remember, whether
it's 2D, 3D, 10D,

00:24:57.880 --> 00:25:02.080 align:middle line:90%
what is coming out is always 2D.

00:25:02.080 --> 00:25:04.360 align:middle line:84%
Because ultimately when you
apply all this operation

00:25:04.360 --> 00:25:06.640 align:middle line:84%
at each position, you
just have one number.

00:25:06.640 --> 00:25:08.440 align:middle line:84%
And then ultimately you just
do all those things, you just

00:25:08.440 --> 00:25:10.065 align:middle line:84%
come up with a table
of numbers always.

00:25:10.065 --> 00:25:14.240 align:middle line:84%
So what's coming out is always
a 2D number table like that.

00:25:14.240 --> 00:25:16.340 align:middle line:84%
But when you have
lots of filters,

00:25:16.340 --> 00:25:18.780 align:middle line:84%
you have lots of these 2D
tables, one after the other,

00:25:18.780 --> 00:25:21.290 align:middle line:84%
and therefore, they get
packaged up into a tensor.

00:25:21.290 --> 00:25:27.040 align:middle line:90%


00:25:27.040 --> 00:25:30.080 align:middle line:84%
So textbook Chapter
8.1 has a lot

00:25:30.080 --> 00:25:32.820 align:middle line:84%
of detail on intuition,
which I think is really good.

00:25:32.820 --> 00:25:35.120 align:middle line:90%
So please try it out.

00:25:35.120 --> 00:25:39.040 align:middle line:84%
And folks, by the way,
this convolutional stuff,

00:25:39.040 --> 00:25:41.080 align:middle line:90%
it grows in the telling.

00:25:41.080 --> 00:25:42.780 align:middle line:84%
So I would encourage
you to revisit it,

00:25:42.780 --> 00:25:45.600 align:middle line:84%
revisit it a few times,
and then it slowly

00:25:45.600 --> 00:25:48.023 align:middle line:84%
becomes part of
your muscle memory.

00:25:48.023 --> 00:25:49.440 align:middle line:84%
So don't expect
to just understand

00:25:49.440 --> 00:25:51.600 align:middle line:90%
all the nuances like one shot.

00:25:51.600 --> 00:25:53.160 align:middle line:90%
Do it a few times.

00:25:53.160 --> 00:25:55.700 align:middle line:84%
It'll become wired
into your head.

00:25:55.700 --> 00:25:58.240 align:middle line:90%


00:25:58.240 --> 00:26:00.615 align:middle line:84%
The big question-- these seem
excellent, but how are we

00:26:00.615 --> 00:26:02.240 align:middle line:84%
supposed to come up
with these numbers?

00:26:02.240 --> 00:26:05.240 align:middle line:84%
In fact, traditionally
these filters actually

00:26:05.240 --> 00:26:07.185 align:middle line:90%
used to be designed by hand.

00:26:07.185 --> 00:26:08.560 align:middle line:84%
Computer vision
researchers would

00:26:08.560 --> 00:26:10.360 align:middle line:84%
invest prodigious
amounts of time,

00:26:10.360 --> 00:26:15.440 align:middle line:84%
and effort, and talent to figure
out the right kinds of filters

00:26:15.440 --> 00:26:17.860 align:middle line:84%
to use for various
specific applications.

00:26:17.860 --> 00:26:19.840 align:middle line:84%
So if you wanted to build
an application which

00:26:19.840 --> 00:26:22.700 align:middle line:84%
would look at, say, MRI
images and figure out, OK,

00:26:22.700 --> 00:26:25.960 align:middle line:84%
what kind of features should
I extract from this MRI thing

00:26:25.960 --> 00:26:29.430 align:middle line:84%
to be able to predict the
evidence for a stroke?

00:26:29.430 --> 00:26:32.483 align:middle line:84%
They would actually
hand design the filter,

00:26:32.483 --> 00:26:34.650 align:middle line:84%
try lots of different values,
and then come up with,

00:26:34.650 --> 00:26:36.850 align:middle line:84%
I got the perfect filter
for this thing here.

00:26:36.850 --> 00:26:39.010 align:middle line:84%
So that's the way
it used to be done.

00:26:39.010 --> 00:26:41.733 align:middle line:90%


00:26:41.733 --> 00:26:44.150 align:middle line:84%
But as we figured out how to
train deep networks with lots

00:26:44.150 --> 00:26:46.390 align:middle line:84%
of parameters, we
figured out things

00:26:46.390 --> 00:26:50.030 align:middle line:84%
like ReLU activation, stochastic
gradient descent, GPUs,

00:26:50.030 --> 00:26:54.255 align:middle line:84%
backprop, things like that,
this big idea emerged.

00:26:54.255 --> 00:26:55.630 align:middle line:84%
Why don't we think
of the numbers

00:26:55.630 --> 00:26:57.725 align:middle line:90%
in the filter as just weights?

00:26:57.725 --> 00:26:59.350 align:middle line:84%
And why don't we just
simply learn them

00:26:59.350 --> 00:27:02.363 align:middle line:84%
from the data using
backprop, just like we

00:27:02.363 --> 00:27:03.530 align:middle line:90%
learn all the other weights?

00:27:03.530 --> 00:27:06.550 align:middle line:90%
What's the big deal?

00:27:06.550 --> 00:27:09.470 align:middle line:84%
This simple idea, and it
feels a bit, I don't know,

00:27:09.470 --> 00:27:12.070 align:middle line:90%
blindingly obvious in hindsight.

00:27:12.070 --> 00:27:14.990 align:middle line:84%
I'm sure it was not
obvious in foresight.

00:27:14.990 --> 00:27:16.550 align:middle line:90%
This was the breakthrough.

00:27:16.550 --> 00:27:18.190 align:middle line:90%
This was the key breakthrough.

00:27:18.190 --> 00:27:20.390 align:middle line:84%
And now it's actually
possible to do this

00:27:20.390 --> 00:27:23.590 align:middle line:84%
because a convolutional
filter that we've seen

00:27:23.590 --> 00:27:26.100 align:middle line:90%
is actually just a neuron.

00:27:26.100 --> 00:27:29.540 align:middle line:84%
The underlying arithmetic of it
is just a neuronal arithmetic.

00:27:29.540 --> 00:27:33.560 align:middle line:84%
And so it just happens to
be a slightly special one.

00:27:33.560 --> 00:27:36.260 align:middle line:84%
It's actually even simpler
than a regular neuron.

00:27:36.260 --> 00:27:38.637 align:middle line:84%
And in the interest of
time, I have one or two

00:27:38.637 --> 00:27:40.220 align:middle line:84%
slides in the appendix
which tells you

00:27:40.220 --> 00:27:42.740 align:middle line:84%
exactly why it's a
neuron, so check it out.

00:27:42.740 --> 00:27:44.120 align:middle line:90%
But just take my word for it.

00:27:44.120 --> 00:27:45.940 align:middle line:84%
It's just a particular
kind of neuron.

00:27:45.940 --> 00:27:48.060 align:middle line:84%
And because it's a
particular kind of neuron,

00:27:48.060 --> 00:27:50.660 align:middle line:84%
and we know how to
work with neurons,

00:27:50.660 --> 00:27:52.220 align:middle line:84%
we know how to
work with neurons,

00:27:52.220 --> 00:27:56.180 align:middle line:84%
which means that our
entire machinery, layers,

00:27:56.180 --> 00:27:58.920 align:middle line:84%
loss functions, gradient
descent, SGD, blah, blah,

00:27:58.920 --> 00:28:01.300 align:middle line:84%
everything is
immediately applicable.

00:28:01.300 --> 00:28:04.210 align:middle line:84%
We don't have to invent any
new stuff to make it work.

00:28:04.210 --> 00:28:09.760 align:middle line:90%


00:28:09.760 --> 00:28:11.260 align:middle line:84%
AUDIENCE: Do you
initialize the data

00:28:11.260 --> 00:28:14.220 align:middle line:84%
basically between
applications or just

00:28:14.220 --> 00:28:16.280 align:middle line:84%
because the network
has different sizes?

00:28:16.280 --> 00:28:19.280 align:middle line:84%
Does computer vision versus
[INAUDIBLE] do it differently,

00:28:19.280 --> 00:28:21.060 align:middle line:84%
or it's just because
the network has

00:28:21.060 --> 00:28:22.420 align:middle line:90%
different numbers of neurons?

00:28:22.420 --> 00:28:23.420 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:28:23.420 --> 00:28:27.237 align:middle line:84%
So the initialization,
it's a good question.

00:28:27.237 --> 00:28:29.820 align:middle line:84%
Let's come back to it when we
get to something called transfer

00:28:29.820 --> 00:28:32.450 align:middle line:84%
learning, which I'm going
to get to by about 9:30.

00:28:32.450 --> 00:28:35.400 align:middle line:90%


00:28:35.400 --> 00:28:36.558 align:middle line:90%
So that's it.

00:28:36.558 --> 00:28:39.100 align:middle line:84%
So this turned out to be a huge
turning point in the computer

00:28:39.100 --> 00:28:39.940 align:middle line:90%
vision field.

00:28:39.940 --> 00:28:43.100 align:middle line:84%
And this was the massive
unlock in the year 2012.

00:28:43.100 --> 00:28:45.780 align:middle line:84%
This computer vision system
that used this technology called

00:28:45.780 --> 00:28:48.140 align:middle line:84%
AlexNet burst out
onto the world stage

00:28:48.140 --> 00:28:51.620 align:middle line:84%
because it crushed the
competition in a competition

00:28:51.620 --> 00:28:52.740 align:middle line:90%
called ImageNet.

00:28:52.740 --> 00:28:56.900 align:middle line:84%
And the previous best
score was 26% error rate.

00:28:56.900 --> 00:29:00.008 align:middle line:84%
And this thing came in
and had 16% error rate.

00:29:00.008 --> 00:29:01.800 align:middle line:84%
It's the kind of thing
where if you see it,

00:29:01.800 --> 00:29:03.467 align:middle line:84%
you'll be like, oh,
this must be a typo.

00:29:03.467 --> 00:29:05.900 align:middle line:84%
Because every year
the improvements

00:29:05.900 --> 00:29:07.960 align:middle line:90%
are very little, 0.5%, 1%.

00:29:07.960 --> 00:29:09.642 align:middle line:84%
And then this year
was 10%, and that

00:29:09.642 --> 00:29:10.850 align:middle line:90%
was because of this approach.

00:29:10.850 --> 00:29:14.040 align:middle line:90%


00:29:14.040 --> 00:29:15.740 align:middle line:84%
Now one other thing
I want to talk about

00:29:15.740 --> 00:29:20.260 align:middle line:84%
is that with every succeeding
convolutional layer,

00:29:20.260 --> 00:29:23.290 align:middle line:84%
this particular convolution, any
particular convolutional filter

00:29:23.290 --> 00:29:26.530 align:middle line:84%
is basically implicitly seeing
much more of the input image

00:29:26.530 --> 00:29:30.310 align:middle line:84%
as we go along, which means
that if, in the very beginning,

00:29:30.310 --> 00:29:33.070 align:middle line:84%
if this is the input, this
little convolutional filter,

00:29:33.070 --> 00:29:36.410 align:middle line:84%
this number here in the
first layer, let's say only

00:29:36.410 --> 00:29:40.130 align:middle line:84%
sees the top of the chimney
or whatever of this house.

00:29:40.130 --> 00:29:43.890 align:middle line:84%
But then the next layer,
remember, the next layer's input

00:29:43.890 --> 00:29:45.930 align:middle line:90%
is this particular layer.

00:29:45.930 --> 00:29:48.410 align:middle line:84%
And so this particular
little thing

00:29:48.410 --> 00:29:52.210 align:middle line:84%
here is getting information
from this whole square here.

00:29:52.210 --> 00:29:54.330 align:middle line:84%
And every one of the
points in that square

00:29:54.330 --> 00:29:57.330 align:middle line:84%
is actually something big
in the original picture.

00:29:57.330 --> 00:30:00.250 align:middle line:84%
So with every additional layer,
you're seeing more, and more,

00:30:00.250 --> 00:30:03.490 align:middle line:90%
and more of the image.

00:30:03.490 --> 00:30:05.670 align:middle line:84%
And this is a key part
of why these things work,

00:30:05.670 --> 00:30:08.410 align:middle line:84%
because you're essentially
hierarchically building

00:30:08.410 --> 00:30:10.650 align:middle line:84%
a better and better
understanding of the image.

00:30:10.650 --> 00:30:12.990 align:middle line:84%
It is the hierarchical
understanding,

00:30:12.990 --> 00:30:17.810 align:middle line:84%
the hierarchical learning that's
a very key part of the unlock.

00:30:17.810 --> 00:30:20.970 align:middle line:84%
And so if you look at networks
and what they're visualizing,

00:30:20.970 --> 00:30:23.850 align:middle line:84%
this actually is a face
detection deep network

00:30:23.850 --> 00:30:25.518 align:middle line:84%
visualized as to
what it's learning.

00:30:25.518 --> 00:30:27.810 align:middle line:84%
You'll see that the first
layer is just learning lines,

00:30:27.810 --> 00:30:29.890 align:middle line:90%
and edges, and so on, lines.

00:30:29.890 --> 00:30:32.118 align:middle line:84%
And the second layer is
actually learning edges.

00:30:32.118 --> 00:30:32.910 align:middle line:90%
Look at this thing.

00:30:32.910 --> 00:30:37.050 align:middle line:84%
It's learning to put
these lines together

00:30:37.050 --> 00:30:39.630 align:middle line:84%
to get some sort of an edge
here, another edge here.

00:30:39.630 --> 00:30:43.890 align:middle line:84%
This looks like 3/4
of somebody's ears.

00:30:43.890 --> 00:30:46.250 align:middle line:84%
And then these things
are now being assembled

00:30:46.250 --> 00:30:49.087 align:middle line:90%
to get whole faces out.

00:30:49.087 --> 00:30:51.170 align:middle line:84%
Can you imagine the
researchers who did this work?

00:30:51.170 --> 00:30:52.030 align:middle line:90%
They built the network.

00:30:52.030 --> 00:30:53.790 align:middle line:84%
It's doing really well
on detecting faces.

00:30:53.790 --> 00:30:56.248 align:middle line:84%
And they're saying, OK, let's
see what it's actually doing.

00:30:56.248 --> 00:30:58.090 align:middle line:90%
And then this picture pops up.

00:30:58.090 --> 00:31:00.490 align:middle line:90%
I mean, the goosebumps.

00:31:00.490 --> 00:31:03.090 align:middle line:90%
So pooling layers, the next one.

00:31:03.090 --> 00:31:06.030 align:middle line:84%
So far you talked about
convolutional layers.

00:31:06.030 --> 00:31:08.110 align:middle line:84%
This is the second thing,
second building block,

00:31:08.110 --> 00:31:10.670 align:middle line:90%
and then we'll go to the Colabs.

00:31:10.670 --> 00:31:12.770 align:middle line:84%
So pooling layers are
also called subsampling

00:31:12.770 --> 00:31:15.010 align:middle line:90%
or down-sampling layers.

00:31:15.010 --> 00:31:17.130 align:middle line:84%
And so the idea is that
every time a tensor

00:31:17.130 --> 00:31:20.680 align:middle line:84%
is coming out of these
convolutional layers,

00:31:20.680 --> 00:31:23.480 align:middle line:84%
we'll try to make
it slightly smaller.

00:31:23.480 --> 00:31:25.440 align:middle line:84%
Because the act of
making it smaller

00:31:25.440 --> 00:31:28.960 align:middle line:84%
will force the network to try
to summarize and learn what's

00:31:28.960 --> 00:31:32.040 align:middle line:84%
going on in this complicated
thing that's coming into it.

00:31:32.040 --> 00:31:35.160 align:middle line:84%
So I will describe
the mechanics first.

00:31:35.160 --> 00:31:40.720 align:middle line:84%
So let's say that this is the
output of a convolutional layer.

00:31:40.720 --> 00:31:42.840 align:middle line:90%
This is four of them, a 4 by 4.

00:31:42.840 --> 00:31:46.760 align:middle line:84%
So what we do is that there
are two kinds of pooling, max

00:31:46.760 --> 00:31:48.180 align:middle line:90%
pooling and average pooling.

00:31:48.180 --> 00:31:49.540 align:middle line:90%
This is called max pooling.

00:31:49.540 --> 00:31:50.865 align:middle line:90%
And the idea is really simple.

00:31:50.865 --> 00:31:53.240 align:middle line:84%
In this max pooling layer
there are no weights parameters

00:31:53.240 --> 00:31:53.823 align:middle line:90%
to be learned.

00:31:53.823 --> 00:31:56.080 align:middle line:84%
It's just a simple
arithmetic operation.

00:31:56.080 --> 00:31:58.720 align:middle line:90%
We basically take this.

00:31:58.720 --> 00:32:03.400 align:middle line:84%
We basically superimpose a
2-by-2 empty grid on the top

00:32:03.400 --> 00:32:04.200 align:middle line:90%
left.

00:32:04.200 --> 00:32:06.880 align:middle line:84%
And then we say, hey, what's the
biggest number among these four

00:32:06.880 --> 00:32:07.468 align:middle line:90%
numbers?

00:32:07.468 --> 00:32:08.760 align:middle line:90%
Well, the biggest number is 43.

00:32:08.760 --> 00:32:09.260 align:middle line:90%
Boom.

00:32:09.260 --> 00:32:11.240 align:middle line:84%
OK, I'm going to
stick a 43 here.

00:32:11.240 --> 00:32:13.600 align:middle line:84%
Then, I move my 2
by 2 to the right

00:32:13.600 --> 00:32:16.182 align:middle line:84%
so that it overlaps with
these numbers in blue.

00:32:16.182 --> 00:32:18.140 align:middle line:84%
And I say, hey, what's
the biggest number here?

00:32:18.140 --> 00:32:20.238 align:middle line:84%
OK, that's 109,
and I move it down.

00:32:20.238 --> 00:32:21.530 align:middle line:90%
What's the biggest number here?

00:32:21.530 --> 00:32:23.190 align:middle line:90%
105, stick it in here.

00:32:23.190 --> 00:32:24.050 align:middle line:90%
Biggest number here?

00:32:24.050 --> 00:32:25.450 align:middle line:90%
35, and I stick it in there.

00:32:25.450 --> 00:32:26.110 align:middle line:90%
That's it.

00:32:26.110 --> 00:32:28.932 align:middle line:90%
This is max pooling.

00:32:28.932 --> 00:32:31.390 align:middle line:84%
Similarly, there's this thing
called average pooling, where

00:32:31.390 --> 00:32:33.557 align:middle line:84%
instead of taking the maximum
of these four numbers,

00:32:33.557 --> 00:32:35.430 align:middle line:84%
we just average
the four numbers.

00:32:35.430 --> 00:32:38.470 align:middle line:84%
The average of these
four things in yellow.

00:32:38.470 --> 00:32:41.082 align:middle line:90%
Am I done?

00:32:41.082 --> 00:32:43.610 align:middle line:84%
The average of these
four numbers is 22.2.

00:32:43.610 --> 00:32:45.610 align:middle line:84%
The average of these
blue numbers is 45.5.

00:32:45.610 --> 00:32:46.830 align:middle line:90%
You get the idea.

00:32:46.830 --> 00:32:48.990 align:middle line:84%
That's it, max pooling
and average pooling.

00:32:48.990 --> 00:32:52.590 align:middle line:84%
Now as you can see,
when you apply pooling,

00:32:52.590 --> 00:32:55.910 align:middle line:84%
the number of entries
drops significantly.

00:32:55.910 --> 00:32:58.150 align:middle line:84%
The number of entries
drops significantly.

00:32:58.150 --> 00:32:59.870 align:middle line:84%
And the output from
this layer is just

00:32:59.870 --> 00:33:03.110 align:middle line:90%
fit to the next layer as usual.

00:33:03.110 --> 00:33:05.670 align:middle line:90%
There's nothing crazy going on.

00:33:05.670 --> 00:33:07.430 align:middle line:84%
So it's a way to
shrink the output

00:33:07.430 --> 00:33:09.230 align:middle line:90%
from one convolutional layer.

00:33:09.230 --> 00:33:11.450 align:middle line:84%
Before it passes on to the
next convolutional layer,

00:33:11.450 --> 00:33:13.790 align:middle line:84%
you interject with
a pooling layer.

00:33:13.790 --> 00:33:17.690 align:middle line:84%
Now I have actually, even
if I may say so myself,

00:33:17.690 --> 00:33:20.870 align:middle line:84%
a very nice handwritten
explanation of what pooling

00:33:20.870 --> 00:33:23.270 align:middle line:90%
does, the effect of pooling.

00:33:23.270 --> 00:33:25.190 align:middle line:84%
And unfortunately,
I can't get my iPad

00:33:25.190 --> 00:33:27.927 align:middle line:84%
to actually show up on
my laptop so I'm not

00:33:27.927 --> 00:33:29.010 align:middle line:90%
going to be able to do it.

00:33:29.010 --> 00:33:32.450 align:middle line:84%
But I will record a
walkthrough and I'll post it.

00:33:32.450 --> 00:33:33.050 align:middle line:90%
Check it out.

00:33:33.050 --> 00:33:37.310 align:middle line:84%
But the intuition that I try to
convey with that thing is that--

00:33:37.310 --> 00:33:38.290 align:middle line:90%
oh, sorry.

00:33:38.290 --> 00:33:39.310 align:middle line:90%
I'll come back to this.

00:33:39.310 --> 00:33:42.030 align:middle line:84%
So max pooling acts
like an OR condition.

00:33:42.030 --> 00:33:44.870 align:middle line:84%
It basically says, I
have this big picture.

00:33:44.870 --> 00:33:47.230 align:middle line:84%
So in the four things
that I'm looking at,

00:33:47.230 --> 00:33:50.270 align:middle line:84%
if there is any number
which is really high,

00:33:50.270 --> 00:33:54.067 align:middle line:84%
that means that some
feature is being detected.

00:33:54.067 --> 00:33:56.650 align:middle line:84%
The number is really high coming
out of a convolutional layer.

00:33:56.650 --> 00:34:00.510 align:middle line:84%
That means that something
somewhere fired up, lit up.

00:34:00.510 --> 00:34:03.770 align:middle line:84%
And so I'm just looking to see
if anything lit up in that part.

00:34:03.770 --> 00:34:06.781 align:middle line:84%
If it did, I'm going to
say, yeah, something lit up.

00:34:06.781 --> 00:34:08.489 align:middle line:84%
If nothing lit up,
then I'm going to say,

00:34:08.489 --> 00:34:09.590 align:middle line:90%
oh, nothing lit up.

00:34:09.590 --> 00:34:12.949 align:middle line:84%
So in that sense,
you can imagine it's

00:34:12.949 --> 00:34:15.350 align:middle line:90%
acting like an OR condition.

00:34:15.350 --> 00:34:16.159 align:middle line:90%
Anything fired up?

00:34:16.159 --> 00:34:16.719 align:middle line:90%
Anything fired up?

00:34:16.719 --> 00:34:17.239 align:middle line:90%
Anything fired up?

00:34:17.239 --> 00:34:17.620 align:middle line:90%
Anything?

00:34:17.620 --> 00:34:17.730 align:middle line:90%
Yes?

00:34:17.730 --> 00:34:18.230 align:middle line:90%
OK.

00:34:18.230 --> 00:34:19.000 align:middle line:90%
Otherwise, no.

00:34:19.000 --> 00:34:22.340 align:middle line:90%


00:34:22.340 --> 00:34:24.620 align:middle line:84%
Sadly, I can't
switch to Notability.

00:34:24.620 --> 00:34:26.639 align:middle line:84%
So it acts like a
feature detector.

00:34:26.639 --> 00:34:28.300 align:middle line:84%
So if you have lots
of things going on

00:34:28.300 --> 00:34:29.900 align:middle line:84%
in a particular
picture, you want

00:34:29.900 --> 00:34:32.340 align:middle line:84%
to be able to summarize and
aggregate all the things that

00:34:32.340 --> 00:34:34.242 align:middle line:84%
are going on so
that you can say,

00:34:34.242 --> 00:34:36.659 align:middle line:84%
you may have a big picture
with lots of things lighting up

00:34:36.659 --> 00:34:37.400 align:middle line:90%
here and there.

00:34:37.400 --> 00:34:39.442 align:middle line:84%
But you want to step back
and say, you know what?

00:34:39.442 --> 00:34:42.460 align:middle line:84%
In this picture, the top
left, nothing lit up.

00:34:42.460 --> 00:34:44.800 align:middle line:90%
The top right, something lit up.

00:34:44.800 --> 00:34:46.199 align:middle line:90%
Bottom left, something lit up.

00:34:46.199 --> 00:34:48.300 align:middle line:84%
And the bottom right,
nothing lit up.

00:34:48.300 --> 00:34:51.739 align:middle line:84%
So you're operating at a
higher level of abstraction.

00:34:51.739 --> 00:34:55.179 align:middle line:90%
That's the effect of pooling.

00:34:55.179 --> 00:35:00.220 align:middle line:84%
AUDIENCE: But once you use
spatial elements, [INAUDIBLE]?

00:35:00.220 --> 00:35:03.060 align:middle line:84%
RAMA RAMAKRISHNAN: You don't,
because what you're actually

00:35:03.060 --> 00:35:06.220 align:middle line:84%
saying is, the top
left has this thing.

00:35:06.220 --> 00:35:08.580 align:middle line:84%
You already know it
is in the top left.

00:35:08.580 --> 00:35:12.140 align:middle line:84%
And you've already moved up
to that level of abstraction.

00:35:12.140 --> 00:35:15.890 align:middle line:84%
So the fact, for example, in the
top left there is a human eye

00:35:15.890 --> 00:35:18.258 align:middle line:90%
and there is a circle detector.

00:35:18.258 --> 00:35:20.550 align:middle line:84%
It's going to fire up and
saying, hey, in the top left,

00:35:20.550 --> 00:35:22.677 align:middle line:90%
there is an eye lit up.

00:35:22.677 --> 00:35:24.510 align:middle line:84%
So you're not looking
at the pixels anymore.

00:35:24.510 --> 00:35:26.590 align:middle line:84%
You're already operating at a
higher level of abstraction,

00:35:26.590 --> 00:35:27.923 align:middle line:90%
and that's how we get around it.

00:35:27.923 --> 00:35:30.330 align:middle line:84%
But this proceeds slowly
and incrementally, which

00:35:30.330 --> 00:35:31.920 align:middle line:84%
is why you have
these big networks.

00:35:31.920 --> 00:35:35.690 align:middle line:90%


00:35:35.690 --> 00:35:39.010 align:middle line:84%
So now, as we saw, some
successive convolutional layers

00:35:39.010 --> 00:35:41.610 align:middle line:84%
can see more and more
of the original image.

00:35:41.610 --> 00:35:43.210 align:middle line:84%
The max pooling
layers that follow

00:35:43.210 --> 00:35:45.010 align:middle line:84%
them can detect if
a feature exists

00:35:45.010 --> 00:35:47.610 align:middle line:84%
in more and more of the
original input as well.

00:35:47.610 --> 00:35:49.710 align:middle line:84%
So by the time you get to
the seventh, and eighth,

00:35:49.710 --> 00:35:52.750 align:middle line:84%
ninth layers, and so on, this
thing is actually really smart.

00:35:52.750 --> 00:35:56.145 align:middle line:84%
It's operating at a very
high level of abstraction.

00:35:56.145 --> 00:35:57.770 align:middle line:84%
You can think of it
as it has basically

00:35:57.770 --> 00:36:01.390 align:middle line:84%
tagged all the features of that
image at various resolutions,

00:36:01.390 --> 00:36:04.557 align:middle line:90%
and it can work with it.

00:36:04.557 --> 00:36:06.390 align:middle line:84%
AUDIENCE: Is there a
trade off between doing

00:36:06.390 --> 00:36:10.570 align:middle line:84%
the pre-processing as opposed
to adding additional convolution

00:36:10.570 --> 00:36:11.070 align:middle line:90%
layers?

00:36:11.070 --> 00:36:12.890 align:middle line:84%
I'm thinking if you
had a video, turning it

00:36:12.890 --> 00:36:16.890 align:middle line:84%
into black-and-white static
images in a sequence, as opposed

00:36:16.890 --> 00:36:20.690 align:middle line:84%
to shoving in a color
video with a time,

00:36:20.690 --> 00:36:22.270 align:middle line:90%
the parameters have to expand.

00:36:22.270 --> 00:36:23.590 align:middle line:90%
Is there a trade-off element?

00:36:23.590 --> 00:36:25.257 align:middle line:84%
RAMA RAMAKRISHNAN:
There is a trade off.

00:36:25.257 --> 00:36:28.050 align:middle line:84%
If your particular
data set and input

00:36:28.050 --> 00:36:30.570 align:middle line:84%
has some-- there is some
very important domain

00:36:30.570 --> 00:36:33.890 align:middle line:84%
knowledge that you want
to encode when you set up

00:36:33.890 --> 00:36:36.570 align:middle line:84%
the network, so that the
network doesn't waste

00:36:36.570 --> 00:36:39.730 align:middle line:84%
its capacity learning
things that have to be true,

00:36:39.730 --> 00:36:41.650 align:middle line:90%
then, yeah, modify the input.

00:36:41.650 --> 00:36:45.530 align:middle line:84%
But if you're not sure, then
you want just the network,

00:36:45.530 --> 00:36:47.290 align:middle line:84%
learn whatever it
can, as long as it's

00:36:47.290 --> 00:36:50.750 align:middle line:84%
focused on predicting
accuracy as well as possible,

00:36:50.750 --> 00:36:51.710 align:middle line:90%
then just let it be.

00:36:51.710 --> 00:36:56.750 align:middle line:90%


00:36:56.750 --> 00:36:57.950 align:middle line:90%
So that's the basic idea.

00:36:57.950 --> 00:37:01.490 align:middle line:84%
And again, I'm sorry this is
Notability thing is not working.

00:37:01.490 --> 00:37:04.090 align:middle line:84%
But take a look to
really understand

00:37:04.090 --> 00:37:06.850 align:middle line:84%
how this max pooling
thing business works.

00:37:06.850 --> 00:37:09.290 align:middle line:90%
Oh, I think I skipped over this.

00:37:09.290 --> 00:37:12.940 align:middle line:84%
So when you have something
like this, so this, let's say,

00:37:12.940 --> 00:37:15.400 align:middle line:84%
is a tensor coming out of
some convolutional layer

00:37:15.400 --> 00:37:18.240 align:middle line:84%
and its size is
224 by 224 by 64.

00:37:18.240 --> 00:37:20.102 align:middle line:84%
Then you apply something
like a pooling.

00:37:20.102 --> 00:37:22.560 align:middle line:84%
The thing I want to point out
is that the pooling will work

00:37:22.560 --> 00:37:25.840 align:middle line:90%
with every slice of the tensor.

00:37:25.840 --> 00:37:28.880 align:middle line:84%
So if the tensor is
224 by 224 by 64,

00:37:28.880 --> 00:37:31.160 align:middle line:84%
it has a depth of 64,
which is basically

00:37:31.160 --> 00:37:34.780 align:middle line:84%
like saying it's got 64
tables of 224 by 224.

00:37:34.780 --> 00:37:38.440 align:middle line:84%
And the pooling will work on
every one of those tables, which

00:37:38.440 --> 00:37:43.340 align:middle line:84%
means that you'll still have
64 things at the very end.

00:37:43.340 --> 00:37:46.840 align:middle line:84%
It's just that every one of the
things of the 64, the 224 by 224

00:37:46.840 --> 00:37:49.800 align:middle line:90%
will shrink to 112 by 112.

00:37:49.800 --> 00:37:52.382 align:middle line:84%
So each table shrinks
due to pooling,

00:37:52.382 --> 00:37:54.090 align:middle line:84%
but the number of
tables does not change.

00:37:54.090 --> 00:37:59.960 align:middle line:90%


00:37:59.960 --> 00:38:04.512 align:middle line:84%
By the way, this link here
is a beautiful explanation

00:38:04.512 --> 00:38:06.720 align:middle line:84%
of all these things with a
little bit more complexity

00:38:06.720 --> 00:38:09.390 align:middle line:84%
as well from a course
taught at Stanford in 2018,

00:38:09.390 --> 00:38:11.360 align:middle line:90%
or 2019, or something, I forget.

00:38:11.360 --> 00:38:13.860 align:middle line:84%
So just check it out if you're
curious about this stuff.

00:38:13.860 --> 00:38:14.580 align:middle line:90%
It's really good.

00:38:14.580 --> 00:38:19.060 align:middle line:90%


00:38:19.060 --> 00:38:22.400 align:middle line:84%
So that brings us to the
architecture of a basic CNN.

00:38:22.400 --> 00:38:25.780 align:middle line:84%
And so what we do
is we have an input.

00:38:25.780 --> 00:38:26.740 align:middle line:90%
We take that input.

00:38:26.740 --> 00:38:29.600 align:middle line:84%
We run it through a bunch
of convolutional and pooling

00:38:29.600 --> 00:38:30.440 align:middle line:90%
layers.

00:38:30.440 --> 00:38:32.320 align:middle line:84%
So this is a
convolutional layer.

00:38:32.320 --> 00:38:37.400 align:middle line:84%
And then we pool it, which
is why it has shrunk in size.

00:38:37.400 --> 00:38:39.620 align:middle line:84%
And then it goes through
another convolutional layer.

00:38:39.620 --> 00:38:42.280 align:middle line:84%
Then, we pool it,
which is shrunk again,

00:38:42.280 --> 00:38:43.820 align:middle line:90%
and then it keeps on doing it.

00:38:43.820 --> 00:38:45.480 align:middle line:90%
So we have a series of these.

00:38:45.480 --> 00:38:47.560 align:middle line:84%
These are called
convolutional blocks.

00:38:47.560 --> 00:38:49.680 align:middle line:84%
So a convolutional
block is typically

00:38:49.680 --> 00:38:52.960 align:middle line:84%
one to two convolutional layers
followed by a pooling layer.

00:38:52.960 --> 00:38:57.580 align:middle line:84%
So you have a series of
convolutional blocks.

00:38:57.580 --> 00:39:01.360 align:middle line:84%
And the thing to notice is that,
as you go further and further

00:39:01.360 --> 00:39:04.760 align:middle line:84%
in the network, the
blocks will actually

00:39:04.760 --> 00:39:08.120 align:middle line:84%
get smaller and smaller
because of max pooling.

00:39:08.120 --> 00:39:12.260 align:middle line:84%
It gets smaller and smaller, but
they'll get deeper and deeper.

00:39:12.260 --> 00:39:14.790 align:middle line:90%


00:39:14.790 --> 00:39:16.390 align:middle line:84%
And we have
empirically figured out

00:39:16.390 --> 00:39:20.430 align:middle line:84%
that actually that model of
reducing the size, the height

00:39:20.430 --> 00:39:22.350 align:middle line:84%
and the width, but
then making it deeper,

00:39:22.350 --> 00:39:25.510 align:middle line:84%
tends to work really
well in practice.

00:39:25.510 --> 00:39:30.630 align:middle line:84%
And so, in fact, apologies to
the live stream that I can't use

00:39:30.630 --> 00:39:31.130 align:middle line:90%
iPad.

00:39:31.130 --> 00:39:32.463 align:middle line:90%
I'm going to do it on the board.

00:39:32.463 --> 00:39:35.990 align:middle line:90%


00:39:35.990 --> 00:39:38.590 align:middle line:84%
So let's say that you
have a picture, which

00:39:38.590 --> 00:39:44.830 align:middle line:90%
is coming in as 224, 224.

00:39:44.830 --> 00:39:47.950 align:middle line:84%
And then you have,
say, three of them

00:39:47.950 --> 00:39:49.530 align:middle line:90%
because it's a color picture.

00:39:49.530 --> 00:39:52.350 align:middle line:90%
So you have three of them.

00:39:52.350 --> 00:39:54.350 align:middle line:90%
Can you folks see this OK?

00:39:54.350 --> 00:39:55.110 align:middle line:90%
All right.

00:39:55.110 --> 00:39:57.670 align:middle line:84%
So let's say this is
the input coming in.

00:39:57.670 --> 00:40:00.790 align:middle line:84%
And ResNet, which is a
very famous network that

00:40:00.790 --> 00:40:03.750 align:middle line:84%
we're actually going to
work with in a few minutes,

00:40:03.750 --> 00:40:06.790 align:middle line:84%
when it actually gets done with
all this convolution pooling

00:40:06.790 --> 00:40:13.980 align:middle line:84%
business, the final tensor that
it has is actually of shape 7

00:40:13.980 --> 00:40:23.920 align:middle line:90%
by 7, but it is 2,048 long.

00:40:23.920 --> 00:40:26.700 align:middle line:84%
It has processed something
which is 224, 224 times 3,

00:40:26.700 --> 00:40:29.160 align:middle line:84%
to a much smaller height
and width, just 7 by 7.

00:40:29.160 --> 00:40:32.940 align:middle line:84%
But it has gotten much
deeper 2,048 layers.

00:40:32.940 --> 00:40:35.060 align:middle line:84%
This is a numerical
example of what

00:40:35.060 --> 00:40:37.820 align:middle line:84%
I am talking about there in
terms of, as you go along,

00:40:37.820 --> 00:40:39.670 align:middle line:90%
things get smaller but deeper.

00:40:39.670 --> 00:40:44.460 align:middle line:90%


00:40:44.460 --> 00:40:46.020 align:middle line:90%
Yes?

00:40:46.020 --> 00:40:49.740 align:middle line:84%
AUDIENCE: Is the reason that
it gets deeper because each--

00:40:49.740 --> 00:40:52.660 align:middle line:84%
it gets deeper because each
layer has a single feature that

00:40:52.660 --> 00:40:55.160 align:middle line:84%
is picked up, and then it gets
stacked on top of each other?

00:40:55.160 --> 00:40:55.860 align:middle line:84%
RAMA RAMAKRISHNAN:
It's not so much

00:40:55.860 --> 00:40:58.200 align:middle line:84%
that each layer has picking
up a single feature.

00:40:58.200 --> 00:41:02.220 align:middle line:84%
It's more that,
basically, the way I

00:41:02.220 --> 00:41:06.540 align:middle line:84%
think about it is that the
number of atomic features

00:41:06.540 --> 00:41:09.460 align:middle line:84%
that you may want to detect
are probably not that many.

00:41:09.460 --> 00:41:12.760 align:middle line:84%
Lines, curves, gradations, and
color, and things like that.

00:41:12.760 --> 00:41:14.300 align:middle line:84%
But the way in which
you can combine

00:41:14.300 --> 00:41:18.540 align:middle line:84%
these atomic features to
depict real world things

00:41:18.540 --> 00:41:20.140 align:middle line:90%
is combinatorial.

00:41:20.140 --> 00:41:23.000 align:middle line:84%
It's sort of like, I
have 10 kinds of atoms.

00:41:23.000 --> 00:41:25.020 align:middle line:84%
How many molecules
can I make from it?

00:41:25.020 --> 00:41:27.740 align:middle line:84%
You can make a lot of molecules
from those 10 atoms, which

00:41:27.740 --> 00:41:30.660 align:middle line:84%
means that you better give
the network the ability

00:41:30.660 --> 00:41:32.900 align:middle line:84%
to capture more and more
of these possible things

00:41:32.900 --> 00:41:35.380 align:middle line:84%
that the real world
can come up with.

00:41:35.380 --> 00:41:39.120 align:middle line:84%
And so as the depth increases,
you have more filters.

00:41:39.120 --> 00:41:41.540 align:middle line:84%
And every filter
now has the ability

00:41:41.540 --> 00:41:44.480 align:middle line:84%
to pick up some combinatorial
combination of what's coming in.

00:41:44.480 --> 00:41:49.757 align:middle line:90%


00:41:49.757 --> 00:41:51.840 align:middle line:84%
AUDIENCE: Sorry, a quick
question related to this.

00:41:51.840 --> 00:41:53.940 align:middle line:84%
So right now our
model is being trained

00:41:53.940 --> 00:41:56.780 align:middle line:84%
to detect specific features,
like a line, eye color,

00:41:56.780 --> 00:41:57.940 align:middle line:90%
or something of this sort.

00:41:57.940 --> 00:42:00.900 align:middle line:84%
But still, it doesn't
have meaning to this.

00:42:00.900 --> 00:42:06.230 align:middle line:84%
Still, they don't know if that
arc is a sun or is an eye.

00:42:06.230 --> 00:42:07.230 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:42:07.230 --> 00:42:09.270 align:middle line:84%
So we don't tell
it what to learn.

00:42:09.270 --> 00:42:10.690 align:middle line:90%
It just learns.

00:42:10.690 --> 00:42:12.450 align:middle line:84%
All we tell it is,
make sure that you

00:42:12.450 --> 00:42:14.010 align:middle line:90%
minimize the loss function.

00:42:14.010 --> 00:42:15.830 align:middle line:84%
Now, once it has
finished learning,

00:42:15.830 --> 00:42:18.330 align:middle line:84%
if it's a good network,
it has good accuracy.

00:42:18.330 --> 00:42:20.170 align:middle line:90%
Then, we can introspect.

00:42:20.170 --> 00:42:23.170 align:middle line:84%
We can peek into the internals
and try to understand,

00:42:23.170 --> 00:42:24.530 align:middle line:90%
what is it learning?

00:42:24.530 --> 00:42:27.180 align:middle line:84%
And sometimes, like you saw
in the face detection example,

00:42:27.180 --> 00:42:28.930 align:middle line:84%
it's actually learning
interesting things,

00:42:28.930 --> 00:42:32.290 align:middle line:84%
like basic lines, and edges, and
then slowly, more complicated

00:42:32.290 --> 00:42:32.790 align:middle line:90%
shapes.

00:42:32.790 --> 00:42:35.550 align:middle line:84%
And then finally,
entire human faces.

00:42:35.550 --> 00:42:37.268 align:middle line:84%
Sometimes it may not
be understandable.

00:42:37.268 --> 00:42:38.810 align:middle line:84%
AUDIENCE: And the
way it's doing this

00:42:38.810 --> 00:42:41.960 align:middle line:84%
is by constructing
features, not by learning.

00:42:41.960 --> 00:42:44.460 align:middle line:84%
RAMA RAMAKRISHNAN: How do you
figure out what it's learning?

00:42:44.460 --> 00:42:45.050 align:middle line:90%
AUDIENCE: Yes.

00:42:45.050 --> 00:42:46.258 align:middle line:90%
RAMA RAMAKRISHNAN: Oh, I see.

00:42:46.258 --> 00:42:49.090 align:middle line:84%
So I'm going to give a
reference in just a few minutes.

00:42:49.090 --> 00:42:50.030 align:middle line:90%
Read the paper.

00:42:50.030 --> 00:42:52.197 align:middle line:84%
That was one of the first
ones to actually visualize

00:42:52.197 --> 00:42:53.630 align:middle line:90%
what these things are learning.

00:42:53.630 --> 00:42:55.750 align:middle line:84%
And that will give you an
idea of how it actually works.

00:42:55.750 --> 00:42:57.583 align:middle line:84%
And I'm also happy to
talk about it offline.

00:42:57.583 --> 00:43:00.310 align:middle line:84%
It's a bit of a tangent, but
it's a really rich tangent.

00:43:00.310 --> 00:43:02.095 align:middle line:84%
So if I keep talking
about it, I'll

00:43:02.095 --> 00:43:03.470 align:middle line:84%
end up spending
10 minutes on it.

00:43:03.470 --> 00:43:04.512 align:middle line:90%
So I'm going to back off.

00:43:04.512 --> 00:43:09.720 align:middle line:90%


00:43:09.720 --> 00:43:13.520 align:middle line:84%
So now, once we do
that, now we are back

00:43:13.520 --> 00:43:16.240 align:middle line:84%
in familiar territory, where
we take whatever tensor

00:43:16.240 --> 00:43:18.880 align:middle line:84%
is coming out from these
convolutional operations

00:43:18.880 --> 00:43:22.600 align:middle line:84%
and pooling operations, and then
we just flatten them, only now

00:43:22.600 --> 00:43:23.960 align:middle line:90%
into a long vector.

00:43:23.960 --> 00:43:26.200 align:middle line:84%
And once we flatten
them, we can connect them

00:43:26.200 --> 00:43:30.260 align:middle line:84%
to some good, old dense
layers, like we know how to do.

00:43:30.260 --> 00:43:33.120 align:middle line:84%
And then we finally
connect them with whatever

00:43:33.120 --> 00:43:34.800 align:middle line:90%
output layer you want.

00:43:34.800 --> 00:43:36.600 align:middle line:84%
In this case, this
example is using

00:43:36.600 --> 00:43:40.440 align:middle line:84%
some multi-class classification
of classifying images

00:43:40.440 --> 00:43:42.660 align:middle line:84%
to what kind of automobile
or whatever it is.

00:43:42.660 --> 00:43:43.980 align:middle line:90%
So it's like a softmax.

00:43:43.980 --> 00:43:45.460 align:middle line:84%
So this is the
general framework.

00:43:45.460 --> 00:43:48.680 align:middle line:90%


00:43:48.680 --> 00:43:50.900 align:middle line:90%
Any questions?

00:43:50.900 --> 00:43:54.600 align:middle line:90%


00:43:54.600 --> 00:43:55.720 align:middle line:90%
Yeah?

00:43:55.720 --> 00:43:59.110 align:middle line:84%
AUDIENCE: Can you explain again
how the depth increases exactly?

00:43:59.110 --> 00:44:01.360 align:middle line:84%
RAMA RAMAKRISHNAN: Oh, the
depth increases because you

00:44:01.360 --> 00:44:02.580 align:middle line:90%
decide what the depth is.

00:44:02.580 --> 00:44:05.960 align:middle line:84%
So when you add a
convolutional layer,

00:44:05.960 --> 00:44:07.680 align:middle line:84%
you decide how many
filters it has.

00:44:07.680 --> 00:44:11.120 align:middle line:84%
So you just keep adding more
and more filters the later

00:44:11.120 --> 00:44:13.280 align:middle line:90%
on you go in the network.

00:44:13.280 --> 00:44:14.300 align:middle line:90%
So it's in your hands.

00:44:14.300 --> 00:44:16.640 align:middle line:84%
So remember, the number of
neurons in a hidden layer

00:44:16.640 --> 00:44:18.120 align:middle line:90%
is in your control.

00:44:18.120 --> 00:44:20.300 align:middle line:84%
Similarly, the number of
filters is in your control.

00:44:20.300 --> 00:44:22.560 align:middle line:90%
It's a design choice.

00:44:22.560 --> 00:44:24.760 align:middle line:84%
And we design it so
that the later we go,

00:44:24.760 --> 00:44:25.940 align:middle line:90%
the more depth we have.

00:44:25.940 --> 00:44:30.200 align:middle line:84%
AUDIENCE: So you
stack layers with--

00:44:30.200 --> 00:44:32.880 align:middle line:84%
each of those layers
has a different filter

00:44:32.880 --> 00:44:34.600 align:middle line:90%
applied to the end output?

00:44:34.600 --> 00:44:35.600 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:44:35.600 --> 00:44:37.333 align:middle line:90%
A layer is made up of filters.

00:44:37.333 --> 00:44:40.000 align:middle line:84%
And so the depth just comes from
having lots, and lots, and lots

00:44:40.000 --> 00:44:40.520 align:middle line:90%
of filters.

00:44:40.520 --> 00:44:42.020 align:middle line:84%
And you get to
choose what they are.

00:44:42.020 --> 00:44:45.080 align:middle line:90%


00:44:45.080 --> 00:44:48.920 align:middle line:84%
So now let's go to the
Fashion MNIST Colab

00:44:48.920 --> 00:44:52.240 align:middle line:84%
that I did the video walkthrough
on, and then actually solve

00:44:52.240 --> 00:44:55.880 align:middle line:84%
it using a
convolutional network.

00:44:55.880 --> 00:44:56.380 align:middle line:90%
All right.

00:44:56.380 --> 00:44:56.880 align:middle line:90%
Cool.

00:44:56.880 --> 00:44:59.560 align:middle line:84%
So at this point I'm going to
zip through some of this stuff

00:44:59.560 --> 00:45:01.950 align:middle line:84%
because the preliminaries
have to be done.

00:45:01.950 --> 00:45:03.610 align:middle line:90%
Import all these packages.

00:45:03.610 --> 00:45:05.570 align:middle line:90%
Set the random seed here.

00:45:05.570 --> 00:45:06.070 align:middle line:90%
Great.

00:45:06.070 --> 00:45:08.370 align:middle line:84%
And then we will load
the MNIST data set,

00:45:08.370 --> 00:45:10.910 align:middle line:84%
just like I did in
the Colab yesterday.

00:45:10.910 --> 00:45:13.790 align:middle line:84%
We create these little
labels, and then

00:45:13.790 --> 00:45:15.430 align:middle line:84%
we just have these
standard functions

00:45:15.430 --> 00:45:18.870 align:middle line:84%
to plot accuracy and loss
that we've been using so far.

00:45:18.870 --> 00:45:21.030 align:middle line:84%
All right, now we come to
the convolutional thing.

00:45:21.030 --> 00:45:25.190 align:middle line:84%
And so as before, we're
going to divide it

00:45:25.190 --> 00:45:28.310 align:middle line:84%
by 255 to normalize
everything to a 0 to 1 range.

00:45:28.310 --> 00:45:30.710 align:middle line:84%
Let's confirm to make sure
that the data, nothing has

00:45:30.710 --> 00:45:32.570 align:middle line:90%
gotten tampered with.

00:45:32.570 --> 00:45:34.410 align:middle line:90%
Yes, we have 60,000 images.

00:45:34.410 --> 00:45:36.710 align:middle line:84%
Each one is 28 by 28
in the training set.

00:45:36.710 --> 00:45:41.510 align:middle line:84%
Now convolutional networks,
they expect the input

00:45:41.510 --> 00:45:44.510 align:middle line:84%
to have three
channels, or it expects

00:45:44.510 --> 00:45:48.390 align:middle line:84%
to have an additional thing,
which is like a channel.

00:45:48.390 --> 00:45:50.750 align:middle line:84%
The color images
have three channels.

00:45:50.750 --> 00:45:52.930 align:middle line:84%
But black and white images
have only one channel,

00:45:52.930 --> 00:45:54.450 align:middle line:90%
one table of numbers.

00:45:54.450 --> 00:45:58.350 align:middle line:84%
So instead of saying 28 by 28,
we tell the convolutional layer,

00:45:58.350 --> 00:46:01.660 align:middle line:90%
expect 28 by 28 by 1.

00:46:01.660 --> 00:46:03.040 align:middle line:84%
It's the same
thing conceptually,

00:46:03.040 --> 00:46:05.700 align:middle line:84%
but that's the format
that it expects.

00:46:05.700 --> 00:46:08.200 align:middle line:90%
And so we go here.

00:46:08.200 --> 00:46:10.180 align:middle line:84%
And then we say,
all right, there's

00:46:10.180 --> 00:46:11.920 align:middle line:90%
a thing called expand_dimension.

00:46:11.920 --> 00:46:13.840 align:middle line:84%
I'm just telling it to
expand its dimension.

00:46:13.840 --> 00:46:17.000 align:middle line:84%
And once I do that, you can
see here, it's still 60,000.

00:46:17.000 --> 00:46:20.800 align:middle line:84%
But instead of 28 by 28, it
has become 28 by 28 by 1.

00:46:20.800 --> 00:46:21.820 align:middle line:90%
Same thing.

00:46:21.820 --> 00:46:24.450 align:middle line:84%
Now, let's define
our very first CNN.

00:46:24.450 --> 00:46:27.700 align:middle line:90%


00:46:27.700 --> 00:46:31.940 align:middle line:84%
As before, the input is
just keras.Input, as before.

00:46:31.940 --> 00:46:33.158 align:middle line:90%
No difference here.

00:46:33.158 --> 00:46:34.200 align:middle line:90%
And we tell it the shape.

00:46:34.200 --> 00:46:37.340 align:middle line:84%
And the shape is, of
course, just 28 by 28 by 1.

00:46:37.340 --> 00:46:39.220 align:middle line:90%
That's what I have here.

00:46:39.220 --> 00:46:43.860 align:middle line:84%
And then we come to the
first convolutional block.

00:46:43.860 --> 00:46:45.820 align:middle line:90%
And this is the key thing.

00:46:45.820 --> 00:46:49.740 align:middle line:84%
If you want to tell Keras to
use a convolutional layer,

00:46:49.740 --> 00:46:53.223 align:middle line:84%
you use this keyword,
layers.Conv2D.

00:46:53.223 --> 00:46:55.140 align:middle line:84%
And from this you can
probably also figure out

00:46:55.140 --> 00:46:57.460 align:middle line:84%
that there's a Conv1D,
and there's a Conv3D,

00:46:57.460 --> 00:47:00.680 align:middle line:84%
and so on, and so
forth, which, explore.

00:47:00.680 --> 00:47:01.940 align:middle line:90%
It's really good stuff.

00:47:01.940 --> 00:47:05.180 align:middle line:84%
But for image processing,
Conv2D is all you need.

00:47:05.180 --> 00:47:08.200 align:middle line:84%
And now we tell it how
many filters you want.

00:47:08.200 --> 00:47:10.720 align:middle line:84%
So we decide on the
number of filters.

00:47:10.720 --> 00:47:13.060 align:middle line:84%
So I've decided to
have 32 filters.

00:47:13.060 --> 00:47:16.820 align:middle line:84%
And then we also have to
decide the size of the filter.

00:47:16.820 --> 00:47:19.180 align:middle line:90%
The simplest size is 2 by 2.

00:47:19.180 --> 00:47:22.587 align:middle line:84%
So I'm just going to go with
that, kernel_size is 2 by 2.

00:47:22.587 --> 00:47:24.420 align:middle line:84%
And then the activation
is, of course, ReLU.

00:47:24.420 --> 00:47:27.580 align:middle line:84%
I give it a name, convolution
1, and then I feed it the input.

00:47:27.580 --> 00:47:30.780 align:middle line:84%
And then once I do that, I
followed up with a pooling layer

00:47:30.780 --> 00:47:33.620 align:middle line:90%
where I use MaxPool2D.

00:47:33.620 --> 00:47:36.000 align:middle line:84%
And MaxPool2D, you just
literally pass the input.

00:47:36.000 --> 00:47:37.940 align:middle line:84%
You get the output
back, it just shrinks

00:47:37.940 --> 00:47:39.420 align:middle line:90%
everything using pooling.

00:47:39.420 --> 00:47:41.620 align:middle line:84%
So that is the first
convolutional block.

00:47:41.620 --> 00:47:43.820 align:middle line:90%
And you know what?

00:47:43.820 --> 00:47:45.240 align:middle line:90%
I know how to cut and paste.

00:47:45.240 --> 00:47:48.635 align:middle line:84%
Cut and paste, I get the
second convolutional block.

00:47:48.635 --> 00:47:50.260 align:middle line:84%
Here is the second
convolutional block.

00:47:50.260 --> 00:47:52.780 align:middle line:84%
And I know in just the
lecture I mentioned

00:47:52.780 --> 00:47:55.880 align:middle line:84%
that, as you go deeper,
you get more depth to it.

00:47:55.880 --> 00:47:57.488 align:middle line:84%
But this is just
a starting point.

00:47:57.488 --> 00:47:59.030 align:middle line:84%
I'm just going to
use the same depth.

00:47:59.030 --> 00:47:59.630 align:middle line:90%
It's not a big deal.

00:47:59.630 --> 00:48:01.047 align:middle line:84%
It's a simple
problem, so which is

00:48:01.047 --> 00:48:04.490 align:middle line:84%
why in the second convolutional
block, I'm still using only 32.

00:48:04.490 --> 00:48:06.510 align:middle line:84%
But you can totally go
to 64, for instance,

00:48:06.510 --> 00:48:07.730 align:middle line:90%
to make it much deeper.

00:48:07.730 --> 00:48:11.970 align:middle line:84%
And once I do that, I
finally come to the point

00:48:11.970 --> 00:48:14.290 align:middle line:84%
where I flatten everything
into a long vector.

00:48:14.290 --> 00:48:19.450 align:middle line:84%
Then, I connect it to one
dense layer of 256 neurons.

00:48:19.450 --> 00:48:22.130 align:middle line:84%
And then finally, I
come to the softmax

00:48:22.130 --> 00:48:25.950 align:middle line:84%
where I have 10 outputs, 10
categories of clothing, softmax.

00:48:25.950 --> 00:48:28.750 align:middle line:84%
And then I tell Keras, OK,
take this input and the output.

00:48:28.750 --> 00:48:30.130 align:middle line:90%
String them up together.

00:48:30.130 --> 00:48:32.290 align:middle line:90%
Define a model for me.

00:48:32.290 --> 00:48:32.910 align:middle line:90%
That's it.

00:48:32.910 --> 00:48:34.350 align:middle line:84%
That's the
convolutional network.

00:48:34.350 --> 00:48:36.250 align:middle line:84%
The new concepts
we are seeing here

00:48:36.250 --> 00:48:39.010 align:middle line:84%
are Conv2D for the
convolutional layer,

00:48:39.010 --> 00:48:42.810 align:middle line:84%
and then MaxPool2D for
the max pooling layer.

00:48:42.810 --> 00:48:44.250 align:middle line:90%
That's it.

00:48:44.250 --> 00:48:45.010 align:middle line:90%
Coming.

00:48:45.010 --> 00:48:47.730 align:middle line:84%
So let me just run this
thing, make sure it runs.

00:48:47.730 --> 00:48:48.230 align:middle line:90%
OK.

00:48:48.230 --> 00:48:48.770 align:middle line:90%
Good.

00:48:48.770 --> 00:48:50.290 align:middle line:90%
Yeah?

00:48:50.290 --> 00:48:52.650 align:middle line:84%
AUDIENCE: How do you
decide when to flatten?

00:48:52.650 --> 00:48:54.450 align:middle line:84%
And would there
ever be a situation

00:48:54.450 --> 00:48:57.810 align:middle line:84%
in which we just use the
method that we used before

00:48:57.810 --> 00:48:59.017 align:middle line:90%
and not using CNN?

00:48:59.017 --> 00:49:01.350 align:middle line:84%
RAMA RAMAKRISHNAN: Well, we
already tried it with MNIST.

00:49:01.350 --> 00:49:02.190 align:middle line:90%
We didn't use the CNN.

00:49:02.190 --> 00:49:03.190 align:middle line:90%
We just flattened it right away.

00:49:03.190 --> 00:49:03.870 align:middle line:90%
AUDIENCE: Yeah, it did work.

00:49:03.870 --> 00:49:04.990 align:middle line:90%
RAMA RAMAKRISHNAN: It's not bad.

00:49:04.990 --> 00:49:07.063 align:middle line:84%
But we are like, can we
do better than 85, or 88,

00:49:07.063 --> 00:49:08.230 align:middle line:90%
or whatever the percent was?

00:49:08.230 --> 00:49:10.650 align:middle line:90%
But we are working with images.

00:49:10.650 --> 00:49:12.970 align:middle line:84%
It's typically a good idea
to just start with the CNN

00:49:12.970 --> 00:49:15.095 align:middle line:84%
straight off the bat because
you're not losing any.

00:49:15.095 --> 00:49:16.770 align:middle line:90%
You're not giving up anything.

00:49:16.770 --> 00:49:20.090 align:middle line:84%
So in terms of how many
layers you should have,

00:49:20.090 --> 00:49:21.790 align:middle line:90%
my philosophy is start simple.

00:49:21.790 --> 00:49:23.750 align:middle line:84%
And if it works,
stop working on it.

00:49:23.750 --> 00:49:27.290 align:middle line:90%
If it doesn't, add more layers.

00:49:27.290 --> 00:49:27.935 align:middle line:90%
Yeah?

00:49:27.935 --> 00:49:29.810 align:middle line:84%
AUDIENCE: Yeah, just to
build on that, is it,

00:49:29.810 --> 00:49:32.150 align:middle line:84%
the architecture design,
so the number of filters,

00:49:32.150 --> 00:49:33.950 align:middle line:84%
the kernel size, the
number of layers,

00:49:33.950 --> 00:49:35.570 align:middle line:84%
convolution,
pooling, is that just

00:49:35.570 --> 00:49:37.430 align:middle line:84%
all based on trial and error
to see what works best?

00:49:37.430 --> 00:49:37.490 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:49:37.490 --> 00:49:39.282 align:middle line:84%
So typically it's based
on trial and error,

00:49:39.282 --> 00:49:40.758 align:middle line:90%
to answer your question.

00:49:40.758 --> 00:49:43.050 align:middle line:84%
But as you will see in the
transfer learning discussion

00:49:43.050 --> 00:49:45.850 align:middle line:84%
we're going to have
soon, you can actually,

00:49:45.850 --> 00:49:47.990 align:middle line:84%
instead of doing
anything from scratch,

00:49:47.990 --> 00:49:50.490 align:middle line:84%
it's much better to just
download a pre-trained model

00:49:50.490 --> 00:49:52.630 align:middle line:84%
and just adapt it for
your particular problem.

00:49:52.630 --> 00:49:55.270 align:middle line:84%
That is actually the norm by
which people do these things.

00:49:55.270 --> 00:49:56.770 align:middle line:84%
The reason I'm doing
it from scratch

00:49:56.770 --> 00:50:00.560 align:middle line:84%
is because you should
know how it was done.

00:50:00.560 --> 00:50:02.420 align:middle line:84%
It should not be a
black box to you.

00:50:02.420 --> 00:50:03.920 align:middle line:90%
That's my goal.

00:50:03.920 --> 00:50:05.160 align:middle line:90%
Yeah?

00:50:05.160 --> 00:50:07.060 align:middle line:84%
AUDIENCE: Just from a
notation perspective,

00:50:07.060 --> 00:50:09.718 align:middle line:84%
I noticed that you made
all of these layers x.

00:50:09.718 --> 00:50:12.260 align:middle line:84%
Is that a habit we should get
into, making them all the same,

00:50:12.260 --> 00:50:13.100 align:middle line:90%
or is that just--

00:50:13.100 --> 00:50:16.320 align:middle line:84%
RAMA RAMAKRISHNAN: Actually,
I'm not naming the layers as x.

00:50:16.320 --> 00:50:19.720 align:middle line:84%
What's going on here
is I'm feeding it x.

00:50:19.720 --> 00:50:22.680 align:middle line:84%
And whatever is coming out
of it, I'm just calling it x.

00:50:22.680 --> 00:50:23.180 align:middle line:90%
That's all.

00:50:23.180 --> 00:50:25.190 align:middle line:84%
It's just a notational
convenience for me.

00:50:25.190 --> 00:50:26.940 align:middle line:84%
I'm just calling the
input and the output,

00:50:26.940 --> 00:50:29.240 align:middle line:84%
and Keras, under the hood,
will track everything and make

00:50:29.240 --> 00:50:30.500 align:middle line:90%
sure the right thing happens.

00:50:30.500 --> 00:50:32.802 align:middle line:84%
Otherwise, I'd have
to be, x1, x2, x3, x4.

00:50:32.802 --> 00:50:34.760 align:middle line:84%
And then if I want to
add a new layer somewhere

00:50:34.760 --> 00:50:37.980 align:middle line:84%
in the middle between x3 and
x4, I'll have to call that x4.

00:50:37.980 --> 00:50:40.060 align:middle line:84%
And then I'll change
everything to 5, 6, 7.

00:50:40.060 --> 00:50:41.200 align:middle line:90%
Complete pain in the neck.

00:50:41.200 --> 00:50:43.600 align:middle line:90%
That's why I do this.

00:50:43.600 --> 00:50:50.920 align:middle line:84%
So model.summary, it has
got 302,000 parameters.

00:50:50.920 --> 00:50:53.240 align:middle line:90%
I'll just plot it.

00:50:53.240 --> 00:50:54.030 align:middle line:90%
Great.

00:50:54.030 --> 00:50:58.030 align:middle line:84%
And I encourage you to
hand calculate it later on

00:50:58.030 --> 00:51:00.790 align:middle line:90%
and make sure the numbers tally.

00:51:00.790 --> 00:51:02.430 align:middle line:90%
For now, let's just go.

00:51:02.430 --> 00:51:06.370 align:middle line:84%
So as before, we'll just
use the same compilation.

00:51:06.370 --> 00:51:10.890 align:middle line:84%
We'll use adam, and then we'll
train it for just 10 epochs.

00:51:10.890 --> 00:51:13.970 align:middle line:84%
We'll use a validation split,
again, as usual of 20%.

00:51:13.970 --> 00:51:15.790 align:middle line:90%
So let's just run it.

00:51:15.790 --> 00:51:17.010 align:middle line:90%
It's actually going to run.

00:51:17.010 --> 00:51:19.042 align:middle line:84%
And as you will see,
convolution networks,

00:51:19.042 --> 00:51:20.750 align:middle line:84%
there's a lot more
going on so it's going

00:51:20.750 --> 00:51:22.610 align:middle line:90%
to be a bit slower to run.

00:51:22.610 --> 00:51:25.350 align:middle line:90%
Hopefully not too much slower.

00:51:25.350 --> 00:51:26.890 align:middle line:84%
While it's doing,
other questions?

00:51:26.890 --> 00:51:30.685 align:middle line:90%


00:51:30.685 --> 00:51:33.310 align:middle line:84%
AUDIENCE: So if we have a task
other than image classification,

00:51:33.310 --> 00:51:36.870 align:middle line:84%
do we still flatten the model
like for semantic segmentation?

00:51:36.870 --> 00:51:37.870 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:51:37.870 --> 00:51:39.790 align:middle line:84%
So this is for image
classification.

00:51:39.790 --> 00:51:43.312 align:middle line:84%
For other kinds of
applications, typically you

00:51:43.312 --> 00:51:45.270 align:middle line:84%
run it through a bunch
of convolutional layers,

00:51:45.270 --> 00:51:46.670 align:middle line:90%
and so on, and so forth.

00:51:46.670 --> 00:51:48.630 align:middle line:84%
But the output side
of the equation

00:51:48.630 --> 00:51:50.710 align:middle line:90%
gets much more complicated.

00:51:50.710 --> 00:51:54.790 align:middle line:84%
Because instead of classifying
just the whole picture

00:51:54.790 --> 00:51:58.790 align:middle line:84%
into dog or cat, if you
have to take every pixel

00:51:58.790 --> 00:52:01.630 align:middle line:84%
and classify it,
then, well, you better

00:52:01.630 --> 00:52:04.950 align:middle line:84%
have an output shape that is
the same dimensions as the input

00:52:04.950 --> 00:52:06.350 align:middle line:90%
shape.

00:52:06.350 --> 00:52:08.210 align:middle line:84%
So for that we use a
different architecture.

00:52:08.210 --> 00:52:11.030 align:middle line:84%
It's called U-Net, and so
on, which unfortunately I

00:52:11.030 --> 00:52:12.270 align:middle line:90%
won't be able to get into.

00:52:12.270 --> 00:52:15.790 align:middle line:84%
But I am planning to post
another video walkthrough where

00:52:15.790 --> 00:52:18.510 align:middle line:84%
I show you how to
use the Hugging Face

00:52:18.510 --> 00:52:23.170 align:middle line:84%
Hub to very quickly build models
for the other applications,

00:52:23.170 --> 00:52:24.530 align:middle line:90%
like segmentation, and so on.

00:52:24.530 --> 00:52:26.030 align:middle line:84%
I'm hoping to post
that tomorrow.

00:52:26.030 --> 00:52:27.322 align:middle line:90%
It's an optional viewing thing.

00:52:27.322 --> 00:52:30.110 align:middle line:90%
That might help with that.

00:52:30.110 --> 00:52:31.870 align:middle line:90%
So is it done?

00:52:31.870 --> 00:52:32.370 align:middle line:90%
OK, good.

00:52:32.370 --> 00:52:33.670 align:middle line:90%
It's done.

00:52:33.670 --> 00:52:36.522 align:middle line:84%
All right, let's
plot the thing here.

00:52:36.522 --> 00:52:38.230 align:middle line:84%
All right, so it seems
like this training

00:52:38.230 --> 00:52:39.950 align:middle line:90%
is going down nice and nicely.

00:52:39.950 --> 00:52:43.910 align:middle line:84%
Validation is flattening out
somewhere here around the 8th

00:52:43.910 --> 00:52:44.670 align:middle line:90%
epoch.

00:52:44.670 --> 00:52:47.390 align:middle line:90%
Let's look at the accuracy.

00:52:47.390 --> 00:52:48.370 align:middle line:90%
Same situation here.

00:52:48.370 --> 00:52:50.212 align:middle line:90%
The accuracy is in the 90s.

00:52:50.212 --> 00:52:51.920 align:middle line:84%
Of course, the final
question, of course,

00:52:51.920 --> 00:52:55.900 align:middle line:84%
is how well it
does on the thing.

00:52:55.900 --> 00:52:58.300 align:middle line:90%
Whoa, 90.5%.

00:52:58.300 --> 00:52:59.162 align:middle line:90%
It's pretty good.

00:52:59.162 --> 00:53:00.620 align:middle line:84%
By the way, if
you're not impressed

00:53:00.620 --> 00:53:05.780 align:middle line:84%
that we went from 88 to
90, these applications

00:53:05.780 --> 00:53:08.480 align:middle line:84%
are the proverbial
diminishing returns problems.

00:53:08.480 --> 00:53:10.940 align:middle line:84%
So what you should
always think of is,

00:53:10.940 --> 00:53:13.960 align:middle line:84%
look at the amount of error
that's left, and ask yourself,

00:53:13.960 --> 00:53:17.500 align:middle line:84%
how much of that error
am I able to reduce?

00:53:17.500 --> 00:53:20.380 align:middle line:84%
We had 12%, roughly,
of error left when we

00:53:20.380 --> 00:53:22.140 align:middle line:90%
did the simple Colab yesterday.

00:53:22.140 --> 00:53:24.900 align:middle line:84%
From that 12%, we have
knocked off 2% of the 12%

00:53:24.900 --> 00:53:27.260 align:middle line:84%
to get to over 90%,
which is amazing.

00:53:27.260 --> 00:53:34.420 align:middle line:84%
And in fact, I think the state
of the art on this is 97%.

00:53:34.420 --> 00:53:38.045 align:middle line:84%
So I invite you to
take this thing,

00:53:38.045 --> 00:53:40.420 align:middle line:84%
and try different filters,
and so on, and so forth to see

00:53:40.420 --> 00:53:42.820 align:middle line:90%
if you can get to the mid 90s.

00:53:42.820 --> 00:53:44.700 align:middle line:90%
It's not easy, but try it.

00:53:44.700 --> 00:53:46.045 align:middle line:90%
Yeah?

00:53:46.045 --> 00:53:47.420 align:middle line:84%
AUDIENCE: Is the
number of epochs

00:53:47.420 --> 00:53:50.330 align:middle line:84%
has to do with it's related
to the number of batches?

00:53:50.330 --> 00:53:52.253 align:middle line:84%
Because you put 64
batches and then epoch.

00:53:52.253 --> 00:53:53.170 align:middle line:90%
RAMA RAMAKRISHNAN: No.

00:53:53.170 --> 00:53:58.370 align:middle line:84%
The epochs is just the number of
passes through the whole data.

00:53:58.370 --> 00:54:01.330 align:middle line:84%
But within each pass,
within each epoch,

00:54:01.330 --> 00:54:03.690 align:middle line:84%
the batch size tells
you how many batches

00:54:03.690 --> 00:54:05.050 align:middle line:90%
you're going to process.

00:54:05.050 --> 00:54:06.970 align:middle line:84%
So it is basically
the number of examples

00:54:06.970 --> 00:54:10.010 align:middle line:84%
you have in your training
data divided by the batch size

00:54:10.010 --> 00:54:11.770 align:middle line:90%
that you have chosen.

00:54:11.770 --> 00:54:14.250 align:middle line:84%
That number rounded
up is the number

00:54:14.250 --> 00:54:16.865 align:middle line:90%
of batches within each epoch.

00:54:16.865 --> 00:54:18.490 align:middle line:84%
And here I'm just
choosing 10 because--

00:54:18.490 --> 00:54:19.930 align:middle line:90%
SIRI: I found this on the web.

00:54:19.930 --> 00:54:23.090 align:middle line:84%
RAMA RAMAKRISHNAN: Siri
found something on the web.

00:54:23.090 --> 00:54:24.890 align:middle line:84%
I chose 10 because
it's going to be fast

00:54:24.890 --> 00:54:26.070 align:middle line:90%
for me to do it in class.

00:54:26.070 --> 00:54:27.528 align:middle line:84%
And 10 is actually
more than enough

00:54:27.528 --> 00:54:31.050 align:middle line:84%
because you can see it's
already beginning to overfit.

00:54:31.050 --> 00:54:33.730 align:middle line:90%
Yeah?

00:54:33.730 --> 00:54:36.250 align:middle line:84%
AUDIENCE: This is more of a
conceptual question, but is it

00:54:36.250 --> 00:54:38.330 align:middle line:84%
always the case that
a neural network will

00:54:38.330 --> 00:54:42.510 align:middle line:84%
have better accuracy than its
machine learning algorithm?

00:54:42.510 --> 00:54:45.563 align:middle line:84%
And I'm asking more on the case
of the heart disease problem.

00:54:45.563 --> 00:54:46.730 align:middle line:90%
RAMA RAMAKRISHNAN: Oh, yeah.

00:54:46.730 --> 00:54:47.590 align:middle line:90%
Great question.

00:54:47.590 --> 00:54:50.415 align:middle line:84%
So neural networks are really
good for unstructured data,

00:54:50.415 --> 00:54:51.790 align:middle line:84%
like the ones we
are having here.

00:54:51.790 --> 00:54:54.370 align:middle line:84%
But if you have structured data
like the heart disease problem,

00:54:54.370 --> 00:54:56.037 align:middle line:84%
sometimes it actually
works really well.

00:54:56.037 --> 00:54:59.810 align:middle line:84%
Sometimes things like
gradient boosting, XGBoost,

00:54:59.810 --> 00:55:00.970 align:middle line:90%
work really well.

00:55:00.970 --> 00:55:03.550 align:middle line:84%
So if I am actually working
on a structured data problem,

00:55:03.550 --> 00:55:04.570 align:middle line:90%
I'll try both.

00:55:04.570 --> 00:55:06.130 align:middle line:84%
I'm not going to
axiomatically assume

00:55:06.130 --> 00:55:07.922 align:middle line:84%
that the DNN is going
to be the best thing.

00:55:07.922 --> 00:55:10.590 align:middle line:84%
But if you have structured data,
it's the best game in town.

00:55:10.590 --> 00:55:13.865 align:middle line:90%


00:55:13.865 --> 00:55:15.490 align:middle line:84%
By the way, I have
a whole section here

00:55:15.490 --> 00:55:17.907 align:middle line:84%
on once you build a model, how
do you actually improve it?

00:55:17.907 --> 00:55:18.870 align:middle line:90%
Check it out.

00:55:18.870 --> 00:55:20.450 align:middle line:90%
It's an optional thing.

00:55:20.450 --> 00:55:23.070 align:middle line:84%
All right, I'm going
to stop this here.

00:55:23.070 --> 00:55:25.570 align:middle line:84%
So the next thing
I want to do is,

00:55:25.570 --> 00:55:28.650 align:middle line:84%
so we went from 88% to 90-plus
percent using convolutional

00:55:28.650 --> 00:55:29.970 align:middle line:90%
networks.

00:55:29.970 --> 00:55:31.590 align:middle line:84%
Now let's work
with color images.

00:55:31.590 --> 00:55:33.050 align:middle line:90%
Let's kick it up a notch.

00:55:33.050 --> 00:55:40.010 align:middle line:84%
So I actually web scraped all
these pictures for you folks

00:55:40.010 --> 00:55:41.170 align:middle line:90%
for your enjoyment.

00:55:41.170 --> 00:55:45.490 align:middle line:84%
I web scraped about 100 color
images of handbags and shoes,

00:55:45.490 --> 00:55:47.350 align:middle line:90%
roughly 100 handbags and shoes.

00:55:47.350 --> 00:55:51.200 align:middle line:84%
So the question is, with
these essentially 200 images,

00:55:51.200 --> 00:55:53.040 align:middle line:84%
can we build a really
good neural network

00:55:53.040 --> 00:55:54.780 align:middle line:90%
to classify handbags and shoes?

00:55:54.780 --> 00:55:57.860 align:middle line:84%
It seems kind of absurd
because 200 examples,

00:55:57.860 --> 00:55:59.020 align:middle line:90%
it's not that much.

00:55:59.020 --> 00:56:00.200 align:middle line:90%
It doesn't feel like a lot.

00:56:00.200 --> 00:56:05.520 align:middle line:84%
The MNIST data, Fashion
MNIST, has 60,000 images.

00:56:05.520 --> 00:56:08.220 align:middle line:84%
Even with that, we are
overfitting in 5, 6, 7,

00:56:08.220 --> 00:56:09.200 align:middle line:90%
8 epochs.

00:56:09.200 --> 00:56:11.420 align:middle line:84%
With 200 images, maybe,
is there any hope?

00:56:11.420 --> 00:56:12.560 align:middle line:90%
Obviously, there is hope.

00:56:12.560 --> 00:56:14.260 align:middle line:84%
Otherwise it wouldn't
be in the lecture.

00:56:14.260 --> 00:56:15.760 align:middle line:84%
So we're going to
take this data set

00:56:15.760 --> 00:56:17.220 align:middle line:84%
and let's see what
we can do with it.

00:56:17.220 --> 00:56:19.400 align:middle line:84%
So we will first actually
build a convolutional network

00:56:19.400 --> 00:56:20.858 align:middle line:84%
from scratch to
solve this problem.

00:56:20.858 --> 00:56:24.370 align:middle line:90%


00:56:24.370 --> 00:56:26.120 align:middle line:84%
I'm actually going to
run through the code

00:56:26.120 --> 00:56:28.540 align:middle line:84%
because at the end of it,
we'll have a live demo.

00:56:28.540 --> 00:56:30.800 align:middle line:84%
So I would like one
volunteer to give me

00:56:30.800 --> 00:56:32.800 align:middle line:84%
a handbag and one
volunteer to give me

00:56:32.800 --> 00:56:35.750 align:middle line:84%
their footwear
[INAUDIBLE] in class.

00:56:35.750 --> 00:56:39.720 align:middle line:90%


00:56:39.720 --> 00:56:41.640 align:middle line:84%
Unlike the previous
data set, this one

00:56:41.640 --> 00:56:44.980 align:middle line:90%
actually, I just web scraped it.

00:56:44.980 --> 00:56:47.440 align:middle line:84%
I've stuck it in
this Dropbox folder.

00:56:47.440 --> 00:56:49.140 align:middle line:84%
Let's just download
it and unzip it.

00:56:49.140 --> 00:56:52.520 align:middle line:84%
And once we do that, we
have to now organize it

00:56:52.520 --> 00:56:53.900 align:middle line:90%
with these 200 images.

00:56:53.900 --> 00:57:00.520 align:middle line:84%
So I have to do some
boring-ish Python stuff here.

00:57:00.520 --> 00:57:03.540 align:middle line:84%
So here what we're doing is
that we have 100 handbags,

00:57:03.540 --> 00:57:04.700 align:middle line:90%
roughly 100 shoes.

00:57:04.700 --> 00:57:06.800 align:middle line:84%
And what this code is
doing is it's actually

00:57:06.800 --> 00:57:09.040 align:middle line:84%
creating a directory
of saying, it's

00:57:09.040 --> 00:57:11.580 align:middle line:84%
splitting stuff into train,
and validation, and test.

00:57:11.580 --> 00:57:13.200 align:middle line:84%
And then for each
of the splits, it's

00:57:13.200 --> 00:57:15.020 align:middle line:84%
doing the handbags
and the shoes folder.

00:57:15.020 --> 00:57:19.320 align:middle line:84%
So once we do that, basically
this directory structure

00:57:19.320 --> 00:57:21.320 align:middle line:90%
is created.

00:57:21.320 --> 00:57:24.200 align:middle line:84%
Training, validation folder,
test folder, handbags,

00:57:24.200 --> 00:57:24.860 align:middle line:90%
and shoes.

00:57:24.860 --> 00:57:28.040 align:middle line:84%
In fact, actually, I think
you can see it here, see here,

00:57:28.040 --> 00:57:29.360 align:middle line:90%
handbags and shoes.

00:57:29.360 --> 00:57:32.180 align:middle line:84%
And within that, there is
train, test, validation.

00:57:32.180 --> 00:57:34.440 align:middle line:84%
And within each of these
there's handbags and shoes.

00:57:34.440 --> 00:57:37.040 align:middle line:84%
So the idea is that when you're
working with images, what

00:57:37.040 --> 00:57:38.920 align:middle line:84%
you can do is you can
just create folders

00:57:38.920 --> 00:57:40.400 align:middle line:90%
for each kind of image.

00:57:40.400 --> 00:57:43.820 align:middle line:84%
Let's say dogs, cats, two
folders with cat images and dog

00:57:43.820 --> 00:57:44.320 align:middle line:90%
images.

00:57:44.320 --> 00:57:46.410 align:middle line:90%
And then just point Keras at it.

00:57:46.410 --> 00:57:47.910 align:middle line:84%
It'll automatically
figure out those

00:57:47.910 --> 00:57:50.370 align:middle line:84%
are the labels that
it needs to use.

00:57:50.370 --> 00:57:52.870 align:middle line:84%
So it's very convenient when
you're working with images.

00:57:52.870 --> 00:57:56.210 align:middle line:84%
And the book explains this
thing in great detail.

00:57:56.210 --> 00:57:57.967 align:middle line:84%
So when working with
these color images

00:57:57.967 --> 00:57:59.050 align:middle line:90%
we'll follow this process.

00:57:59.050 --> 00:58:00.370 align:middle line:90%
We'll read in the JPEGs.

00:58:00.370 --> 00:58:01.652 align:middle line:90%
We'll convert them to tensors.

00:58:01.652 --> 00:58:03.110 align:middle line:84%
And then since I'm
web scraping it,

00:58:03.110 --> 00:58:04.943 align:middle line:84%
they all come in different
shapes and sizes.

00:58:04.943 --> 00:58:07.470 align:middle line:84%
So I need to bring it
all to the same size.

00:58:07.470 --> 00:58:10.370 align:middle line:84%
I resize it, and then I'm going
to batch it into whatever.

00:58:10.370 --> 00:58:13.390 align:middle line:84%
I'm going to batch it using
a batch size of 32 here.

00:58:13.390 --> 00:58:17.670 align:middle line:84%
And this utility from Keras
will do all that for you

00:58:17.670 --> 00:58:19.572 align:middle line:90%
very quickly.

00:58:19.572 --> 00:58:21.030 align:middle line:84%
So basically what
it says is that I

00:58:21.030 --> 00:58:25.610 align:middle line:84%
found 98 images in the training
data belonging to two classes,

00:58:25.610 --> 00:58:28.210 align:middle line:84%
49 in the validation,
and 38 in the test.

00:58:28.210 --> 00:58:30.170 align:middle line:84%
So less than 100 examples
in the training set.

00:58:30.170 --> 00:58:32.263 align:middle line:90%
That's what we have here.

00:58:32.263 --> 00:58:32.930 align:middle line:90%
What's the time?

00:58:32.930 --> 00:58:34.950 align:middle line:90%
9:38.

00:58:34.950 --> 00:58:37.610 align:middle line:84%
Now let's just check the
dimensions to make sure.

00:58:37.610 --> 00:58:38.110 align:middle line:90%
Good.

00:58:38.110 --> 00:58:39.650 align:middle line:90%
So 224, 224 by 3.

00:58:39.650 --> 00:58:41.853 align:middle line:84%
And the reason why
did I pick 224 to 24?

00:58:41.853 --> 00:58:43.270 align:middle line:84%
As you will see
later, we're going

00:58:43.270 --> 00:58:45.100 align:middle line:90%
to use something called ResNet.

00:58:45.100 --> 00:58:48.320 align:middle line:84%
And the ResNet expects
it to be 224 by 224 by 3.

00:58:48.320 --> 00:58:50.740 align:middle line:84%
That's why I resize
it to 224, 224.

00:58:50.740 --> 00:58:53.820 align:middle line:84%
Let's look at a few examples
of my wonderful web scraping

00:58:53.820 --> 00:58:56.160 align:middle line:90%
in action.

00:58:56.160 --> 00:58:56.660 align:middle line:90%
Oh.

00:58:56.660 --> 00:59:01.140 align:middle line:90%


00:59:01.140 --> 00:59:04.060 align:middle line:90%
It's pretty wild, right?

00:59:04.060 --> 00:59:07.020 align:middle line:84%
Now let's do a simple
convolutional network.

00:59:07.020 --> 00:59:10.140 align:middle line:84%
And before, we would take all
the x values in Fashion MNIST

00:59:10.140 --> 00:59:13.407 align:middle line:84%
and divide them manually by
255 to normalize it to 0, 1.

00:59:13.407 --> 00:59:14.240 align:middle line:90%
Well, you know what?

00:59:14.240 --> 00:59:16.573 align:middle line:84%
We are actually graduating
to the higher levels of Keras

00:59:16.573 --> 00:59:17.780 align:middle line:90%
now so let's not do that.

00:59:17.780 --> 00:59:18.960 align:middle line:90%
Manual stuff is bad.

00:59:18.960 --> 00:59:21.193 align:middle line:84%
So we'll do it within Keras
by using something called

00:59:21.193 --> 00:59:22.860 align:middle line:84%
a Rescaling layer,
where we just tell it

00:59:22.860 --> 00:59:25.540 align:middle line:84%
how much to rescale, and
boom, it will do it for you.

00:59:25.540 --> 00:59:28.580 align:middle line:84%
The first convolutional block,
just like the Fashion MNIST, 32.

00:59:28.580 --> 00:59:30.740 align:middle line:90%
Second block, again, 32.

00:59:30.740 --> 00:59:32.240 align:middle line:90%
Maxpool, flatten.

00:59:32.240 --> 00:59:34.520 align:middle line:84%
And then here we only have
handbags versus shoes.

00:59:34.520 --> 00:59:35.940 align:middle line:90%
Just a sigmoid is enough.

00:59:35.940 --> 00:59:37.760 align:middle line:84%
It's just a binary
classification problem.

00:59:37.760 --> 00:59:40.480 align:middle line:84%
So I'm just using one
output layer with a sigmoid

00:59:40.480 --> 00:59:41.660 align:middle line:90%
and that's our model.

00:59:41.660 --> 00:59:44.520 align:middle line:90%
So let's do the model.

00:59:44.520 --> 00:59:45.520 align:middle line:90%
model.summary.

00:59:45.520 --> 00:59:49.620 align:middle line:90%


00:59:49.620 --> 00:59:52.980 align:middle line:84%
101,000 parameters
in this little model.

00:59:52.980 --> 00:59:55.260 align:middle line:90%
OK, let's compile it and run it.

00:59:55.260 --> 00:59:58.580 align:middle line:84%
And note here, because it's a
binary classification problem

00:59:58.580 --> 01:00:01.660 align:middle line:90%
I'm using binary cross entropy.

01:00:01.660 --> 01:00:05.060 align:middle line:84%
And this is the same Adam
and accuracy compile.

01:00:05.060 --> 01:00:06.800 align:middle line:90%
And then, boom, let's run it.

01:00:06.800 --> 01:00:08.540 align:middle line:90%
We'll do it for 20 epochs.

01:00:08.540 --> 01:00:12.482 align:middle line:84%
Hopefully it won't
take too long.

01:00:12.482 --> 01:00:13.940 align:middle line:84%
While it's doing
this business, I'm

01:00:13.940 --> 01:00:17.780 align:middle line:84%
going to shift to
the PowerPoint.

01:00:17.780 --> 01:00:19.800 align:middle line:84%
So we'll go back to
see how well it did.

01:00:19.800 --> 01:00:23.040 align:middle line:84%
But the question is, whatever it
did, we built it from scratch.

01:00:23.040 --> 01:00:25.460 align:middle line:84%
So the question is, can
we do better than that?

01:00:25.460 --> 01:00:27.640 align:middle line:84%
Because we only have 100
examples of each class,

01:00:27.640 --> 01:00:30.140 align:middle line:84%
and which brings us to something
very cool and very powerful

01:00:30.140 --> 01:00:32.030 align:middle line:90%
called transfer learning.

01:00:32.030 --> 01:00:33.780 align:middle line:84%
So the key thing is
there are two research

01:00:33.780 --> 01:00:35.905 align:middle line:84%
trends that are going on
that we take advantage of.

01:00:35.905 --> 01:00:37.340 align:middle line:84%
The first one is
that researchers

01:00:37.340 --> 01:00:41.810 align:middle line:84%
have designed architectures
which exploit the kind of input

01:00:41.810 --> 01:00:42.310 align:middle line:90%
you have.

01:00:42.310 --> 01:00:43.810 align:middle line:84%
So Olivia asked the
question, if you

01:00:43.810 --> 01:00:45.485 align:middle line:84%
have a particular
kind of input images,

01:00:45.485 --> 01:00:47.610 align:middle line:84%
do you actually change the
input or do you actually

01:00:47.610 --> 01:00:48.670 align:middle line:90%
change the network?

01:00:48.670 --> 01:00:50.830 align:middle line:84%
As it turns out here, for
example, if it's images,

01:00:50.830 --> 01:00:52.290 align:middle line:84%
we know that we should
use convolutional layers

01:00:52.290 --> 01:00:54.890 align:middle line:84%
because convolutional layers
were designed to exploit

01:00:54.890 --> 01:00:57.770 align:middle line:90%
the image nest of the input.

01:00:57.770 --> 01:01:01.290 align:middle line:84%
Similarly, if you have sequences
of information, obviously

01:01:01.290 --> 01:01:03.588 align:middle line:84%
natural language, audio,
video, gene sequence,

01:01:03.588 --> 01:01:05.130 align:middle line:84%
and so on, and so
forth, these things

01:01:05.130 --> 01:01:07.942 align:middle line:84%
called transformers were
invented to exploit them.

01:01:07.942 --> 01:01:10.650 align:middle line:84%
And we'll spend a lot of time on
transformers starting next week.

01:01:10.650 --> 01:01:12.110 align:middle line:90%
So that's the first trend.

01:01:12.110 --> 01:01:14.050 align:middle line:84%
The second trend
is that researchers

01:01:14.050 --> 01:01:18.730 align:middle line:84%
have used these innovations to
actually create and train models

01:01:18.730 --> 01:01:20.450 align:middle line:90%
on vast data sets.

01:01:20.450 --> 01:01:23.410 align:middle line:84%
And thankfully, they have
made them publicly available

01:01:23.410 --> 01:01:24.610 align:middle line:90%
for us to use.

01:01:24.610 --> 01:01:27.210 align:middle line:84%
So transfer learning
is the idea that if you

01:01:27.210 --> 01:01:28.850 align:middle line:84%
have a particular
problem, let's just

01:01:28.850 --> 01:01:30.570 align:middle line:84%
take a pre-trained
network somebody may

01:01:30.570 --> 01:01:32.310 align:middle line:90%
have already created.

01:01:32.310 --> 01:01:34.650 align:middle line:84%
And then let's just
customize it to our problem

01:01:34.650 --> 01:01:37.090 align:middle line:84%
rather than actually build
anything from scratch.

01:01:37.090 --> 01:01:38.610 align:middle line:90%
That's the basic idea.

01:01:38.610 --> 01:01:41.245 align:middle line:90%
So here, we have this.

01:01:41.245 --> 01:01:43.120 align:middle line:84%
Basically we have to
build a classifier which

01:01:43.120 --> 01:01:45.000 align:middle line:84%
takes in an arbitrary
image and figures out

01:01:45.000 --> 01:01:46.220 align:middle line:90%
if it's a handbag or a shoe.

01:01:46.220 --> 01:01:48.640 align:middle line:90%
That's our goal.

01:01:48.640 --> 01:01:50.740 align:middle line:84%
Now handbags and shoes
are everyday objects.

01:01:50.740 --> 01:01:53.520 align:middle line:84%
And so what you can do is
you can look around and see

01:01:53.520 --> 01:01:55.520 align:middle line:84%
if there are any networks
that have been trained

01:01:55.520 --> 01:01:58.560 align:middle line:84%
by other people which
actually have been trained

01:01:58.560 --> 01:02:02.620 align:middle line:84%
on everyday images, as
opposed to MRI, or x-rays,

01:02:02.620 --> 01:02:04.120 align:middle line:84%
specialized images,
everyday images.

01:02:04.120 --> 01:02:06.162 align:middle line:84%
Of course, the first thing
you should probably do

01:02:06.162 --> 01:02:09.060 align:middle line:84%
is to see if anybody has built
the specific thing you want,

01:02:09.060 --> 01:02:10.800 align:middle line:84%
handbag, shoes
classifier on GitHub.

01:02:10.800 --> 01:02:13.920 align:middle line:84%
Assuming it's not, then
you do transfer learning.

01:02:13.920 --> 01:02:17.840 align:middle line:84%
So now it turns out that there
is this thing called ImageNet

01:02:17.840 --> 01:02:21.400 align:middle line:84%
which is a database of millions
of images of everyday objects

01:02:21.400 --> 01:02:25.420 align:middle line:84%
in 1,000 different categories,
furniture, animals, automobiles.

01:02:25.420 --> 01:02:26.340 align:middle line:90%
You get the idea.

01:02:26.340 --> 01:02:29.280 align:middle line:84%
And so we can look
for networks that

01:02:29.280 --> 01:02:31.858 align:middle line:90%
have been trained on ImageNet.

01:02:31.858 --> 01:02:33.400 align:middle line:84%
Let me just go back
to the Colab just

01:02:33.400 --> 01:02:34.775 align:middle line:84%
to make sure it
doesn't time out.

01:02:34.775 --> 01:02:38.180 align:middle line:90%


01:02:38.180 --> 01:02:40.680 align:middle line:90%
So it has finished doing it.

01:02:40.680 --> 01:02:41.890 align:middle line:90%
Let's just plot these things.

01:02:41.890 --> 01:02:48.680 align:middle line:90%


01:02:48.680 --> 01:02:52.920 align:middle line:84%
So there is some overfitting
that happens around here

01:02:52.920 --> 01:02:55.000 align:middle line:84%
on the training
and the 10th epoch.

01:02:55.000 --> 01:02:58.640 align:middle line:90%
Let's look at the history.

01:02:58.640 --> 01:02:59.460 align:middle line:90%
OK, interesting.

01:02:59.460 --> 01:03:03.440 align:middle line:84%
So the training accuracy is
actually getting almost to 100%.

01:03:03.440 --> 01:03:06.100 align:middle line:84%
But we're not impressed
by training accuracy.

01:03:06.100 --> 01:03:08.660 align:middle line:84%
We care about validation
and test accuracy.

01:03:08.660 --> 01:03:11.840 align:middle line:84%
And that seems to be
hovering around in the 80s.

01:03:11.840 --> 01:03:15.240 align:middle line:84%
So let's just evaluate it
anyway to see what happens.

01:03:15.240 --> 01:03:21.040 align:middle line:84%
So it gets to 87%
accuracy on this data set.

01:03:21.040 --> 01:03:24.040 align:middle line:84%
It's actually pretty good, given
that we only have 100 examples.

01:03:24.040 --> 01:03:26.980 align:middle line:84%
So 87% accuracy, but we
pre-trained the whole thing.

01:03:26.980 --> 01:03:28.770 align:middle line:84%
Sorry, we did
everything from scratch.

01:03:28.770 --> 01:03:31.800 align:middle line:90%


01:03:31.800 --> 01:03:34.420 align:middle line:84%
There's this whole section
about data augmentation,

01:03:34.420 --> 01:03:35.560 align:middle line:90%
which, you know what?

01:03:35.560 --> 01:03:36.700 align:middle line:90%
Do we have time?

01:03:36.700 --> 01:03:37.200 align:middle line:90%
Yeah.

01:03:37.200 --> 01:03:40.110 align:middle line:90%


01:03:40.110 --> 01:03:44.350 align:middle line:84%
So the idea of augmentation is
that when you have an image,

01:03:44.350 --> 01:03:47.350 align:middle line:84%
let's say you take this image
and you just rotate it slightly

01:03:47.350 --> 01:03:49.230 align:middle line:90%
by 10 degrees.

01:03:49.230 --> 01:03:52.150 align:middle line:84%
If it's a handbag before you
rotated it, it sure as hell

01:03:52.150 --> 01:03:55.110 align:middle line:84%
is a handbag after
you rotated it.

01:03:55.110 --> 01:03:55.910 align:middle line:90%
It doesn't change.

01:03:55.910 --> 01:03:58.577 align:middle line:84%
The meaning of the image doesn't
change just because you rotated

01:03:58.577 --> 01:04:00.577 align:middle line:84%
it slightly, or maybe
you zoom in slightly,

01:04:00.577 --> 01:04:02.410 align:middle line:84%
you zoom out slightly,
you crop it slightly.

01:04:02.410 --> 01:04:03.670 align:middle line:90%
Nothing happens.

01:04:03.670 --> 01:04:05.150 align:middle line:84%
So what you can
do is you can take

01:04:05.150 --> 01:04:10.090 align:middle line:84%
any image you have and you just
perturb it slightly, like that.

01:04:10.090 --> 01:04:14.156 align:middle line:84%
And then add it as a new
example to your training data.

01:04:14.156 --> 01:04:16.990 align:middle line:84%
It's an unbelievable
free lunch, frankly.

01:04:16.990 --> 01:04:19.630 align:middle line:84%
And the same thing actually,
the same kinds of techniques

01:04:19.630 --> 01:04:22.590 align:middle line:84%
actually work for text also,
which we'll cover later on.

01:04:22.590 --> 01:04:26.217 align:middle line:84%
This broad area is
called data augmentation.

01:04:26.217 --> 01:04:28.550 align:middle line:84%
And it's a great way, when
you don't have a lot of data,

01:04:28.550 --> 01:04:31.450 align:middle line:84%
to artificially bolster the
amount of data you have.

01:04:31.450 --> 01:04:34.030 align:middle line:90%


01:04:34.030 --> 01:04:35.550 align:middle line:84%
Of course, Keras
makes it very easy

01:04:35.550 --> 01:04:37.050 align:middle line:90%
for you to do all these things.

01:04:37.050 --> 01:04:40.670 align:middle line:84%
It has already predefined a
whole bunch of data augmentation

01:04:40.670 --> 01:04:41.670 align:middle line:90%
layers for you.

01:04:41.670 --> 01:04:44.910 align:middle line:84%
So here's a little example
where I basically take a picture

01:04:44.910 --> 01:04:46.610 align:middle line:90%
and then I randomly flip it.

01:04:46.610 --> 01:04:50.430 align:middle line:84%
So if it looks like this, I
flip it this way horizontal.

01:04:50.430 --> 01:04:52.770 align:middle line:84%
And then I randomly
rotate it by 0.1.

01:04:52.770 --> 01:04:54.730 align:middle line:84%
I forget if it's 0.1
degrees or radians.

01:04:54.730 --> 01:04:56.190 align:middle line:84%
You can look up
the documentation.

01:04:56.190 --> 01:05:00.110 align:middle line:84%
And then random zoom, zoom
in and out a little bit.

01:05:00.110 --> 01:05:03.030 align:middle line:84%
But it won't do this
for every picture.

01:05:03.030 --> 01:05:04.723 align:middle line:90%
You only do it randomly.

01:05:04.723 --> 01:05:07.390 align:middle line:84%
So there are only some pictures
will get perturbed in some ways.

01:05:07.390 --> 01:05:08.390 align:middle line:84%
And that's how you
make sure there's

01:05:08.390 --> 01:05:10.630 align:middle line:84%
enough diversity of
pictures that you have.

01:05:10.630 --> 01:05:13.630 align:middle line:84%
So once you do that, you
can actually take a picture

01:05:13.630 --> 01:05:15.825 align:middle line:90%
and see what it does.

01:05:15.825 --> 01:05:18.450 align:middle line:84%
I just randomly grab a picture
so it keeps changing every time.

01:05:18.450 --> 01:05:21.050 align:middle line:90%


01:05:21.050 --> 01:05:21.550 align:middle line:90%
Yay.

01:05:21.550 --> 01:05:22.910 align:middle line:90%
Look at this handbag.

01:05:22.910 --> 01:05:26.510 align:middle line:84%
Handbag slightly rotated
this way, rotated that way,

01:05:26.510 --> 01:05:29.690 align:middle line:84%
some more, maybe a little bit
of zooming going on, and so on.

01:05:29.690 --> 01:05:30.683 align:middle line:90%
You get the idea.

01:05:30.683 --> 01:05:32.850 align:middle line:84%
And there's a whole list
of these things you can do.

01:05:32.850 --> 01:05:36.837 align:middle line:84%
But when you do those things,
make sure that what you're doing

01:05:36.837 --> 01:05:38.420 align:middle line:84%
doesn't actually
change the underlying

01:05:38.420 --> 01:05:39.740 align:middle line:90%
meaning of the picture.

01:05:39.740 --> 01:05:42.220 align:middle line:90%
It's really important.

01:05:42.220 --> 01:05:45.732 align:middle line:84%
So for example, if you're
working with satellite data,

01:05:45.732 --> 01:05:49.162 align:middle line:84%
you must be very careful not
to do flips of crazy flips,

01:05:49.162 --> 01:05:51.120 align:middle line:84%
or even if you're working
with everyday images,

01:05:51.120 --> 01:05:52.520 align:middle line:90%
horizontal flips are OK.

01:05:52.520 --> 01:05:54.728 align:middle line:90%
Don't do vertical flips.

01:05:54.728 --> 01:05:57.020 align:middle line:84%
How many times will you have
an upside down dog picture

01:05:57.020 --> 01:05:59.010 align:middle line:90%
that you need to classify?

01:05:59.010 --> 01:06:01.050 align:middle line:84%
Make sure your augmentation
doesn't go nuts.

01:06:01.050 --> 01:06:05.715 align:middle line:90%


01:06:05.715 --> 01:06:07.340 align:middle line:84%
Once you do that,
you can actually just

01:06:07.340 --> 01:06:09.740 align:middle line:84%
insert the data augmentation
layers in your model

01:06:09.740 --> 01:06:11.800 align:middle line:84%
right there, right
after the input.

01:06:11.800 --> 01:06:14.300 align:middle line:84%
The rest of it can
stay unchanged.

01:06:14.300 --> 01:06:15.740 align:middle line:84%
So this is a great
way to increase

01:06:15.740 --> 01:06:17.200 align:middle line:90%
the size of your training data.

01:06:17.200 --> 01:06:18.360 align:middle line:90%
And here is a model.

01:06:18.360 --> 01:06:20.900 align:middle line:84%
And then I invite you to
actually just play with it

01:06:20.900 --> 01:06:21.818 align:middle line:90%
and train it.

01:06:21.818 --> 01:06:24.360 align:middle line:84%
In the interest of time, you
won't actually train this model.

01:06:24.360 --> 01:06:24.960 align:middle line:90%
But it's in the Colab.

01:06:24.960 --> 01:06:25.793 align:middle line:90%
You can just try it.

01:06:25.793 --> 01:06:28.080 align:middle line:84%
It also figures
prominently in homework 1,

01:06:28.080 --> 01:06:29.400 align:middle line:90%
by the way, data augmentation.

01:06:29.400 --> 01:06:31.620 align:middle line:84%
So you'll get more
experience with this.

01:06:31.620 --> 01:06:34.890 align:middle line:90%
So back to the PPT.

01:06:34.890 --> 01:06:36.690 align:middle line:90%
So this is what we have.

01:06:36.690 --> 01:06:40.370 align:middle line:84%
And so any network that has been
trained on this ImageNet thing,

01:06:40.370 --> 01:06:42.970 align:middle line:84%
turns out, learns all kinds
of interesting features

01:06:42.970 --> 01:06:44.350 align:middle line:90%
in every one of its layers.

01:06:44.350 --> 01:06:45.910 align:middle line:90%
So here this is the first layer.

01:06:45.910 --> 01:06:47.930 align:middle line:84%
And you can see it's
picking up gradations

01:06:47.930 --> 01:06:50.810 align:middle line:84%
of color, line-ish
kind of behavior.

01:06:50.810 --> 01:06:53.650 align:middle line:84%
Layer 2 is actually
picking up, hey, look,

01:06:53.650 --> 01:06:55.170 align:middle line:90%
it's picking up an edge.

01:06:55.170 --> 01:06:59.610 align:middle line:84%
Can you see that
edge, like that?

01:06:59.610 --> 01:07:02.810 align:middle line:84%
And then layer 3 is picking
up these interesting honeycomb

01:07:02.810 --> 01:07:04.830 align:middle line:90%
shapes, and so on.

01:07:04.830 --> 01:07:07.850 align:middle line:84%
Oh, it's actually, this
thing is already picking up

01:07:07.850 --> 01:07:09.080 align:middle line:90%
the shape of a human torso.

01:07:09.080 --> 01:07:12.202 align:middle line:90%


01:07:12.202 --> 01:07:13.970 align:middle line:84%
Yeah, this here is
actually picking up

01:07:13.970 --> 01:07:17.290 align:middle line:84%
what looks like a
Labrador retriever.

01:07:17.290 --> 01:07:19.410 align:middle line:90%
Isn't that cute?

01:07:19.410 --> 01:07:23.150 align:middle line:84%
Come on, even if you're
not a dog person.

01:07:23.150 --> 01:07:27.130 align:middle line:84%
So this visualization I
was referring to earlier,

01:07:27.130 --> 01:07:30.130 align:middle line:84%
to figure out what are these
networks actually learning,

01:07:30.130 --> 01:07:31.690 align:middle line:84%
this paper was one
of the first ones

01:07:31.690 --> 01:07:33.710 align:middle line:84%
to actually visualize
what's going on inside.

01:07:33.710 --> 01:07:36.168 align:middle line:84%
So if you folks are curious
how these pictures are actually

01:07:36.168 --> 01:07:39.275 align:middle line:84%
produced, I would encourage
you to check this out.

01:07:39.275 --> 01:07:39.775 align:middle line:90%
Yeah?

01:07:39.775 --> 01:07:44.027 align:middle line:90%


01:07:44.027 --> 01:07:45.610 align:middle line:84%
AUDIENCE: Sorry, we
spoke about images

01:07:45.610 --> 01:07:49.310 align:middle line:84%
and you referred to classes on
text next week on transformers.

01:07:49.310 --> 01:07:52.050 align:middle line:84%
But what about,
say, an email which

01:07:52.050 --> 01:07:53.770 align:middle line:84%
has both text and
image, and there'll

01:07:53.770 --> 01:07:56.510 align:middle line:84%
be whitespace depending
on who has written it?

01:07:56.510 --> 01:08:00.918 align:middle line:84%
How does that get put in
as an input for an image?

01:08:00.918 --> 01:08:03.210 align:middle line:84%
RAMA RAMAKRISHNAN: So we'll
revisit this great question

01:08:03.210 --> 01:08:04.890 align:middle line:90%
a bit later on in the course.

01:08:04.890 --> 01:08:06.470 align:middle line:84%
So the answer is
a bit complicated

01:08:06.470 --> 01:08:07.950 align:middle line:90%
so I want to do it justice.

01:08:07.950 --> 01:08:09.830 align:middle line:90%
So we'll come back to it.

01:08:09.830 --> 01:08:12.170 align:middle line:84%
So it turns out
this thing called

01:08:12.170 --> 01:08:14.690 align:middle line:84%
ResNet is a family of
networks which are trained

01:08:14.690 --> 01:08:16.069 align:middle line:90%
on this ImageNet data set.

01:08:16.069 --> 01:08:18.569 align:middle line:84%
And they did really
well in this competition

01:08:18.569 --> 01:08:21.290 align:middle line:84%
that's associated with dimension
data set called ImageNet.

01:08:21.290 --> 01:08:23.350 align:middle line:84%
And so this is an example
of such a network.

01:08:23.350 --> 01:08:27.470 align:middle line:84%
So you would expect the weights
and the parameters of ResNet,

01:08:27.470 --> 01:08:29.770 align:middle line:84%
given that it's been
trained on ImageNet, to have

01:08:29.770 --> 01:08:32.680 align:middle line:84%
some knowledge about lines,
and shapes, and curves,

01:08:32.680 --> 01:08:33.779 align:middle line:90%
and things like that.

01:08:33.779 --> 01:08:37.700 align:middle line:90%
So maybe we can just use that.

01:08:37.700 --> 01:08:40.399 align:middle line:84%
But the thing is, we can't use
ResNet as is because remember,

01:08:40.399 --> 01:08:42.720 align:middle line:84%
it was trained to classify
an incoming image in 1,000

01:08:42.720 --> 01:08:43.923 align:middle line:90%
possibilities.

01:08:43.923 --> 01:08:45.840 align:middle line:84%
Here, we only have two
possibilities, handbags

01:08:45.840 --> 01:08:46.600 align:middle line:90%
and shoes.

01:08:46.600 --> 01:08:49.319 align:middle line:84%
So what we do is very
simple and elegant.

01:08:49.319 --> 01:08:51.880 align:middle line:84%
We do just a little
bit of surgery.

01:08:51.880 --> 01:08:55.520 align:middle line:84%
We take ResNet and stop
just before the final layer.

01:08:55.520 --> 01:08:57.560 align:middle line:90%
So take my word for it.

01:08:57.560 --> 01:08:59.399 align:middle line:84%
This thing here,
what it says is,

01:08:59.399 --> 01:09:03.060 align:middle line:84%
fully connected 1,000
because it's got 1,000 way,

01:09:03.060 --> 01:09:03.979 align:middle line:90%
1,000 objects.

01:09:03.979 --> 01:09:06.200 align:middle line:84%
So what we do is we just
take everything except,

01:09:06.200 --> 01:09:08.279 align:middle line:84%
and we stop just
before that last layer.

01:09:08.279 --> 01:09:10.279 align:middle line:84%
And then what comes
out of that layer

01:09:10.279 --> 01:09:12.960 align:middle line:84%
hopefully will be a very smart
representation of the images

01:09:12.960 --> 01:09:14.520 align:middle line:90%
that has been trained on.

01:09:14.520 --> 01:09:17.760 align:middle line:84%
And so what we do is we
can think of headless

01:09:17.760 --> 01:09:19.850 align:middle line:90%
ResNet as our model.

01:09:19.850 --> 01:09:22.700 align:middle line:90%


01:09:22.700 --> 01:09:24.359 align:middle line:84%
We can take all
our data and run it

01:09:24.359 --> 01:09:28.740 align:middle line:84%
through ResNet, up to but
not including the last layer.

01:09:28.740 --> 01:09:30.069 align:middle line:90%
You'll get some tensor.

01:09:30.069 --> 01:09:32.888 align:middle line:84%
And that tensor probably has
a very rich understanding

01:09:32.888 --> 01:09:34.430 align:middle line:84%
of what's going on
in that image, all

01:09:34.430 --> 01:09:36.569 align:middle line:84%
the objects, and features,
and things like that.

01:09:36.569 --> 01:09:39.553 align:middle line:84%
And then we can just
simply connect that.

01:09:39.553 --> 01:09:41.970 align:middle line:84%
We can think of it as a smart
representation of our input.

01:09:41.970 --> 01:09:44.370 align:middle line:84%
We can connect it to just
a little hidden layer.

01:09:44.370 --> 01:09:45.830 align:middle line:84%
And then we have
a little sigmoid

01:09:45.830 --> 01:09:47.609 align:middle line:90%
which tells you handbag or shoe.

01:09:47.609 --> 01:09:50.350 align:middle line:90%
We can just run this network.

01:09:50.350 --> 01:09:53.590 align:middle line:84%
And so since the outputs
to the hidden layer

01:09:53.590 --> 01:09:57.110 align:middle line:84%
now are not raw images anymore,
but this much higher level

01:09:57.110 --> 01:09:59.050 align:middle line:84%
of abstraction that
ResNet has learned,

01:09:59.050 --> 01:10:02.510 align:middle line:84%
hopefully it can get the job
done with hardly any examples.

01:10:02.510 --> 01:10:04.563 align:middle line:90%
And now you can get fancier.

01:10:04.563 --> 01:10:06.730 align:middle line:84%
That's the basic idea, but
you can get much fancier.

01:10:06.730 --> 01:10:09.138 align:middle line:84%
You can connect up
headless ResNet directly

01:10:09.138 --> 01:10:10.930 align:middle line:84%
with our little network
with a hidden layer

01:10:10.930 --> 01:10:11.763 align:middle line:90%
and the final thing.

01:10:11.763 --> 01:10:15.948 align:middle line:84%
And the whole thing can
be trained end to end.

01:10:15.948 --> 01:10:17.990 align:middle line:84%
But when you do that, you
must start the training

01:10:17.990 --> 01:10:20.123 align:middle line:84%
with the weights that you
downloaded with ResNet

01:10:20.123 --> 01:10:22.290 align:middle line:84%
because that is the crown
jewel that's been learned.

01:10:22.290 --> 01:10:24.070 align:middle line:84%
So you'll want to
start from there.

01:10:24.070 --> 01:10:27.570 align:middle line:84%
And you will do
this in homework 1.

01:10:27.570 --> 01:10:29.070 align:middle line:84%
By the way, these
pre-trained models

01:10:29.070 --> 01:10:30.570 align:middle line:84%
are available all
over the internet.

01:10:30.570 --> 01:10:32.570 align:middle line:84%
There is the TensorFlow
Hub, the PyTorch Hub,

01:10:32.570 --> 01:10:34.195 align:middle line:84%
and then there is
the Hugging Face Hub.

01:10:34.195 --> 01:10:36.110 align:middle line:84%
When I checked it on
the 13th, yesterday,

01:10:36.110 --> 01:10:39.770 align:middle line:84%
it had over half a million
models available for download,

01:10:39.770 --> 01:10:41.270 align:middle line:90%
half a million.

01:10:41.270 --> 01:10:44.750 align:middle line:84%
I think last year it was 50,000
when I taught the course.

01:10:44.750 --> 01:10:46.990 align:middle line:90%
Yes?

01:10:46.990 --> 01:10:49.070 align:middle line:84%
AUDIENCE: I was just
wondering, doesn't this

01:10:49.070 --> 01:10:51.510 align:middle line:84%
make you prone to
adversarial attacks

01:10:51.510 --> 01:10:54.290 align:middle line:84%
because the weights
are known by everybody?

01:10:54.290 --> 01:10:55.390 align:middle line:90%
RAMA RAMAKRISHNAN: Yes.

01:10:55.390 --> 01:10:56.890 align:middle line:90%
There is some adversarial risk.

01:10:56.890 --> 01:10:59.230 align:middle line:84%
I'm happy to talk
about it offline.

01:10:59.230 --> 01:10:59.810 align:middle line:90%
All right.

01:10:59.810 --> 01:11:02.790 align:middle line:84%
So that's what we
have, so back to Colab.

01:11:02.790 --> 01:11:04.290 align:middle line:90%
So that's what we have.

01:11:04.290 --> 01:11:06.390 align:middle line:90%
This is ResNet.

01:11:06.390 --> 01:11:07.650 align:middle line:90%
And ResNet is all packaged up.

01:11:07.650 --> 01:11:10.560 align:middle line:84%
It's available for download,
so we download it here.

01:11:10.560 --> 01:11:13.710 align:middle line:90%


01:11:13.710 --> 01:11:19.430 align:middle line:84%
And you see here that I'm
saying, include_top=False.

01:11:19.430 --> 01:11:21.990 align:middle line:84%
So basically, you
are telling Keras,

01:11:21.990 --> 01:11:25.010 align:middle line:84%
the top, the very final layer of
the thing, don't give it to me.

01:11:25.010 --> 01:11:27.380 align:middle line:84%
Just give me everything up
to but not including that.

01:11:27.380 --> 01:11:29.660 align:middle line:84%
And of course, I think
of it as left to right.

01:11:29.660 --> 01:11:31.500 align:middle line:84%
People think of it
as bottom to top.

01:11:31.500 --> 01:11:33.833 align:middle line:84%
So the very, very top
layer, don't give it to me.

01:11:33.833 --> 01:11:35.500 align:middle line:84%
You're telling it so
that you don't have

01:11:35.500 --> 01:11:37.380 align:middle line:90%
to manually go and remove it.

01:11:37.380 --> 01:11:39.697 align:middle line:84%
And then I'm not
going to summarize.

01:11:39.697 --> 01:11:42.280 align:middle line:84%
Well, I'll just summarize the
model to show you how big it is.

01:11:42.280 --> 01:11:45.860 align:middle line:90%


01:11:45.860 --> 01:11:49.820 align:middle line:90%
23 million parameters, ResNet.

01:11:49.820 --> 01:11:51.500 align:middle line:84%
And I won't plot it
because then I'll

01:11:51.500 --> 01:11:53.540 align:middle line:90%
be scrolling for five minutes.

01:11:53.540 --> 01:11:54.960 align:middle line:90%
So let's just do this now.

01:11:54.960 --> 01:11:56.100 align:middle line:84%
So what we're now
going to do is we're

01:11:56.100 --> 01:11:57.920 align:middle line:84%
going to run all the
data through this thing.

01:11:57.920 --> 01:12:00.000 align:middle line:84%
And whatever comes out in
that penultimate thing,

01:12:00.000 --> 01:12:01.900 align:middle line:84%
I'm going to just
grab it and store it.

01:12:01.900 --> 01:12:05.500 align:middle line:90%
So that's what this thing does.

01:12:05.500 --> 01:12:06.820 align:middle line:90%
And now we create--

01:12:06.820 --> 01:12:09.580 align:middle line:84%
this is a little, handy
function to do all these things.

01:12:09.580 --> 01:12:11.900 align:middle line:84%
And once I do that,
every image has

01:12:11.900 --> 01:12:15.137 align:middle line:84%
been sent through ResNet, up
to but not the final layer.

01:12:15.137 --> 01:12:17.720 align:middle line:84%
And then whatever comes to the
final layer, we are storing it.

01:12:17.720 --> 01:12:19.637 align:middle line:84%
And then we're going to
create a network where

01:12:19.637 --> 01:12:23.300 align:middle line:84%
we'll only feed that layer, that
information to a simple network.

01:12:23.300 --> 01:12:26.120 align:middle line:90%
So what is coming out of ResNet?

01:12:26.120 --> 01:12:29.360 align:middle line:84%
You can see here 98 examples
in the training data.

01:12:29.360 --> 01:12:33.608 align:middle line:84%
And each example is now
a 7-by-7-by-2,048 tensor.

01:12:33.608 --> 01:12:34.900 align:middle line:90%
That's what came out of ResNet.

01:12:34.900 --> 01:12:38.200 align:middle line:84%
And you saw that's
what I did there.

01:12:38.200 --> 01:12:39.500 align:middle line:90%
So that's what it looks like.

01:12:39.500 --> 01:12:42.440 align:middle line:84%
Now let's just create
our actual model now.

01:12:42.440 --> 01:12:46.540 align:middle line:84%
We have our input, which is
just a 7 by 7 by 224, 2,048.

01:12:46.540 --> 01:12:49.380 align:middle line:84%
We flatten it immediately,
then we run it

01:12:49.380 --> 01:12:51.960 align:middle line:84%
through a dense layer
with 256 ReLU neurons.

01:12:51.960 --> 01:12:54.600 align:middle line:84%
And then we use Dropout, which
I haven't talked about yet,

01:12:54.600 --> 01:12:57.557 align:middle line:84%
which I will talk
about early next week.

01:12:57.557 --> 01:12:58.640 align:middle line:90%
And we'll come back to it.

01:12:58.640 --> 01:13:00.740 align:middle line:84%
Don't worry about this
detail for the moment.

01:13:00.740 --> 01:13:06.100 align:middle line:84%
And then we just run through a
sigmoid and that's our model.

01:13:06.100 --> 01:13:07.820 align:middle line:90%
Finished, plot the model.

01:13:07.820 --> 01:13:09.720 align:middle line:90%
This is what we have.

01:13:09.720 --> 01:13:14.360 align:middle line:84%
model.summary, so
on, and so forth.

01:13:14.360 --> 01:13:15.200 align:middle line:90%
All right, good.

01:13:15.200 --> 01:13:18.780 align:middle line:84%
Now let's actually
train this thing.

01:13:18.780 --> 01:13:20.620 align:middle line:84%
I'm just going to
run it for 10 epochs

01:13:20.620 --> 01:13:22.800 align:middle line:84%
because I tried
running it previously

01:13:22.800 --> 01:13:25.510 align:middle line:84%
and it seems to do a fine
job with just 10 epochs.

01:13:25.510 --> 01:13:26.590 align:middle line:90%
OK, it's already done.

01:13:26.590 --> 01:13:29.610 align:middle line:84%
It's so fast because we ran
everything through this monster

01:13:29.610 --> 01:13:33.488 align:middle line:84%
ResNet thing, and basically took
all the output values and used

01:13:33.488 --> 01:13:34.530 align:middle line:90%
them as a starting point.

01:13:34.530 --> 01:13:36.490 align:middle line:84%
We don't have to run
it every single time.

01:13:36.490 --> 01:13:42.212 align:middle line:84%
So you can see here, the
accuracy is quite high.

01:13:42.212 --> 01:13:44.510 align:middle line:90%
Let me just plot it.

01:13:44.510 --> 01:13:45.010 align:middle line:90%
Interesting.

01:13:45.010 --> 01:13:48.570 align:middle line:84%
So the 10th epoch,
something bad happened.

01:13:48.570 --> 01:13:50.670 align:middle line:84%
So maybe I should have
stopped at the 9th epoch.

01:13:50.670 --> 01:13:52.628 align:middle line:84%
I didn't see this yesterday
when I was running.

01:13:52.628 --> 01:13:55.210 align:middle line:84%
So much for random
reproducibility.

01:13:55.210 --> 01:13:56.410 align:middle line:90%
So let's just run this.

01:13:56.410 --> 01:13:56.910 align:middle line:90%
Oh, wow.

01:13:56.910 --> 01:14:01.610 align:middle line:84%
Look, on the test set, it's
achieving 100% accuracy.

01:14:01.610 --> 01:14:03.690 align:middle line:90%
It's unbelievable.

01:14:03.690 --> 01:14:06.410 align:middle line:84%
OK, folks, now for
the moment of truth.

01:14:06.410 --> 01:14:06.910 align:middle line:90%
All right.

01:14:06.910 --> 01:14:08.285 align:middle line:84%
I have a little
code snippet here

01:14:08.285 --> 01:14:10.050 align:middle line:84%
to capture stuff
from the webcam.

01:14:10.050 --> 01:14:12.270 align:middle line:84%
Because that last
epoch it went down,

01:14:12.270 --> 01:14:14.970 align:middle line:84%
I'm a little worried that
the demo is going to flunk.

01:14:14.970 --> 01:14:15.950 align:middle line:90%
But you know what?

01:14:15.950 --> 01:14:19.690 align:middle line:84%
We all have to live dangerously,
so here's a little function

01:14:19.690 --> 01:14:21.690 align:middle line:84%
to predict what's
going to happen.

01:14:21.690 --> 01:14:25.320 align:middle line:84%
OK, now I tried it at home
yesterday, by the way,

01:14:25.320 --> 01:14:28.040 align:middle line:84%
and it's like, yeah,
it's a handbag.

01:14:28.040 --> 01:14:30.800 align:middle line:84%
OK, now let's just
do something else.

01:14:30.800 --> 01:14:32.600 align:middle line:90%
OK, any volunteers?

01:14:32.600 --> 01:14:35.570 align:middle line:84%
I want a piece of
footwear or a handbag.

01:14:35.570 --> 01:14:36.820 align:middle line:90%
AUDIENCE: I'll give a handbag.

01:14:36.820 --> 01:14:39.195 align:middle line:84%
RAMA RAMAKRISHNAN: That's kind
of like a backpack, right?

01:14:39.195 --> 01:14:39.838 align:middle line:90%
[LAUGHTER]

01:14:39.838 --> 01:14:40.380 align:middle line:90%
I don't know.

01:14:40.380 --> 01:14:42.005 align:middle line:84%
This feels like an
adversarial example,

01:14:42.005 --> 01:14:43.560 align:middle line:90%
but yeah, let's just try it.

01:14:43.560 --> 01:14:45.160 align:middle line:90%
AUDIENCE: Someone else?

01:14:45.160 --> 01:14:46.660 align:middle line:84%
RAMA RAMAKRISHNAN:
No disrespect.

01:14:46.660 --> 01:14:48.440 align:middle line:90%
Let me go with the shoe first.

01:14:48.440 --> 01:14:51.080 align:middle line:84%
I have a better
chance of it working.

01:14:51.080 --> 01:14:52.680 align:middle line:90%
So it's a pretty big shoe.

01:14:52.680 --> 01:14:56.040 align:middle line:84%
If it can't get this shoe,
I'm worried about this model.

01:14:56.040 --> 01:14:56.660 align:middle line:90%
All right.

01:14:56.660 --> 01:15:05.200 align:middle line:90%


01:15:05.200 --> 01:15:05.700 align:middle line:90%
OK.

01:15:05.700 --> 01:15:08.000 align:middle line:90%
Hold on, hold on, hold on.

01:15:08.000 --> 01:15:10.800 align:middle line:90%
All right.

01:15:10.800 --> 01:15:14.640 align:middle line:84%
Please don't get
distracted by my hand.

01:15:14.640 --> 01:15:17.040 align:middle line:90%
Capture.

01:15:17.040 --> 01:15:18.040 align:middle line:90%
It's a shoe!

01:15:18.040 --> 01:15:18.900 align:middle line:90%
Look at that!

01:15:18.900 --> 01:15:21.840 align:middle line:90%
[APPLAUSE]

01:15:21.840 --> 01:15:22.560 align:middle line:90%
Phew.

01:15:22.560 --> 01:15:25.040 align:middle line:90%
All right, thanks.

01:15:25.040 --> 01:15:26.340 align:middle line:90%
OK, now let's try that.

01:15:26.340 --> 01:15:27.608 align:middle line:90%
I'm feeling brave now.

01:15:27.608 --> 01:15:28.400 align:middle line:90%
AUDIENCE: OK, yeah.

01:15:28.400 --> 01:15:29.608 align:middle line:90%
RAMA RAMAKRISHNAN: Thank you.

01:15:29.608 --> 01:15:32.880 align:middle line:90%
All right, let's do this.

01:15:32.880 --> 01:15:35.080 align:middle line:90%
All right.

01:15:35.080 --> 01:15:35.810 align:middle line:90%
Camera Capture.

01:15:35.810 --> 01:15:40.473 align:middle line:90%


01:15:40.473 --> 01:15:40.973 align:middle line:90%
OK.

01:15:40.973 --> 01:15:43.691 align:middle line:90%


01:15:43.691 --> 01:15:45.235 align:middle line:90%
I'll put its better side.

01:15:45.235 --> 01:15:45.860 align:middle line:90%
AUDIENCE: Yeah.

01:15:45.860 --> 01:15:54.883 align:middle line:90%


01:15:54.883 --> 01:15:56.300 align:middle line:84%
RAMA RAMAKRISHNAN:
It's a handbag!

01:15:56.300 --> 01:15:57.411 align:middle line:90%
Look at that!

01:15:57.411 --> 01:15:58.704 align:middle line:90%
[APPLAUSE]

01:15:58.704 --> 01:16:00.000 align:middle line:90%
Phew.

01:16:00.000 --> 01:16:03.455 align:middle line:84%
I swear, every time I do
the demo, I age a few years.

01:16:03.455 --> 01:16:04.580 align:middle line:90%
All right, folks, I'm done.

01:16:04.580 --> 01:16:06.350 align:middle line:90%
Thank you.

01:16:06.350 --> 01:16:22.000 align:middle line:90%