WEBVTT

00:00:00.000 --> 00:00:00.992 align:middle line:90%


00:00:00.992 --> 00:00:02.976 align:middle line:90%
[SQUEAKING]

00:00:02.976 --> 00:00:04.960 align:middle line:90%
[RUSTLING]

00:00:04.960 --> 00:00:16.425 align:middle line:90%
[CLICKING]

00:00:16.425 --> 00:00:17.300 align:middle line:90%
PROFESSOR: All right.

00:00:17.300 --> 00:00:19.440 align:middle line:84%
So today, we actually come to
the last lecture of the class,

00:00:19.440 --> 00:00:21.780 align:middle line:84%
because Wednesday is going
to be project presentations.

00:00:21.780 --> 00:00:25.700 align:middle line:84%
And so I want to talk to you
about diffusion models today,

00:00:25.700 --> 00:00:29.520 align:middle line:84%
which is an incredibly exciting
area, which I don't think

00:00:29.520 --> 00:00:32.360 align:middle line:84%
gets the same amount of
attention, in some ways,

00:00:32.360 --> 00:00:34.720 align:middle line:84%
compared to large
language models.

00:00:34.720 --> 00:00:37.480 align:middle line:90%
But it's got enormous potential.

00:00:37.480 --> 00:00:39.960 align:middle line:84%
So I'm very excited to
talk to you about it.

00:00:39.960 --> 00:00:43.040 align:middle line:84%
So just for kicks
last night, I asked

00:00:43.040 --> 00:00:45.440 align:middle line:84%
ChatGPT to create a
photorealistic image

00:00:45.440 --> 00:00:48.320 align:middle line:84%
of graduate students in class,
in a class on deep learning,

00:00:48.320 --> 00:00:51.640 align:middle line:84%
and this is what
it came back with.

00:00:51.640 --> 00:00:54.816 align:middle line:84%
There is a noticeable
absence of an instructor.

00:00:54.816 --> 00:00:57.200 align:middle line:90%
[LAUGHS]

00:00:57.200 --> 00:01:01.350 align:middle line:84%
Plus, various students are
facing in various directions.

00:01:01.350 --> 00:01:04.750 align:middle line:84%
But apart from
that, it's not bad.

00:01:04.750 --> 00:01:08.990 align:middle line:84%
And here is an example
of a Midjourney

00:01:08.990 --> 00:01:11.990 align:middle line:84%
text-to-image
diffusion model, which

00:01:11.990 --> 00:01:15.230 align:middle line:84%
produces the amazing
picture from this prompt.

00:01:15.230 --> 00:01:17.930 align:middle line:84%
A quaint Italian seaside
village with colorful buildings,

00:01:17.930 --> 00:01:21.430 align:middle line:84%
blah blah, blah, blah,
blah, rendered in the style

00:01:21.430 --> 00:01:23.930 align:middle line:84%
of Claude Monet and
so on and so forth.

00:01:23.930 --> 00:01:24.970 align:middle line:90%
And that's what you get.

00:01:24.970 --> 00:01:26.790 align:middle line:90%
It's pretty unbelievable.

00:01:26.790 --> 00:01:29.290 align:middle line:84%
And I'm sure you folks have
played around with these things,

00:01:29.290 --> 00:01:31.248 align:middle line:84%
and you have your favorite
pictures and prompts

00:01:31.248 --> 00:01:33.550 align:middle line:90%
and whatnot.

00:01:33.550 --> 00:01:39.990 align:middle line:84%
Now, February 15, OpenAI
released a text-to-video model

00:01:39.990 --> 00:01:43.790 align:middle line:84%
called Sora, which you
folks may have seen,

00:01:43.790 --> 00:01:46.530 align:middle line:84%
which I find frankly, just
stunning what it can do.

00:01:46.530 --> 00:01:50.630 align:middle line:84%
It can produce a one minute
video from a text prompt.

00:01:50.630 --> 00:01:54.790 align:middle line:84%
And so if you actually
give it this prompt,

00:01:54.790 --> 00:01:58.220 align:middle line:84%
"In an ornate historical hall,
a massive tidal wave peaks

00:01:58.220 --> 00:02:01.760 align:middle line:84%
and begins to crash, and two
surfers, seizing the moment,

00:02:01.760 --> 00:02:04.453 align:middle line:90%
skillfully navigate the wave."

00:02:04.453 --> 00:02:06.620 align:middle line:84%
I think we can all agree
that such a thing has never

00:02:06.620 --> 00:02:08.820 align:middle line:84%
happened in history,
and therefore, it

00:02:08.820 --> 00:02:10.539 align:middle line:90%
was not in the training data.

00:02:10.539 --> 00:02:14.190 align:middle line:84%
And then, you get this
picture, this video.

00:02:14.190 --> 00:02:26.900 align:middle line:90%


00:02:26.900 --> 00:02:29.100 align:middle line:84%
And then, some random
person is coming back

00:02:29.100 --> 00:02:31.722 align:middle line:90%
in a completely dry hall.

00:02:31.722 --> 00:02:33.180 align:middle line:84%
So anyway, but it's
pretty amazing.

00:02:33.180 --> 00:02:34.980 align:middle line:90%
I think you'd agree.

00:02:34.980 --> 00:02:40.780 align:middle line:84%
So if you actually look at the
OpenAI's Sora technical report,

00:02:40.780 --> 00:02:44.380 align:middle line:84%
you actually find
this opening paragraph

00:02:44.380 --> 00:02:46.900 align:middle line:84%
where they say
that we train text

00:02:46.900 --> 00:02:50.860 align:middle line:84%
conditional diffusion models,
blah, blah, blah, using

00:02:50.860 --> 00:02:53.620 align:middle line:90%
a transformer architecture.

00:02:53.620 --> 00:02:56.550 align:middle line:84%
So now, we know what a
transformer architecture is.

00:02:56.550 --> 00:02:57.750 align:middle line:90%
You've been working with it.

00:02:57.750 --> 00:02:59.670 align:middle line:84%
You're quite familiar
with it at this point.

00:02:59.670 --> 00:03:03.090 align:middle line:84%
So today's class is really
about text conditional diffusion

00:03:03.090 --> 00:03:06.090 align:middle line:84%
models, the other
building block.

00:03:06.090 --> 00:03:08.010 align:middle line:90%
OK, so let's get to it.

00:03:08.010 --> 00:03:09.450 align:middle line:84%
What I'm going to
do is I'm going

00:03:09.450 --> 00:03:11.410 align:middle line:90%
to divide this into two parts.

00:03:11.410 --> 00:03:13.410 align:middle line:84%
The first part is, I'm
just going to talk about,

00:03:13.410 --> 00:03:16.110 align:middle line:84%
how do you get a model to just
generate an image for you?

00:03:16.110 --> 00:03:17.850 align:middle line:84%
If you wanted to
generate an image

00:03:17.850 --> 00:03:20.130 align:middle line:84%
from a class of
potential images,

00:03:20.130 --> 00:03:21.710 align:middle line:84%
how can you just
generate an image?

00:03:21.710 --> 00:03:24.030 align:middle line:84%
And then, next, we
talk about, OK, great.

00:03:24.030 --> 00:03:26.770 align:middle line:84%
Now, that you can do that, how
do you actually control or steer

00:03:26.770 --> 00:03:29.130 align:middle line:84%
the model to do an
image based on whatever

00:03:29.130 --> 00:03:30.870 align:middle line:90%
prompting you give it?

00:03:30.870 --> 00:03:31.870 align:middle line:90%
How do you condition it?

00:03:31.870 --> 00:03:32.787 align:middle line:90%
How do you control it?

00:03:32.787 --> 00:03:33.787 align:middle line:90%
Those are all the words.

00:03:33.787 --> 00:03:34.630 align:middle line:90%
How do you steer it?

00:03:34.630 --> 00:03:36.250 align:middle line:84%
You'll find all these
synonyms being used heavily

00:03:36.250 --> 00:03:37.210 align:middle line:90%
in the literature.

00:03:37.210 --> 00:03:38.543 align:middle line:90%
That's basically what they mean.

00:03:38.543 --> 00:03:41.410 align:middle line:84%
How do you give it a prompt, and
then steer what gets produced?

00:03:41.410 --> 00:03:42.210 align:middle line:90%
All right.

00:03:42.210 --> 00:03:44.610 align:middle line:84%
So let's say you want
to build a model that

00:03:44.610 --> 00:03:49.610 align:middle line:84%
can be used to generate images
of stately college buildings.

00:03:49.610 --> 00:03:51.650 align:middle line:84%
Obviously, our very
own Killian Court

00:03:51.650 --> 00:03:54.880 align:middle line:84%
is the finest example
of such a thing.

00:03:54.880 --> 00:03:57.080 align:middle line:84%
But let's say you
want to do that.

00:03:57.080 --> 00:04:00.220 align:middle line:84%
So what you do is you, as we
always do with machine learning,

00:04:00.220 --> 00:04:01.620 align:middle line:90%
we collect a bunch of data.

00:04:01.620 --> 00:04:04.080 align:middle line:84%
In this particular case, we
collect a whole bunch of images

00:04:04.080 --> 00:04:06.200 align:middle line:90%
of stately college buildings.

00:04:06.200 --> 00:04:07.760 align:middle line:84%
And what you see
here is literally me

00:04:07.760 --> 00:04:10.440 align:middle line:84%
just doing a Google image search
with the query stately college

00:04:10.440 --> 00:04:11.280 align:middle line:90%
buildings.

00:04:11.280 --> 00:04:13.600 align:middle line:84%
So this is the kind
of stuff you get.

00:04:13.600 --> 00:04:15.600 align:middle line:84%
So you have your training
data at your disposal.

00:04:15.600 --> 00:04:17.040 align:middle line:90%
It's ready to go.

00:04:17.040 --> 00:04:20.079 align:middle line:84%
Now, the question is, if you
have such a model, let's say.

00:04:20.079 --> 00:04:23.140 align:middle line:84%
And obviously, we talk about how
to build such a model very soon.

00:04:23.140 --> 00:04:24.760 align:middle line:84%
But let's say you
have such a model,

00:04:24.760 --> 00:04:26.980 align:middle line:84%
and every time you
sample this model,

00:04:26.980 --> 00:04:29.320 align:middle line:84%
every time you ask the
model, hey, give me an image,

00:04:29.320 --> 00:04:31.320 align:middle line:84%
you obviously want it to
give a different image,

00:04:31.320 --> 00:04:34.560 align:middle line:90%
otherwise it's kind of boring.

00:04:34.560 --> 00:04:36.340 align:middle line:84%
Maybe you want
the Killian Court,

00:04:36.340 --> 00:04:39.200 align:middle line:84%
maybe you want the rotunda
from University of Virginia.

00:04:39.200 --> 00:04:42.260 align:middle line:90%
Any UVA alums here?

00:04:42.260 --> 00:04:42.760 align:middle line:90%
Nobody?

00:04:42.760 --> 00:04:45.220 align:middle line:90%
OK.

00:04:45.220 --> 00:04:48.040 align:middle line:84%
So the question is, how can we
actually get it to randomly give

00:04:48.040 --> 00:04:49.340 align:middle line:90%
us different images?

00:04:49.340 --> 00:04:51.990 align:middle line:84%
But they all have to be
stately college buildings.

00:04:51.990 --> 00:04:54.310 align:middle line:84%
It can't be just
some random stuff.

00:04:54.310 --> 00:04:55.550 align:middle line:90%
So how do you do that?

00:04:55.550 --> 00:04:59.670 align:middle line:84%
And the way we do that, and I
still find it really astonishing

00:04:59.670 --> 00:05:01.470 align:middle line:84%
that this approach
actually works.

00:05:01.470 --> 00:05:05.790 align:middle line:84%
The way we do that is that
we actually give it noise.

00:05:05.790 --> 00:05:07.590 align:middle line:84%
And I will define
very precisely what

00:05:07.590 --> 00:05:10.850 align:middle line:90%
I mean by noise in just a bit.

00:05:10.850 --> 00:05:14.910 align:middle line:84%
Basically, assume an image
in which all the pixel values

00:05:14.910 --> 00:05:18.310 align:middle line:90%
are randomly picked.

00:05:18.310 --> 00:05:20.390 align:middle line:84%
So every time you
generate a random image

00:05:20.390 --> 00:05:22.110 align:middle line:84%
and you give it to
the model, it will

00:05:22.110 --> 00:05:25.190 align:middle line:84%
use that random starting
point and then create

00:05:25.190 --> 00:05:26.130 align:middle line:90%
an image for you.

00:05:26.130 --> 00:05:29.590 align:middle line:84%
And because, by definition,
if you choose noise randomly,

00:05:29.590 --> 00:05:32.370 align:middle line:84%
they are obviously going
to be different each time.

00:05:32.370 --> 00:05:34.870 align:middle line:84%
It's hopefully going to
generate a different image.

00:05:34.870 --> 00:05:38.370 align:middle line:84%
But if the model is trained
on stately college buildings,

00:05:38.370 --> 00:05:40.990 align:middle line:84%
it will produce images of
stately college buildings.

00:05:40.990 --> 00:05:44.350 align:middle line:84%
It's not going to produce a
picture of a Labrador Retriever.

00:05:44.350 --> 00:05:47.430 align:middle line:84%
So that's basically
what we're going to do.

00:05:47.430 --> 00:05:51.680 align:middle line:84%
Now, if you look at something
like this, the first question,

00:05:51.680 --> 00:05:53.500 align:middle line:84%
of course, is that how
can we train a model

00:05:53.500 --> 00:05:56.140 align:middle line:84%
to generate an image
from pure noise?

00:05:56.140 --> 00:05:59.102 align:middle line:90%
This just sounds ridiculous.

00:05:59.102 --> 00:06:01.060 align:middle line:84%
You basically give it a
bunch of random numbers

00:06:01.060 --> 00:06:04.540 align:middle line:90%
and say, give me clean code.

00:06:04.540 --> 00:06:06.120 align:middle line:90%
It feels really ridiculous.

00:06:06.120 --> 00:06:09.018 align:middle line:84%
And at that point,
folks can come to a stop

00:06:09.018 --> 00:06:11.060 align:middle line:84%
and say, all right, this
approach is probably not

00:06:11.060 --> 00:06:12.180 align:middle line:90%
going to take me anywhere.

00:06:12.180 --> 00:06:14.060 align:middle line:90%
It's a bit of a dead end.

00:06:14.060 --> 00:06:18.340 align:middle line:84%
But then, some clever people
had this very interesting idea.

00:06:18.340 --> 00:06:24.100 align:middle line:84%
They said, it's not
clear how to do this.

00:06:24.100 --> 00:06:28.000 align:middle line:84%
Just a quick aside, there's
this really amazing book,

00:06:28.000 --> 00:06:30.780 align:middle line:84%
which was published maybe
50 years ago, maybe earlier

00:06:30.780 --> 00:06:34.540 align:middle line:84%
than that, called How to
Solve It, by George Polya.

00:06:34.540 --> 00:06:38.020 align:middle line:84%
George Polya was an
eminent mathematician,

00:06:38.020 --> 00:06:40.640 align:middle line:84%
and he wrote this small
book called How to Solve It,

00:06:40.640 --> 00:06:43.260 align:middle line:84%
and it lists a whole
bunch of heuristics

00:06:43.260 --> 00:06:45.920 align:middle line:84%
that mathematicians use
when they solve problems.

00:06:45.920 --> 00:06:48.460 align:middle line:84%
And perhaps the most
commonly used heuristic

00:06:48.460 --> 00:06:51.917 align:middle line:90%
is just reverse the question.

00:06:51.917 --> 00:06:54.500 align:middle line:84%
Just reverse the question and
see if anything comes out of it.

00:06:54.500 --> 00:06:56.417 align:middle line:84%
Most of the time, nothing
will come out of it.

00:06:56.417 --> 00:06:58.980 align:middle line:84%
But maybe some other times,
something amazing comes out.

00:06:58.980 --> 00:07:01.140 align:middle line:84%
This is a great example
of that heuristic at work.

00:07:01.140 --> 00:07:02.940 align:middle line:90%
We don't know how to do this.

00:07:02.940 --> 00:07:05.640 align:middle line:84%
So the question is,
can we do the reverse?

00:07:05.640 --> 00:07:07.140 align:middle line:84%
If I give you Killian
Court, can you

00:07:07.140 --> 00:07:10.380 align:middle line:90%
produce noise out of it for me?

00:07:10.380 --> 00:07:15.140 align:middle line:84%
And the answer is yeah,
of course, we can do that.

00:07:15.140 --> 00:07:18.560 align:middle line:84%
Given an image, we can easily
create a noisy version of it.

00:07:18.560 --> 00:07:20.780 align:middle line:84%
So you can take
the original image,

00:07:20.780 --> 00:07:22.817 align:middle line:84%
you can add some noise
to it to get this.

00:07:22.817 --> 00:07:24.400 align:middle line:84%
And you keep on
adding a lot of noise.

00:07:24.400 --> 00:07:26.525 align:middle line:84%
And finally, you'll get
something that's basically,

00:07:26.525 --> 00:07:29.340 align:middle line:84%
you can't tell that is
Killian Court anymore.

00:07:29.340 --> 00:07:31.100 align:middle line:84%
This process, the
reverse process,

00:07:31.100 --> 00:07:34.380 align:middle line:90%
is actually very easy to do.

00:07:34.380 --> 00:07:36.340 align:middle line:84%
By the way, for
folks who may not

00:07:36.340 --> 00:07:38.660 align:middle line:84%
be very familiar with this
notion of adding noise

00:07:38.660 --> 00:07:40.600 align:middle line:84%
to an image or making
an image noisy,

00:07:40.600 --> 00:07:44.880 align:middle line:84%
let me just show you in a Colab,
just a minute, how easy it is.

00:07:44.880 --> 00:07:47.450 align:middle line:90%


00:07:47.450 --> 00:07:48.070 align:middle line:90%
All right.

00:07:48.070 --> 00:07:52.430 align:middle line:84%
So let's say we import
a bunch of these things.

00:07:52.430 --> 00:07:53.715 align:middle line:90%
As usual, we have NumPy.

00:07:53.715 --> 00:07:55.090 align:middle line:84%
And so there is
this thing called

00:07:55.090 --> 00:07:57.690 align:middle line:84%
the Python Imaging
Library, PIL, which

00:07:57.690 --> 00:07:59.930 align:middle line:84%
is very handy for
image manipulations.

00:07:59.930 --> 00:08:01.270 align:middle line:90%
So we import that.

00:08:01.270 --> 00:08:03.490 align:middle line:84%
And then, I just literally
read this image in.

00:08:03.490 --> 00:08:04.910 align:middle line:90%
I uploaded it before class.

00:08:04.910 --> 00:08:06.202 align:middle line:90%
Let's just make sure it's here.

00:08:06.202 --> 00:08:06.770 align:middle line:90%
OK, good.

00:08:06.770 --> 00:08:07.850 align:middle line:90%
killian.png.

00:08:07.850 --> 00:08:09.650 align:middle line:90%
So I read this image.

00:08:09.650 --> 00:08:10.890 align:middle line:90%
OK.

00:08:10.890 --> 00:08:14.890 align:middle line:84%
And then, once I read it, I
convert it into a NumPy array.

00:08:14.890 --> 00:08:17.770 align:middle line:84%
And then, remember,
in any color image,

00:08:17.770 --> 00:08:20.390 align:middle line:84%
you have three
tables of numbers.

00:08:20.390 --> 00:08:23.870 align:middle line:84%
There is a number for each
pixel for red, blue, and green.

00:08:23.870 --> 00:08:26.690 align:middle line:84%
And then, each number
is between 0 and 255.

00:08:26.690 --> 00:08:28.770 align:middle line:84%
And so here, what we do
is we divide everything

00:08:28.770 --> 00:08:31.790 align:middle line:84%
by 255 just to normalize it
so it's all between 0 and 1.

00:08:31.790 --> 00:08:33.610 align:middle line:84%
And we have done
this in the past.

00:08:33.610 --> 00:08:36.350 align:middle line:90%
I do that here.

00:08:36.350 --> 00:08:38.912 align:middle line:84%
So let me just read this
back in, convert it.

00:08:38.912 --> 00:08:40.370 align:middle line:84%
And then, if you
look at the shape,

00:08:40.370 --> 00:08:44.800 align:middle line:84%
it's basically a
411 by 583 times 3.

00:08:44.800 --> 00:08:46.760 align:middle line:84%
Three channels, as
we have seen before.

00:08:46.760 --> 00:08:49.160 align:middle line:90%
And then, let's just show it.

00:08:49.160 --> 00:08:50.840 align:middle line:90%
All right, that's the picture.

00:08:50.840 --> 00:08:52.480 align:middle line:84%
So now, what we want
to do is we want

00:08:52.480 --> 00:08:54.240 align:middle line:90%
to add noise to this picture.

00:08:54.240 --> 00:08:58.960 align:middle line:84%
All we have to do
for each pixel,

00:08:58.960 --> 00:09:02.880 align:middle line:84%
we basically, randomly
pick a normal variable,

00:09:02.880 --> 00:09:05.760 align:middle line:84%
a normal distribution, normally
distributed random variable

00:09:05.760 --> 00:09:08.740 align:middle line:84%
with a mean of 0 and a
small standard deviation.

00:09:08.740 --> 00:09:10.380 align:middle line:90%
So it's like a small number.

00:09:10.380 --> 00:09:13.520 align:middle line:84%
And then, we just literally
add that number to every pixel.

00:09:13.520 --> 00:09:16.262 align:middle line:84%
But for every pixel we
sample, every pixel we sample,

00:09:16.262 --> 00:09:18.720 align:middle line:84%
it's not like we sample once
an array of all the pixels, we

00:09:18.720 --> 00:09:20.520 align:middle line:90%
sample for every pixel.

00:09:20.520 --> 00:09:23.840 align:middle line:84%
And so the way you do that
is basically, literally

00:09:23.840 --> 00:09:26.560 align:middle line:90%
np.random.normal.

00:09:26.560 --> 00:09:29.540 align:middle line:84%
And then, this 0.3 here
is a standard deviation.

00:09:29.540 --> 00:09:34.560 align:middle line:84%
And we tell it, generate as many
of these things as the shape

00:09:34.560 --> 00:09:36.320 align:middle line:90%
of the image that I gave you.

00:09:36.320 --> 00:09:39.940 align:middle line:84%
And then, add each one of these
numbers to the original image.

00:09:39.940 --> 00:09:42.150 align:middle line:90%
You get this noisy image.

00:09:42.150 --> 00:09:44.210 align:middle line:84%
So if this is the
original image,

00:09:44.210 --> 00:09:46.210 align:middle line:84%
these are all the
values between 0 and 1.

00:09:46.210 --> 00:09:47.893 align:middle line:84%
And then, you add
do this noisy image.

00:09:47.893 --> 00:09:49.810 align:middle line:84%
You can see the numbers
have become different.

00:09:49.810 --> 00:09:52.590 align:middle line:84%
The 0.23 has become
0.18, 0.15 has become

00:09:52.590 --> 00:09:54.830 align:middle line:90%
-0.17, and so on and so forth.

00:09:54.830 --> 00:09:57.610 align:middle line:84%
You've just added a small
number random to everything.

00:09:57.610 --> 00:10:00.290 align:middle line:84%
But as you can see here, now
you have some negative numbers.

00:10:00.290 --> 00:10:02.290 align:middle line:84%
You may have some numbers
that's greater than 1.

00:10:02.290 --> 00:10:04.690 align:middle line:84%
And we do want everything
to be between 0 and 1.

00:10:04.690 --> 00:10:06.590 align:middle line:84%
So all we do is we
do this thing called

00:10:06.590 --> 00:10:10.390 align:middle line:84%
clip it, where essentially,
values smaller than 0

00:10:10.390 --> 00:10:13.010 align:middle line:84%
are set to 0, values
greater than 1 are set to 1.

00:10:13.010 --> 00:10:15.350 align:middle line:90%
And so we'll just do that.

00:10:15.350 --> 00:10:16.010 align:middle line:90%
That's it.

00:10:16.010 --> 00:10:17.810 align:middle line:84%
Everything over
1, squashed to 1.

00:10:17.810 --> 00:10:19.370 align:middle line:90%
Everything under 0, set to 0.

00:10:19.370 --> 00:10:21.030 align:middle line:90%
Others, leave it unchanged.

00:10:21.030 --> 00:10:23.370 align:middle line:84%
Now, it's again,
well-behaved between 0 and 1.

00:10:23.370 --> 00:10:28.350 align:middle line:84%
And we can just plot
it, and you get this.

00:10:28.350 --> 00:10:28.850 align:middle line:90%
That's it.

00:10:28.850 --> 00:10:31.100 align:middle line:84%
That's all it takes to
actually add noise to an image.

00:10:31.100 --> 00:10:33.578 align:middle line:90%
One line of NumPy.

00:10:33.578 --> 00:10:35.870 align:middle line:84%
Obviously, you can just put
this whole thing in a loop,

00:10:35.870 --> 00:10:37.662 align:middle line:84%
and keep increasing
that standard deviation

00:10:37.662 --> 00:10:40.210 align:middle line:84%
number from 0.3, 0.4,
0.5, so on and so forth.

00:10:40.210 --> 00:10:42.380 align:middle line:84%
And when you do that, you
get this nice sequence

00:10:42.380 --> 00:10:45.740 align:middle line:84%
of clean code, and all the way
to some very, very noisy version

00:10:45.740 --> 00:10:46.860 align:middle line:90%
of clean code.

00:10:46.860 --> 00:10:47.620 align:middle line:90%
That's it.

00:10:47.620 --> 00:10:49.690 align:middle line:84%
So that's the basic
idea of adding noise.

00:10:49.690 --> 00:10:52.940 align:middle line:90%


00:10:52.940 --> 00:10:57.180 align:middle line:90%
Any questions on the mechanics?

00:10:57.180 --> 00:10:58.380 align:middle line:90%
OK, good.

00:10:58.380 --> 00:11:01.160 align:middle line:90%
So we can add random numbers.

00:11:01.160 --> 00:11:03.100 align:middle line:84%
And we can, by
increasing the magnitude

00:11:03.100 --> 00:11:06.740 align:middle line:84%
of the standard deviation of
these normal random variables,

00:11:06.740 --> 00:11:08.820 align:middle line:90%
we can make the image noisier.

00:11:08.820 --> 00:11:14.940 align:middle line:84%
So that suggests a
really interesting idea.

00:11:14.940 --> 00:11:16.080 align:middle line:90%
What idea would that be?

00:11:16.080 --> 00:11:19.620 align:middle line:90%


00:11:19.620 --> 00:11:20.140 align:middle line:90%
Yeah.

00:11:20.140 --> 00:11:21.320 align:middle line:90%
STUDENT: Doing the opposite.

00:11:21.320 --> 00:11:25.108 align:middle line:84%
PROFESSOR: Could you
please, microphone please.

00:11:25.108 --> 00:11:26.400 align:middle line:90%
STUDENT: By doing the opposite.

00:11:26.400 --> 00:11:29.580 align:middle line:84%
Like, recreating the
image from the noise.

00:11:29.580 --> 00:11:32.980 align:middle line:84%
PROFESSOR: So we are trying to
create the image from the noise,

00:11:32.980 --> 00:11:36.740 align:middle line:90%
but that feels a little hard.

00:11:36.740 --> 00:11:38.010 align:middle line:90%
So what exactly can we do?

00:11:38.010 --> 00:11:39.260 align:middle line:90%
Be a little bit more specific.

00:11:39.260 --> 00:11:44.490 align:middle line:90%


00:11:44.490 --> 00:11:46.770 align:middle line:84%
So here, we have the
ability to take any image

00:11:46.770 --> 00:11:49.615 align:middle line:84%
and add any amount
of noise to it.

00:11:49.615 --> 00:11:52.550 align:middle line:84%
That's the data we have
that is Killian Court.

00:11:52.550 --> 00:11:54.980 align:middle line:84%
And there's various noisy
versions of Killian Court,

00:11:54.980 --> 00:11:57.230 align:middle line:84%
like that for the rotunda,
the University of Virginia,

00:11:57.230 --> 00:11:58.210 align:middle line:90%
and so on and so forth.

00:11:58.210 --> 00:11:59.585 align:middle line:84%
STUDENT: I would
assume you would

00:11:59.585 --> 00:12:02.090 align:middle line:84%
do some kind of loss
function for the final image

00:12:02.090 --> 00:12:05.170 align:middle line:84%
that you get, and compare
it with the original image

00:12:05.170 --> 00:12:06.690 align:middle line:90%
that you trained it on.

00:12:06.690 --> 00:12:10.610 align:middle line:90%
And then, fine tune as you go.

00:12:10.610 --> 00:12:13.210 align:middle line:84%
PROFESSOR: OK, you're
on the right track.

00:12:13.210 --> 00:12:15.430 align:middle line:90%
Any other proposals?

00:12:15.430 --> 00:12:18.570 align:middle line:90%


00:12:18.570 --> 00:12:21.130 align:middle line:84%
STUDENT: I think we could try
to train the neural network

00:12:21.130 --> 00:12:25.930 align:middle line:84%
to reconstruct the image going
from the noisy one, that would

00:12:25.930 --> 00:12:30.730 align:middle line:84%
have all the set with images,
find their noise counterpart,

00:12:30.730 --> 00:12:34.810 align:middle line:84%
and train them to do the
opposite, the neural network

00:12:34.810 --> 00:12:38.012 align:middle line:90%
to do the opposite task.

00:12:38.012 --> 00:12:40.220 align:middle line:84%
PROFESSOR: Yup, that's
definitely on the right track.

00:12:40.220 --> 00:12:41.762 align:middle line:84%
That's definitely
on the right track.

00:12:41.762 --> 00:12:42.920 align:middle line:90%
Yeah, good ideas.

00:12:42.920 --> 00:12:47.480 align:middle line:84%
So what we do more
concretely, is

00:12:47.480 --> 00:12:50.480 align:middle line:84%
we can take each image
in the training data

00:12:50.480 --> 00:12:53.800 align:middle line:84%
and create noisy versions of
it, as we have seen before.

00:12:53.800 --> 00:12:57.040 align:middle line:84%
And then, what we
do is that we say

00:12:57.040 --> 00:13:01.200 align:middle line:84%
we can create x, y training
data pairs, input output

00:13:01.200 --> 00:13:03.920 align:middle line:90%
pairs, from all these images.

00:13:03.920 --> 00:13:07.920 align:middle line:84%
So specifically,
what we do is we

00:13:07.920 --> 00:13:12.360 align:middle line:84%
take the slightly noisy
version of Killian Court

00:13:12.360 --> 00:13:16.440 align:middle line:84%
and call it the input, and we
take the nice version of Killian

00:13:16.440 --> 00:13:19.160 align:middle line:90%
Court and call it the output.

00:13:19.160 --> 00:13:22.560 align:middle line:90%
That's the y1, x1 pair.

00:13:22.560 --> 00:13:28.720 align:middle line:84%
And then, we get y2, x2, we
get y3, x3 and all the way.

00:13:28.720 --> 00:13:33.452 align:middle line:84%
So at any point in time, the
relationship between x and y,

00:13:33.452 --> 00:13:35.917 align:middle line:84%
what's the relationship
between x and y?

00:13:35.917 --> 00:13:38.250 align:middle line:84%
If you set it up like this,
as the input and the output.

00:13:38.250 --> 00:13:43.110 align:middle line:90%


00:13:43.110 --> 00:13:45.630 align:middle line:84%
STUDENT: It's the set
of standard deviations

00:13:45.630 --> 00:13:49.470 align:middle line:84%
and the values which we
obtain for these pixels.

00:13:49.470 --> 00:13:53.070 align:middle line:84%
Those are, like, weights, which
correspond to another feature.

00:13:53.070 --> 00:13:54.750 align:middle line:84%
PROFESSOR: Right, or
maybe I was looking

00:13:54.750 --> 00:13:57.390 align:middle line:84%
for something simpler, which
was that-- that's correct.

00:13:57.390 --> 00:13:58.910 align:middle line:84%
What I was looking
for is really,

00:13:58.910 --> 00:14:00.550 align:middle line:84%
the relationship
between x and y.

00:14:00.550 --> 00:14:03.590 align:middle line:84%
X is an image, any
image, and y happens

00:14:03.590 --> 00:14:07.950 align:middle line:84%
to be a slightly less
noisy version of the image.

00:14:07.950 --> 00:14:12.350 align:middle line:84%
The slightly less noisy is
really, really important.

00:14:12.350 --> 00:14:14.610 align:middle line:84%
You're not going
from Killian Court,

00:14:14.610 --> 00:14:18.510 align:middle line:84%
you're not going from
the image to full noise.

00:14:18.510 --> 00:14:20.310 align:middle line:90%
That's an impossible leap.

00:14:20.310 --> 00:14:23.270 align:middle line:84%
You're going from the image
to a slightly noisy version

00:14:23.270 --> 00:14:24.670 align:middle line:90%
of that image.

00:14:24.670 --> 00:14:30.470 align:middle line:84%
It is that slightly that
allows all the magic to happen.

00:14:30.470 --> 00:14:33.020 align:middle line:90%
So that's what we have.

00:14:33.020 --> 00:14:35.600 align:middle line:84%
And so here, what we
can do with these x,

00:14:35.600 --> 00:14:37.860 align:middle line:90%
y pairs, when you have an--

00:14:37.860 --> 00:14:38.908 align:middle line:90%
so here's the thing.

00:14:38.908 --> 00:14:40.700 align:middle line:84%
This is like a larger
comment about machine

00:14:40.700 --> 00:14:41.825 align:middle line:90%
learning and deep learning.

00:14:41.825 --> 00:14:45.133 align:middle line:90%


00:14:45.133 --> 00:14:47.300 align:middle line:84%
Basically, what machine
learning, deep learning are,

00:14:47.300 --> 00:14:49.780 align:middle line:84%
are really, it's
like this black box

00:14:49.780 --> 00:14:53.540 align:middle line:84%
where if you can find
interesting input output pairs,

00:14:53.540 --> 00:14:56.400 align:middle line:84%
you can learn a function to go
from the input to the output.

00:14:56.400 --> 00:14:57.700 align:middle line:90%
That's it.

00:14:57.700 --> 00:15:01.120 align:middle line:84%
But this sounds kind of simple
when I describe it like that.

00:15:01.120 --> 00:15:04.100 align:middle line:84%
But there are some
incredibly non-obvious ways

00:15:04.100 --> 00:15:06.155 align:middle line:90%
of applying this idea.

00:15:06.155 --> 00:15:07.780 align:middle line:84%
So for example, a
few years ago, Google

00:15:07.780 --> 00:15:09.445 align:middle line:84%
had this thing,
which may actually

00:15:09.445 --> 00:15:10.820 align:middle line:84%
be in production
in Google Sheets

00:15:10.820 --> 00:15:15.440 align:middle line:84%
now, where whenever you
choose a bunch of numbers,

00:15:15.440 --> 00:15:17.340 align:middle line:84%
a range of numbers
in a spreadsheet

00:15:17.340 --> 00:15:19.460 align:middle line:84%
and then go into another
cell, it will immediately

00:15:19.460 --> 00:15:21.460 align:middle line:90%
suggest informative for you.

00:15:21.460 --> 00:15:24.020 align:middle line:90%
Where is that coming from?

00:15:24.020 --> 00:15:27.980 align:middle line:84%
It's because all the Google
Sheets users all over the world,

00:15:27.980 --> 00:15:31.060 align:middle line:84%
they have been creating all
these numbers with formulas.

00:15:31.060 --> 00:15:33.210 align:middle line:84%
So someone says,
look, wait a second.

00:15:33.210 --> 00:15:36.470 align:middle line:84%
We have all this data on people
choosing a range of numbers,

00:15:36.470 --> 00:15:38.210 align:middle line:90%
and then entering a formula.

00:15:38.210 --> 00:15:40.330 align:middle line:84%
So let's imagine the
range as the input

00:15:40.330 --> 00:15:43.130 align:middle line:90%
and the formula as the output.

00:15:43.130 --> 00:15:46.170 align:middle line:84%
And let's just give a
million examples of this pair

00:15:46.170 --> 00:15:47.850 align:middle line:84%
and see if anything
comes out of it.

00:15:47.850 --> 00:15:50.610 align:middle line:90%
And boom, you get that feature.

00:15:50.610 --> 00:15:55.368 align:middle line:84%
So similarly here, x is an
image, less noisy version

00:15:55.368 --> 00:15:55.910 align:middle line:90%
of the image.

00:15:55.910 --> 00:16:01.890 align:middle line:84%
What that means is that we
can build a denoising network.

00:16:01.890 --> 00:16:04.610 align:middle line:84%
We can take an image
and we can build

00:16:04.610 --> 00:16:08.430 align:middle line:84%
a network using all these x, y
pairs to slightly denoise it.

00:16:08.430 --> 00:16:12.426 align:middle line:90%


00:16:12.426 --> 00:16:15.110 align:middle line:90%
And so how do we do it?

00:16:15.110 --> 00:16:17.890 align:middle line:84%
We just run stochastic
gradient descent on the data.

00:16:17.890 --> 00:16:20.530 align:middle line:84%
We have a network,
which has x and y,

00:16:20.530 --> 00:16:23.350 align:middle line:84%
and then y is slightly
less noisy version.

00:16:23.350 --> 00:16:26.313 align:middle line:84%
And then, boom, boom,
boom, boom, boom.

00:16:26.313 --> 00:16:27.230 align:middle line:90%
You're just a network.

00:16:27.230 --> 00:16:28.313 align:middle line:90%
It has a bunch of weights.

00:16:28.313 --> 00:16:30.080 align:middle line:84%
We have the right
answer in terms

00:16:30.080 --> 00:16:32.482 align:middle line:90%
of what the images need to be.

00:16:32.482 --> 00:16:34.940 align:middle line:84%
We can do stochastic gradient
descent or Adam or something.

00:16:34.940 --> 00:16:36.660 align:middle line:84%
And before you know it,
if you have enough data,

00:16:36.660 --> 00:16:39.080 align:middle line:84%
you have a network which can
denoise anything you give it.

00:16:39.080 --> 00:16:40.160 align:middle line:90%
OK.

00:16:40.160 --> 00:16:41.960 align:middle line:90%
You had a question.

00:16:41.960 --> 00:16:43.320 align:middle line:90%
STUDENT: Why slightly?

00:16:43.320 --> 00:16:44.845 align:middle line:90%
PROFESSOR: Why slightly?

00:16:44.845 --> 00:16:46.220 align:middle line:84%
We'll come back
to that question.

00:16:46.220 --> 00:16:50.000 align:middle line:84%
The reason is that
in general, you

00:16:50.000 --> 00:16:52.560 align:middle line:84%
have to do what you
can to help the model.

00:16:52.560 --> 00:16:56.380 align:middle line:84%
And this is the proverbial,
there is an old adage,

00:16:56.380 --> 00:16:59.840 align:middle line:84%
you can't cross a
ditch in two jumps.

00:16:59.840 --> 00:17:02.460 align:middle line:84%
It's too big, so
you can't do it.

00:17:02.460 --> 00:17:05.240 align:middle line:84%
So what you do is you create a
bridge to go from here to there.

00:17:05.240 --> 00:17:08.560 align:middle line:84%
And so what you do is, if you
can slightly denoise something

00:17:08.560 --> 00:17:10.560 align:middle line:84%
really well, well,
I can actually

00:17:10.560 --> 00:17:12.359 align:middle line:84%
denoise anything you
want really well using

00:17:12.359 --> 00:17:17.055 align:middle line:84%
that fundamental capability,
as you will see in a second.

00:17:17.055 --> 00:17:18.180 align:middle line:90%
STUDENT: Just to follow up.

00:17:18.180 --> 00:17:19.819 align:middle line:84%
So if you go back
the last slide,

00:17:19.819 --> 00:17:23.960 align:middle line:84%
I could have created the
same thing as, that is my x1

00:17:23.960 --> 00:17:25.359 align:middle line:90%
and that is my y1.

00:17:25.359 --> 00:17:28.250 align:middle line:84%
Then, the second one is x2
and still, this is the y.

00:17:28.250 --> 00:17:30.550 align:middle line:84%
So there is effectively,
there is a learning there

00:17:30.550 --> 00:17:34.970 align:middle line:84%
that it could have taken from
those pairs and come back with,

00:17:34.970 --> 00:17:36.575 align:middle line:90%
OK, this is also a possibility.

00:17:36.575 --> 00:17:37.700 align:middle line:90%
This is also a possibility.

00:17:37.700 --> 00:17:38.408 align:middle line:90%
PROFESSOR: Right.

00:17:38.408 --> 00:17:40.790 align:middle line:84%
STUDENT: And found
out that noise matrix

00:17:40.790 --> 00:17:42.175 align:middle line:90%
that it can subtract.

00:17:42.175 --> 00:17:43.550 align:middle line:84%
PROFESSOR: Yeah,
so the thing is,

00:17:43.550 --> 00:17:46.670 align:middle line:84%
you want to make sure that each
time the amount of learning it

00:17:46.670 --> 00:17:50.350 align:middle line:84%
has to do is as bounded
and small as possible.

00:17:50.350 --> 00:17:52.710 align:middle line:84%
If you give it some starting
point and an ending point

00:17:52.710 --> 00:17:54.870 align:middle line:84%
and keep moving
this ending point,

00:17:54.870 --> 00:17:57.110 align:middle line:84%
the gap is still really
high for the first several

00:17:57.110 --> 00:17:59.190 align:middle line:90%
of those starting points.

00:17:59.190 --> 00:18:01.510 align:middle line:90%
That's the problem.

00:18:01.510 --> 00:18:02.256 align:middle line:90%
OK.

00:18:02.256 --> 00:18:03.890 align:middle line:90%
So to come back to this.

00:18:03.890 --> 00:18:05.950 align:middle line:84%
So we can build a
denoising model.

00:18:05.950 --> 00:18:07.170 align:middle line:90%
We can do this.

00:18:07.170 --> 00:18:09.470 align:middle line:84%
And now, once you have
built such a thing

00:18:09.470 --> 00:18:11.470 align:middle line:84%
you give it some noisy
thing, and then it

00:18:11.470 --> 00:18:15.430 align:middle line:84%
will give you a slightly
less noisy version of it.

00:18:15.430 --> 00:18:18.470 align:middle line:84%
The resolution is going to go
up slightly if you do that.

00:18:18.470 --> 00:18:20.230 align:middle line:84%
This, of course,
suggests the obvious way

00:18:20.230 --> 00:18:24.310 align:middle line:84%
in which you would use it,
which is that once you train it,

00:18:24.310 --> 00:18:27.552 align:middle line:90%
we can solve this problem.

00:18:27.552 --> 00:18:29.540 align:middle line:84%
And how can we
solve this problem?

00:18:29.540 --> 00:18:33.580 align:middle line:84%
So what you do is you
start with pure noise,

00:18:33.580 --> 00:18:37.060 align:middle line:90%
and then repeatedly denoise it.

00:18:37.060 --> 00:18:37.900 align:middle line:90%
OK.

00:18:37.900 --> 00:18:39.148 align:middle line:90%
You get that, you get that.

00:18:39.148 --> 00:18:40.940 align:middle line:84%
And then, before you
know it, Killian Court

00:18:40.940 --> 00:18:42.080 align:middle line:90%
has emerged from the fog.

00:18:42.080 --> 00:18:44.790 align:middle line:90%


00:18:44.790 --> 00:18:47.630 align:middle line:84%
It's pretty insane that it
actually works, this idea.

00:18:47.630 --> 00:18:53.780 align:middle line:90%


00:18:53.780 --> 00:18:56.320 align:middle line:84%
So the model will generate a
sequence of less noisy images,

00:18:56.320 --> 00:18:58.980 align:middle line:84%
and the final one you
have is the answer.

00:18:58.980 --> 00:19:01.960 align:middle line:84%
Now, there's a whole
bunch of detail here,

00:19:01.960 --> 00:19:05.860 align:middle line:84%
which I'm glossing over,
about, OK, how many times

00:19:05.860 --> 00:19:09.400 align:middle line:84%
must we run this loop to get
to a really good picture?

00:19:09.400 --> 00:19:12.540 align:middle line:84%
The short answer is initially,
it was like you have to run it

00:19:12.540 --> 00:19:14.480 align:middle line:90%
like 1,000 times.

00:19:14.480 --> 00:19:16.723 align:middle line:84%
Each denoising step
was like a baby step.

00:19:16.723 --> 00:19:18.140 align:middle line:84%
You have to do it
a thousand times

00:19:18.140 --> 00:19:19.540 align:middle line:90%
to get a really good answer.

00:19:19.540 --> 00:19:21.673 align:middle line:84%
Again, research has been
very active in the area,

00:19:21.673 --> 00:19:22.840 align:middle line:90%
continues to be very active.

00:19:22.840 --> 00:19:25.940 align:middle line:84%
Now, we can, I think, do it
with 50 steps or 100 steps.

00:19:25.940 --> 00:19:28.700 align:middle line:84%
But diffusion models,
like this, they

00:19:28.700 --> 00:19:32.153 align:middle line:84%
tend to take more time than
a large language model.

00:19:32.153 --> 00:19:33.820 align:middle line:84%
Which is why if you
give a prompt to one

00:19:33.820 --> 00:19:36.403 align:middle line:84%
of these models like Midjourney,
it will take some time for it

00:19:36.403 --> 00:19:37.900 align:middle line:90%
to come back with an image.

00:19:37.900 --> 00:19:40.180 align:middle line:84%
And the reason for the
delay is because it's

00:19:40.180 --> 00:19:43.700 align:middle line:84%
going through this
incremental denoising loop.

00:19:43.700 --> 00:19:45.365 align:middle line:90%
Yeah.

00:19:45.365 --> 00:19:46.740 align:middle line:84%
STUDENT: From
this, we understand

00:19:46.740 --> 00:19:49.500 align:middle line:84%
that each, the final
noise output sample

00:19:49.500 --> 00:19:52.460 align:middle line:84%
would be very particular to
each image in the beginning.

00:19:52.460 --> 00:19:56.380 align:middle line:84%
So, I mean, say if
you take two images,

00:19:56.380 --> 00:19:58.680 align:middle line:84%
finally we are
getting a start image.

00:19:58.680 --> 00:20:01.900 align:middle line:84%
And after, when we start
noising it, and the final output

00:20:01.900 --> 00:20:04.108 align:middle line:84%
we get with the noise sample,
it will be two distinct

00:20:04.108 --> 00:20:05.108 align:middle line:90%
for each of them, right?

00:20:05.108 --> 00:20:05.920 align:middle line:90%
PROFESSOR: Correct.

00:20:05.920 --> 00:20:08.100 align:middle line:84%
STUDENT: But when we
are picking up image

00:20:08.100 --> 00:20:11.620 align:middle line:84%
to generate a diffusion
model and we work backwards,

00:20:11.620 --> 00:20:14.040 align:middle line:84%
we may not have the exact
thing available to us,

00:20:14.040 --> 00:20:15.162 align:middle line:90%
what was there initially.

00:20:15.162 --> 00:20:16.620 align:middle line:84%
PROFESSOR: No, no,
the thing is, we

00:20:16.620 --> 00:20:18.580 align:middle line:84%
don't want to necessarily
regenerate images

00:20:18.580 --> 00:20:20.558 align:middle line:90%
that were in the training data.

00:20:20.558 --> 00:20:21.600 align:middle line:90%
That's kind of pointless.

00:20:21.600 --> 00:20:24.250 align:middle line:90%
We want to generate new images.

00:20:24.250 --> 00:20:29.490 align:middle line:84%
And for new images, we just
use noise as a starting point.

00:20:29.490 --> 00:20:31.370 align:middle line:84%
The fact that Killian
Court was here,

00:20:31.370 --> 00:20:33.490 align:middle line:84%
and then the fully noised
version of Killian Court

00:20:33.490 --> 00:20:35.055 align:middle line:84%
is here, that is
used for training.

00:20:35.055 --> 00:20:36.430 align:middle line:84%
And once you use
it for training,

00:20:36.430 --> 00:20:38.410 align:middle line:84%
you don't need it anymore
because you're not

00:20:38.410 --> 00:20:40.370 align:middle line:84%
trying to recreate
Killian Court again.

00:20:40.370 --> 00:20:42.450 align:middle line:84%
You want to create
new images which

00:20:42.450 --> 00:20:44.970 align:middle line:84%
belong to the category of
stately college buildings.

00:20:44.970 --> 00:20:47.823 align:middle line:84%
And for that, you just
grab noise, send it in.

00:20:47.823 --> 00:20:49.490 align:middle line:84%
It gives you a stately
college building.

00:20:49.490 --> 00:20:50.310 align:middle line:90%
End of story.

00:20:50.310 --> 00:20:53.730 align:middle line:90%


00:20:53.730 --> 00:20:56.850 align:middle line:84%
And because noise by definition
is different each time you

00:20:56.850 --> 00:20:59.850 align:middle line:84%
pick it, it's going to come up
with a different stately college

00:20:59.850 --> 00:21:01.730 align:middle line:90%
building.

00:21:01.730 --> 00:21:03.390 align:middle line:84%
So the way you think
about it is that--

00:21:03.390 --> 00:21:06.970 align:middle line:90%


00:21:06.970 --> 00:21:07.550 align:middle line:90%
all right.

00:21:07.550 --> 00:21:09.270 align:middle line:90%
So you can think of it as this.

00:21:09.270 --> 00:21:12.610 align:middle line:90%


00:21:12.610 --> 00:21:17.250 align:middle line:84%
So when you sample, think of
this as the noise distribution.

00:21:17.250 --> 00:21:22.560 align:middle line:84%
Each time you sample, there is a
little point you pick from here.

00:21:22.560 --> 00:21:25.680 align:middle line:84%
Another time you sample,
maybe you get a point here.

00:21:25.680 --> 00:21:27.660 align:middle line:84%
Each is just a
nice distribution.

00:21:27.660 --> 00:21:28.360 align:middle line:90%
That's it.

00:21:28.360 --> 00:21:30.520 align:middle line:84%
What actually these
things are doing

00:21:30.520 --> 00:21:35.840 align:middle line:84%
is they are mapping it to the
distribution of stately college

00:21:35.840 --> 00:21:41.120 align:middle line:84%
buildings, which might be in
a strange, crazy distribution.

00:21:41.120 --> 00:21:43.720 align:middle line:84%
So each time you sample,
you just go from here,

00:21:43.720 --> 00:21:47.680 align:middle line:90%
and you land at a point here.

00:21:47.680 --> 00:21:50.550 align:middle line:84%
And when you go from here,
you land at a point there.

00:21:50.550 --> 00:21:53.740 align:middle line:90%


00:21:53.740 --> 00:21:56.120 align:middle line:84%
So what you have done is when
you take the training data,

00:21:56.120 --> 00:21:59.560 align:middle line:84%
you basically created
points here, and then

00:21:59.560 --> 00:22:02.080 align:middle line:84%
found the matching noise
here, and then flipped it

00:22:02.080 --> 00:22:03.768 align:middle line:84%
for training, as we
have seen before.

00:22:03.768 --> 00:22:05.560 align:middle line:84%
And once you're done
with it, you basically

00:22:05.560 --> 00:22:09.360 align:middle line:84%
have a mechanism for
transforming any entry

00:22:09.360 --> 00:22:12.480 align:middle line:84%
in this distribution
of images to an entry

00:22:12.480 --> 00:22:14.340 align:middle line:90%
in this distribution of images.

00:22:14.340 --> 00:22:17.140 align:middle line:84%
So it's a way to
transform one distribution

00:22:17.140 --> 00:22:18.140 align:middle line:90%
to another distribution.

00:22:18.140 --> 00:22:20.430 align:middle line:90%
That's what's going on.

00:22:20.430 --> 00:22:21.910 align:middle line:90%
All right.

00:22:21.910 --> 00:22:26.310 align:middle line:84%
So there was a question,
yeah, and then we'll go.

00:22:26.310 --> 00:22:29.810 align:middle line:84%
STUDENT: I understand going from
noise to image and back, how you

00:22:29.810 --> 00:22:31.330 align:middle line:90%
call how the training works.

00:22:31.330 --> 00:22:35.210 align:middle line:84%
So my question is, in some
of these models today,

00:22:35.210 --> 00:22:37.470 align:middle line:84%
you have, when you
give it the noise

00:22:37.470 --> 00:22:40.970 align:middle line:84%
now to generate an
image, for example,

00:22:40.970 --> 00:22:45.210 align:middle line:84%
it could generate a human with
four fingers or stuff like that.

00:22:45.210 --> 00:22:49.270 align:middle line:84%
So is it that the model,
that the training data is not

00:22:49.270 --> 00:22:52.350 align:middle line:84%
just quite enough to
or as robust enough

00:22:52.350 --> 00:22:55.850 align:middle line:90%
to generate that kind of detail?

00:22:55.850 --> 00:22:58.203 align:middle line:84%
Can you talk through
what's going on with that?

00:22:58.203 --> 00:22:58.870 align:middle line:90%
PROFESSOR: Yeah.

00:22:58.870 --> 00:23:00.990 align:middle line:84%
So fundamentally,
what it is doing

00:23:00.990 --> 00:23:04.390 align:middle line:84%
is it actually does not
understand the notion of fingers

00:23:04.390 --> 00:23:08.630 align:middle line:84%
and things like that, because
we haven't injected any domain

00:23:08.630 --> 00:23:11.110 align:middle line:84%
knowledge into
this whole process,

00:23:11.110 --> 00:23:15.310 align:middle line:84%
by saying that, hey, you need
to generate a human body,

00:23:15.310 --> 00:23:17.760 align:middle line:84%
and here are the semantics
of what the human body is.

00:23:17.760 --> 00:23:21.097 align:middle line:84%
It's got five fingers and
all the anatomical stuff.

00:23:21.097 --> 00:23:22.180 align:middle line:90%
We're not giving anything.

00:23:22.180 --> 00:23:25.360 align:middle line:84%
We're literally giving it pixel
values, a bunch of pictures.

00:23:25.360 --> 00:23:27.460 align:middle line:84%
So everything you're
seeing is basically just

00:23:27.460 --> 00:23:29.820 align:middle line:84%
coming out of that very blind
statistical transformation

00:23:29.820 --> 00:23:32.100 align:middle line:90%
process.

00:23:32.100 --> 00:23:35.180 align:middle line:84%
So you would expect that macro
level details would probably

00:23:35.180 --> 00:23:38.460 align:middle line:84%
get it right because there
are so many right answers.

00:23:38.460 --> 00:23:42.860 align:middle line:84%
So imagine it's actually it's
creating the roof of a house.

00:23:42.860 --> 00:23:44.747 align:middle line:84%
There could be all
kinds of variations

00:23:44.747 --> 00:23:46.580 align:middle line:84%
in the roof of the house
and you would still

00:23:46.580 --> 00:23:48.900 align:middle line:84%
think it's a roof of a
house because there are

00:23:48.900 --> 00:23:50.360 align:middle line:90%
many possible right answers.

00:23:50.360 --> 00:23:51.800 align:middle line:84%
But when it comes
to five fingers,

00:23:51.800 --> 00:23:53.160 align:middle line:84%
there aren't many
possible right answers,

00:23:53.160 --> 00:23:55.260 align:middle line:84%
which is why you notice
the error very quickly.

00:23:55.260 --> 00:23:57.343 align:middle line:84%
As far as the model is
concerned, it doesn't know.

00:23:57.343 --> 00:24:00.460 align:middle line:84%
It's just producing a
statistically plausible sample

00:24:00.460 --> 00:24:01.860 align:middle line:90%
from that distribution.

00:24:01.860 --> 00:24:04.932 align:middle line:84%
And since we haven't forced
it to obey constraints

00:24:04.932 --> 00:24:06.640 align:middle line:84%
like five fingers and
so on and so forth,

00:24:06.640 --> 00:24:07.840 align:middle line:84%
it's not going to do
any of that stuff.

00:24:07.840 --> 00:24:09.240 align:middle line:90%
It's an unconstrained process.

00:24:09.240 --> 00:24:12.360 align:middle line:84%
Now, over time, these things
have gotten better and better,

00:24:12.360 --> 00:24:15.310 align:middle line:84%
and that's because the data has
gotten better, to your point.

00:24:15.310 --> 00:24:17.810 align:middle line:84%
But I think our approach to
doing these things is also

00:24:17.810 --> 00:24:19.250 align:middle line:90%
getting better.

00:24:19.250 --> 00:24:22.070 align:middle line:84%
There are lots of ways to
now steer it and control it

00:24:22.070 --> 00:24:24.130 align:middle line:84%
so it behaves the right
way, and that is actually

00:24:24.130 --> 00:24:25.550 align:middle line:84%
part of what's
happening as well.

00:24:25.550 --> 00:24:28.450 align:middle line:84%
So when we talk about, how do
you actually give a text prompt

00:24:28.450 --> 00:24:31.270 align:middle line:84%
and have it build the image
for that particular prompt?

00:24:31.270 --> 00:24:34.190 align:middle line:90%
We will revisit this question.

00:24:34.190 --> 00:24:34.690 align:middle line:90%
OK.

00:24:34.690 --> 00:24:36.110 align:middle line:90%
There were more questions.

00:24:36.110 --> 00:24:38.370 align:middle line:90%
Yeah.

00:24:38.370 --> 00:24:41.030 align:middle line:84%
STUDENT: Is there some
randomness in the model itself?

00:24:41.030 --> 00:24:43.990 align:middle line:84%
So if you gave it the
same noise image twice,

00:24:43.990 --> 00:24:46.233 align:middle line:84%
will it actually produce
the same final image or will

00:24:46.233 --> 00:24:47.650 align:middle line:84%
it come up with
something similar?

00:24:47.650 --> 00:24:49.410 align:middle line:84%
PROFESSOR: Yeah, there is
randomness, in the process

00:24:49.410 --> 00:24:49.950 align:middle line:90%
as well.

00:24:49.950 --> 00:24:50.370 align:middle line:90%
STUDENT: In the process as well?

00:24:50.370 --> 00:24:50.990 align:middle line:90%
PROFESSOR: In the process.

00:24:50.990 --> 00:24:51.270 align:middle line:90%
STUDENT: Yeah.

00:24:51.270 --> 00:24:52.062 align:middle line:90%
PROFESSOR: Exactly.

00:24:52.062 --> 00:24:54.770 align:middle line:90%


00:24:54.770 --> 00:24:57.930 align:middle line:84%
So to actually, that's
a really good point.

00:24:57.930 --> 00:25:00.510 align:middle line:84%
But now, I'm afraid to
open my laptop or my iPad.

00:25:00.510 --> 00:25:02.890 align:middle line:90%
One second.

00:25:02.890 --> 00:25:04.610 align:middle line:90%
All right.

00:25:04.610 --> 00:25:10.810 align:middle line:84%
So what's going on here is
that if you go to this thing,

00:25:10.810 --> 00:25:13.840 align:middle line:84%
so I talked about
we are transforming

00:25:13.840 --> 00:25:17.240 align:middle line:84%
from here, to some
crazy distribution here.

00:25:17.240 --> 00:25:19.760 align:middle line:84%
So what happens is that let's
say that this is the starting

00:25:19.760 --> 00:25:21.800 align:middle line:90%
point for the noise input.

00:25:21.800 --> 00:25:23.320 align:middle line:90%
This is your noise input.

00:25:23.320 --> 00:25:28.080 align:middle line:84%
And then, what you
actually do is you go here,

00:25:28.080 --> 00:25:29.620 align:middle line:90%
and then you take this point.

00:25:29.620 --> 00:25:32.760 align:middle line:84%
And then, you do a
small sample next to it.

00:25:32.760 --> 00:25:36.780 align:middle line:84%
So you use this as the mean
value and then sample around it,

00:25:36.780 --> 00:25:39.780 align:middle line:84%
and that's actually what gets
published in the user interface.

00:25:39.780 --> 00:25:43.260 align:middle line:84%
That's where the
randomness comes in.

00:25:43.260 --> 00:25:43.760 align:middle line:90%
OK.

00:25:43.760 --> 00:25:48.360 align:middle line:90%


00:25:48.360 --> 00:25:49.220 align:middle line:90%
So back to this.

00:25:49.220 --> 00:25:52.360 align:middle line:84%
Was there another
question somewhere?

00:25:52.360 --> 00:25:53.740 align:middle line:90%
Yeah.

00:25:53.740 --> 00:25:54.720 align:middle line:90%
STUDENT: Not sure.

00:25:54.720 --> 00:25:56.600 align:middle line:90%
PROFESSOR: It's OK.

00:25:56.600 --> 00:26:02.440 align:middle line:84%
STUDENT: I was wondering about,
when training on a clear picture

00:26:02.440 --> 00:26:05.680 align:middle line:84%
to go to a noisy
image, to display

00:26:05.680 --> 00:26:08.640 align:middle line:84%
where a random sample with
a random drop, the sample

00:26:08.640 --> 00:26:10.280 align:middle line:90%
probably is pseudorandom.

00:26:10.280 --> 00:26:13.230 align:middle line:84%
I'm just wondering if it's like
learning relationships that are

00:26:13.230 --> 00:26:14.810 align:middle line:90%
dependent on pseudorandomness.

00:26:14.810 --> 00:26:20.285 align:middle line:84%
And so when it goes from a noisy
image back to a pure image,

00:26:20.285 --> 00:26:22.410 align:middle line:84%
is it dependent on that,
or if that matters at all?

00:26:22.410 --> 00:26:23.070 align:middle line:90%
PROFESSOR: Oh, I see.

00:26:23.070 --> 00:26:24.430 align:middle line:84%
So if I understand your
question, what you're saying

00:26:24.430 --> 00:26:26.388 align:middle line:84%
is that it's pseudorandom,
not actually random.

00:26:26.388 --> 00:26:26.973 align:middle line:90%
STUDENT: Yeah.

00:26:26.973 --> 00:26:28.390 align:middle line:84%
PROFESSOR: And so
therefore, there

00:26:28.390 --> 00:26:31.225 align:middle line:84%
is some signal in the
supposedly random generation.

00:26:31.225 --> 00:26:31.850 align:middle line:90%
STUDENT: Right.

00:26:31.850 --> 00:26:33.267 align:middle line:84%
PROFESSOR: Is it
actually glomming

00:26:33.267 --> 00:26:35.250 align:middle line:84%
onto that signal,
right, is the question?

00:26:35.250 --> 00:26:35.990 align:middle line:90%
STUDENT: Yeah.

00:26:35.990 --> 00:26:38.070 align:middle line:84%
PROFESSOR: Theoretically,
it's probably possible,

00:26:38.070 --> 00:26:39.430 align:middle line:84%
but in practice, it
really doesn't matter.

00:26:39.430 --> 00:26:41.190 align:middle line:84%
Because we basically
say pseudorandom

00:26:41.190 --> 00:26:43.330 align:middle line:90%
is good enough for our purposes.

00:26:43.330 --> 00:26:45.770 align:middle line:84%
And in fact, in practice, you
will see it's not an issue.

00:26:45.770 --> 00:26:50.110 align:middle line:90%


00:26:50.110 --> 00:26:51.408 align:middle line:90%
Oh yeah, go ahead.

00:26:51.408 --> 00:26:52.950 align:middle line:84%
STUDENT: I just had
a quick question.

00:26:52.950 --> 00:26:57.470 align:middle line:84%
When you're doing
text-to-text, let's

00:26:57.470 --> 00:27:00.330 align:middle line:90%
say you're tokenizing the input.

00:27:00.330 --> 00:27:03.910 align:middle line:84%
But here, you somehow have to
identify that this is Killian

00:27:03.910 --> 00:27:06.670 align:middle line:90%
Court and like a stately home.

00:27:06.670 --> 00:27:09.580 align:middle line:84%
And this is just going
from pixel to image

00:27:09.580 --> 00:27:13.020 align:middle line:90%
or decoding a pixel image.

00:27:13.020 --> 00:27:16.340 align:middle line:84%
Where does the tag
or tokenization

00:27:16.340 --> 00:27:19.695 align:middle line:84%
of columns or
fingernails, or like--

00:27:19.695 --> 00:27:20.820 align:middle line:90%
PROFESSOR: There's nothing.

00:27:20.820 --> 00:27:22.810 align:middle line:84%
It's learning everything
from the pixel values.

00:27:22.810 --> 00:27:23.860 align:middle line:84%
STUDENT: Everything's
from the pixel values?

00:27:23.860 --> 00:27:25.540 align:middle line:84%
PROFESSOR: Yeah, and
this is what I was,

00:27:25.540 --> 00:27:27.820 align:middle line:84%
when Ike asked the
question about the four

00:27:27.820 --> 00:27:31.520 align:middle line:84%
fingers, five fingers thing,
it has no idea of fingers.

00:27:31.520 --> 00:27:33.520 align:middle line:84%
It has zero knowledge
about any of these things.

00:27:33.520 --> 00:27:36.300 align:middle line:84%
All it's seeing is a
bunch of photographs.

00:27:36.300 --> 00:27:38.380 align:middle line:84%
STUDENT: So when you
type in, you say, I

00:27:38.380 --> 00:27:40.365 align:middle line:84%
want to see a hand
with green fingers--

00:27:40.365 --> 00:27:41.240 align:middle line:90%
PROFESSOR: Oh, I see.

00:27:41.240 --> 00:27:43.800 align:middle line:84%
So we haven't yet come
to the stage of, OK,

00:27:43.800 --> 00:27:46.840 align:middle line:84%
how do you actually steer this
image using your text prompt?

00:27:46.840 --> 00:27:47.520 align:middle line:90%
It's coming.

00:27:47.520 --> 00:27:48.020 align:middle line:90%
STUDENT: OK.

00:27:48.020 --> 00:27:49.520 align:middle line:84%
PROFESSOR: Right now, all
we are saying is that,

00:27:49.520 --> 00:27:51.980 align:middle line:84%
look, I'm going to give
you a bunch of photographs

00:27:51.980 --> 00:27:54.587 align:middle line:84%
of a particular kind of thing,
stately college buildings.

00:27:54.587 --> 00:27:56.920 align:middle line:84%
And I want to have a model,
which at the end of the day,

00:27:56.920 --> 00:27:57.800 align:middle line:90%
I just poke it.

00:27:57.800 --> 00:27:59.967 align:middle line:84%
Every time I poke it, it
gives me a stately college.

00:27:59.967 --> 00:28:00.500 align:middle line:90%
That's it.

00:28:00.500 --> 00:28:03.787 align:middle line:84%
Now, I'm going to actually start
giving it text and saying, OK,

00:28:03.787 --> 00:28:05.620 align:middle line:84%
create the thing I'm
just telling you about.

00:28:05.620 --> 00:28:06.440 align:middle line:90%
That's coming.

00:28:06.440 --> 00:28:11.030 align:middle line:84%
And that's some additional magic
is going on to get that done.

00:28:11.030 --> 00:28:11.530 align:middle line:90%
OK.

00:28:11.530 --> 00:28:14.090 align:middle line:90%
So this is what we have.

00:28:14.090 --> 00:28:16.390 align:middle line:84%
And this is called
a diffusion model.

00:28:16.390 --> 00:28:19.210 align:middle line:84%
And this is the original
paper that figured this out.

00:28:19.210 --> 00:28:24.530 align:middle line:84%
And the process of
actually creating--

00:28:24.530 --> 00:28:27.490 align:middle line:84%
if taking an image and
creating noisy versions of it

00:28:27.490 --> 00:28:30.250 align:middle line:84%
to create a training data is
called the forward process,

00:28:30.250 --> 00:28:33.810 align:middle line:84%
and then what we did in reverse
is called the reverse process.

00:28:33.810 --> 00:28:34.910 align:middle line:90%
Check out the paper.

00:28:34.910 --> 00:28:37.950 align:middle line:84%
It's actually really well
written and I recommend it.

00:28:37.950 --> 00:28:40.770 align:middle line:84%
Now, in practice,
some other researchers

00:28:40.770 --> 00:28:44.195 align:middle line:84%
came along shortly after this
and made a small improvement.

00:28:44.195 --> 00:28:46.570 align:middle line:84%
It turns out to be actually
a big improvement in practice

00:28:46.570 --> 00:28:48.362 align:middle line:84%
in terms of improving
the quality of what's

00:28:48.362 --> 00:28:49.270 align:middle line:90%
being produced.

00:28:49.270 --> 00:28:52.250 align:middle line:84%
And so what they said is, hey,
instead of training the model

00:28:52.250 --> 00:28:54.810 align:middle line:84%
to predict the less noisy
version of the image,

00:28:54.810 --> 00:29:00.090 align:middle line:84%
we actually ask it to predict
just the noise in the input.

00:29:00.090 --> 00:29:02.450 align:middle line:84%
And then, we will just
simply subtract the noise

00:29:02.450 --> 00:29:04.410 align:middle line:90%
from the input to get the image.

00:29:04.410 --> 00:29:08.917 align:middle line:84%
So instead of saying x is an
image, y is the noisy image,

00:29:08.917 --> 00:29:10.500 align:middle line:84%
we actually tell it,
here's the image.

00:29:10.500 --> 00:29:14.120 align:middle line:84%
Here is the noise that we added
to x to get the noisy version.

00:29:14.120 --> 00:29:15.940 align:middle line:84%
And then, just predict
the noise for me.

00:29:15.940 --> 00:29:17.960 align:middle line:84%
And then, once I get it,
I just do x minus noise

00:29:17.960 --> 00:29:21.240 align:middle line:84%
and I get the less noisy
version of the image.

00:29:21.240 --> 00:29:23.920 align:middle line:84%
This feels arithmetically
equivalent,

00:29:23.920 --> 00:29:26.720 align:middle line:84%
but in practice, it ends up
generating much higher quality

00:29:26.720 --> 00:29:27.540 align:middle line:90%
images.

00:29:27.540 --> 00:29:28.680 align:middle line:84%
And there is some very
interesting theory

00:29:28.680 --> 00:29:30.520 align:middle line:84%
as to why that works
and so on and so forth.

00:29:30.520 --> 00:29:33.200 align:middle line:84%
And you can read this
paper if you're interested.

00:29:33.200 --> 00:29:34.680 align:middle line:84%
So if you actually
look at what's

00:29:34.680 --> 00:29:36.960 align:middle line:84%
going on in most
diffusion models today,

00:29:36.960 --> 00:29:39.080 align:middle line:84%
they're basically using
an approach like this.

00:29:39.080 --> 00:29:41.788 align:middle line:84%
They're actually predicting each
time they predict noise and take

00:29:41.788 --> 00:29:43.040 align:middle line:90%
it away, subtract it.

00:29:43.040 --> 00:29:47.040 align:middle line:84%
So iterative subtracting
of predicted noise.

00:29:47.040 --> 00:29:48.720 align:middle line:90%
That's what's going on.

00:29:48.720 --> 00:29:51.560 align:middle line:84%
So all right,
that's what we have.

00:29:51.560 --> 00:29:55.420 align:middle line:84%
Now, at this point, you
might be wondering, OK,

00:29:55.420 --> 00:29:57.440 align:middle line:84%
so far in the semester,
we have actually

00:29:57.440 --> 00:29:59.680 align:middle line:84%
learned how to take
an image and then

00:29:59.680 --> 00:30:02.680 align:middle line:84%
classify it into one of 20
things, 10 things whatever.

00:30:02.680 --> 00:30:05.070 align:middle line:84%
We've also taken text
and figure out what

00:30:05.070 --> 00:30:06.250 align:middle line:90%
to do with things with it.

00:30:06.250 --> 00:30:09.390 align:middle line:84%
We haven't yet talked about how
do you actually take an image,

00:30:09.390 --> 00:30:12.950 align:middle line:84%
and how can we get the output
also to be another image?

00:30:12.950 --> 00:30:14.910 align:middle line:90%
We haven't done that yet.

00:30:14.910 --> 00:30:18.070 align:middle line:84%
So we have actually not
done image-to-image.

00:30:18.070 --> 00:30:21.070 align:middle line:84%
How do you actually build a
neural network to do images?

00:30:21.070 --> 00:30:22.968 align:middle line:84%
And in the interest
of time, you're not

00:30:22.968 --> 00:30:24.510 align:middle line:84%
going to get into
it massively, but I

00:30:24.510 --> 00:30:27.310 align:middle line:84%
want to give you a quick
idea of how it works.

00:30:27.310 --> 00:30:33.430 align:middle line:84%
So the most, I would say,
the dominant architecture

00:30:33.430 --> 00:30:36.510 align:middle line:84%
to take an image as an input and
produce an image as an output

00:30:36.510 --> 00:30:38.070 align:middle line:90%
is called the U-Net.

00:30:38.070 --> 00:30:40.300 align:middle line:84%
And that's architecture
we see here.

00:30:40.300 --> 00:30:42.910 align:middle line:90%


00:30:42.910 --> 00:30:45.660 align:middle line:84%
So fundamentally, if you
look at the left half.

00:30:45.660 --> 00:30:47.910 align:middle line:84%
So there's a left half to
the network and a right half

00:30:47.910 --> 00:30:49.950 align:middle line:84%
to the network,
hence, the U. If you

00:30:49.950 --> 00:30:51.910 align:middle line:84%
look at the left
half of the network,

00:30:51.910 --> 00:30:54.790 align:middle line:84%
it's a good, old
convolutional neural network,

00:30:54.790 --> 00:30:58.670 align:middle line:84%
like the kind we know
and love and the kind

00:30:58.670 --> 00:31:00.210 align:middle line:90%
that we are very familiar with.

00:31:00.210 --> 00:31:02.340 align:middle line:84%
So you take an input
image, and then you

00:31:02.340 --> 00:31:06.920 align:middle line:84%
run it through a bunch
of convolutional blocks,

00:31:06.920 --> 00:31:10.560 align:middle line:84%
and then we do some max pooling,
and then we keep on doing it.

00:31:10.560 --> 00:31:12.780 align:middle line:84%
And at some point, it
becomes smaller and smaller,

00:31:12.780 --> 00:31:16.280 align:middle line:84%
and we get something like this,
which we are very familiar with.

00:31:16.280 --> 00:31:18.900 align:middle line:84%
The big image with
the three channels

00:31:18.900 --> 00:31:20.480 align:middle line:84%
gets smaller and
smaller and smaller,

00:31:20.480 --> 00:31:22.480 align:middle line:84%
but the number of channels
gets wider and wider.

00:31:22.480 --> 00:31:24.800 align:middle line:84%
It becomes much smaller
but much deeper.

00:31:24.800 --> 00:31:27.020 align:middle line:90%
It becomes like a 3D volume.

00:31:27.020 --> 00:31:29.140 align:middle line:84%
And we have seen
that again and again.

00:31:29.140 --> 00:31:31.940 align:middle line:84%
The left part is just a
good, old convolutional

00:31:31.940 --> 00:31:33.120 align:middle line:90%
with pooling layers.

00:31:33.120 --> 00:31:34.660 align:middle line:84%
And then, you come
to the middle.

00:31:34.660 --> 00:31:37.060 align:middle line:84%
And then, from this
point on, what we do is

00:31:37.060 --> 00:31:39.920 align:middle line:84%
we take whatever
this thing here,

00:31:39.920 --> 00:31:42.580 align:middle line:84%
and then we essentially
reverse the process.

00:31:42.580 --> 00:31:45.260 align:middle line:84%
We go from the small things,
which are really deep,

00:31:45.260 --> 00:31:48.620 align:middle line:84%
to slightly bigger things that
are a little less deep, and so

00:31:48.620 --> 00:31:51.580 align:middle line:84%
on and so forth, till we get
the original size back again.

00:31:51.580 --> 00:31:56.940 align:middle line:84%
And we do that using an inverse
of the convolution layer

00:31:56.940 --> 00:32:00.060 align:middle line:84%
called an up convolution
or deconvolution layer.

00:32:00.060 --> 00:32:03.380 align:middle line:84%
And you can check out
9.2 in the textbook

00:32:03.380 --> 00:32:05.760 align:middle line:90%
to understand how it's done.

00:32:05.760 --> 00:32:09.220 align:middle line:84%
It's also called
conv 2D transpose.

00:32:09.220 --> 00:32:11.602 align:middle line:90%
It's a very similar idea.

00:32:11.602 --> 00:32:13.560 align:middle line:84%
And I'm not going to get
into the details here,

00:32:13.560 --> 00:32:15.685 align:middle line:84%
but you essentially do the
inverse of a convolution

00:32:15.685 --> 00:32:18.700 align:middle line:84%
operation to get the size to
come back to the bigger size.

00:32:18.700 --> 00:32:21.980 align:middle line:84%
And you do it gradually,
till the output

00:32:21.980 --> 00:32:25.780 align:middle line:84%
you have matches the size
of the input that came in.

00:32:25.780 --> 00:32:28.760 align:middle line:84%
So image gets smaller
and smaller into a thing,

00:32:28.760 --> 00:32:31.320 align:middle line:84%
and then you just blow it back
up again to get an image back.

00:32:31.320 --> 00:32:33.100 align:middle line:90%
So that is the U-Net.

00:32:33.100 --> 00:32:35.420 align:middle line:84%
Now, there's really one
very important thing

00:32:35.420 --> 00:32:42.140 align:middle line:84%
that happens in the U-Net, which
is you see these connections.

00:32:42.140 --> 00:32:45.820 align:middle line:84%
Basically, what they do is
at every step, when you're

00:32:45.820 --> 00:32:47.900 align:middle line:84%
coming back up in
the right half,

00:32:47.900 --> 00:32:51.860 align:middle line:84%
you actually attach
whatever was in the mirror

00:32:51.860 --> 00:32:55.320 align:middle line:84%
image of the original input as
we processed on the left side,

00:32:55.320 --> 00:32:57.660 align:middle line:84%
we attach it to
this side as well.

00:32:57.660 --> 00:33:00.170 align:middle line:84%
Remember, I talked about this
whole notion of a residual

00:33:00.170 --> 00:33:04.410 align:middle line:84%
connection, back many
classes ago, where I said,

00:33:04.410 --> 00:33:08.690 align:middle line:84%
when an input goes through
each layer of a neural network,

00:33:08.690 --> 00:33:10.890 align:middle line:84%
at one point, let's say
you're in the 10th layer,

00:33:10.890 --> 00:33:13.450 align:middle line:84%
you're only seeing what is
the ninth layer has produced

00:33:13.450 --> 00:33:13.950 align:middle line:90%
for you.

00:33:13.950 --> 00:33:15.430 align:middle line:90%
That's all you're working with.

00:33:15.430 --> 00:33:18.530 align:middle line:84%
But wouldn't it be nice if the
10th layer actually had access

00:33:18.530 --> 00:33:21.363 align:middle line:84%
to the eighth layer, the
seventh layer, the sixth layer,

00:33:21.363 --> 00:33:22.030 align:middle line:90%
the fifth layer?

00:33:22.030 --> 00:33:23.090 align:middle line:90%
Heck, why not the input?

00:33:23.090 --> 00:33:26.490 align:middle line:84%
Because the more information
it has, the more able

00:33:26.490 --> 00:33:29.450 align:middle line:84%
it's probably to do whatever
it can with the inputs given.

00:33:29.450 --> 00:33:32.490 align:middle line:84%
Why restrict it to only
the input of the output

00:33:32.490 --> 00:33:33.430 align:middle line:90%
of the previous layer?

00:33:33.430 --> 00:33:35.763 align:middle line:84%
Why can't we give it everything
that has came before it?

00:33:35.763 --> 00:33:37.390 align:middle line:84%
Now, giving everything
is too much,

00:33:37.390 --> 00:33:39.950 align:middle line:84%
but we can be selective
in what we give it.

00:33:39.950 --> 00:33:41.670 align:middle line:84%
So what these folks
decided, I'm sure,

00:33:41.670 --> 00:33:44.530 align:middle line:84%
after much experimentation,
is that if they actually

00:33:44.530 --> 00:33:49.290 align:middle line:84%
attach whatever was coming out
of this layer to this layer

00:33:49.290 --> 00:33:51.950 align:middle line:84%
before it goes through the
output, it really helped.

00:33:51.950 --> 00:33:53.750 align:middle line:84%
Similarly, this
thing gets attached,

00:33:53.750 --> 00:33:54.930 align:middle line:90%
and so on and so forth.

00:33:54.930 --> 00:33:56.960 align:middle line:90%
And it kind of makes sense.

00:33:56.960 --> 00:33:59.560 align:middle line:84%
Why force it to
figure out everything

00:33:59.560 --> 00:34:02.320 align:middle line:84%
it has to figure out just
from this thing that came in?

00:34:02.320 --> 00:34:04.240 align:middle line:90%
Let's give this that.

00:34:04.240 --> 00:34:06.860 align:middle line:84%
Let's also give a little
here, a little here.

00:34:06.860 --> 00:34:09.400 align:middle line:84%
So these residual connections
are a huge building block

00:34:09.400 --> 00:34:11.719 align:middle line:84%
for why these things
work as well as they do.

00:34:11.719 --> 00:34:15.080 align:middle line:84%
And in general, giving a
layer as much information

00:34:15.080 --> 00:34:17.420 align:middle line:84%
as we can give it is
always a good idea,

00:34:17.420 --> 00:34:19.159 align:middle line:84%
but you can't go
nuts because then you

00:34:19.159 --> 00:34:21.179 align:middle line:84%
have much more parameters and
all kinds of stuff happens.

00:34:21.179 --> 00:34:23.262 align:middle line:84%
So there is a bit of a
balance you have to strike,

00:34:23.262 --> 00:34:25.679 align:middle line:84%
and this was the balance
struck by these researchers.

00:34:25.679 --> 00:34:28.520 align:middle line:84%
And so this thing was
originally invented

00:34:28.520 --> 00:34:31.300 align:middle line:84%
for some medical
segmentation use cases,

00:34:31.300 --> 00:34:34.239 align:middle line:84%
but it is just heavily
used for everything now.

00:34:34.239 --> 00:34:36.840 align:middle line:84%
It's a really
powerful architecture.

00:34:36.840 --> 00:34:39.760 align:middle line:90%
Questions?

00:34:39.760 --> 00:34:42.520 align:middle line:84%
STUDENT: Can we have an
example of in what scenarios

00:34:42.520 --> 00:34:44.199 align:middle line:84%
would you use to
spin up a network?

00:34:44.199 --> 00:34:46.560 align:middle line:84%
PROFESSOR: Anytime you
have an image-to-image.

00:34:46.560 --> 00:34:49.687 align:middle line:84%
STUDENT: What kind of conversion
do you have [INAUDIBLE]

00:34:49.687 --> 00:34:52.020 align:middle line:84%
PROFESSOR: Oh, like, what
kind of examples of use cases?

00:34:52.020 --> 00:34:53.159 align:middle line:84%
Let's say that, for
example, you want

00:34:53.159 --> 00:34:55.429 align:middle line:84%
to take an image, like
a black and white image,

00:34:55.429 --> 00:34:58.730 align:middle line:84%
and you want to colorize
it, for instance.

00:34:58.730 --> 00:35:00.055 align:middle line:90%
Boom, you U-Net.

00:35:00.055 --> 00:35:02.430 align:middle line:84%
You want to take an image and
make it a higher resolution

00:35:02.430 --> 00:35:03.630 align:middle line:90%
image, U-Net.

00:35:03.630 --> 00:35:06.950 align:middle line:84%
You want to take an image and
for every pixel in the image,

00:35:06.950 --> 00:35:10.950 align:middle line:84%
you want to classify it
into one of 10 things.

00:35:10.950 --> 00:35:13.150 align:middle line:84%
So anytime when
you want the output

00:35:13.150 --> 00:35:16.030 align:middle line:84%
shape, the shape of the
output, to be basically

00:35:16.030 --> 00:35:18.810 align:middle line:84%
the same shape as the
input but with other data,

00:35:18.810 --> 00:35:20.770 align:middle line:90%
you need to use this.

00:35:20.770 --> 00:35:21.270 align:middle line:90%
Yeah.

00:35:21.270 --> 00:35:25.350 align:middle line:90%


00:35:25.350 --> 00:35:28.070 align:middle line:84%
STUDENT: This logic
of having access

00:35:28.070 --> 00:35:30.687 align:middle line:90%
to all the previous iterations--

00:35:30.687 --> 00:35:31.770 align:middle line:90%
PROFESSOR: Not iterations.

00:35:31.770 --> 00:35:33.275 align:middle line:84%
STUDENT: All the
previous layers.

00:35:33.275 --> 00:35:35.400 align:middle line:84%
PROFESSOR: Correct, outputs
of the previous layers.

00:35:35.400 --> 00:35:36.870 align:middle line:90%
STUDENT: Layers.

00:35:36.870 --> 00:35:41.110 align:middle line:84%
But this would also
help for clean up

00:35:41.110 --> 00:35:42.870 align:middle line:90%
and give better categorization.

00:35:42.870 --> 00:35:45.060 align:middle line:84%
Does it always have to
be an image-to-image?

00:35:45.060 --> 00:35:45.810 align:middle line:90%
PROFESSOR: No, no.

00:35:45.810 --> 00:35:47.957 align:middle line:84%
In fact, if you look
at ResNet, ResNet

00:35:47.957 --> 00:35:49.790 align:middle line:84%
is the one, in fact,
that pioneered the idea

00:35:49.790 --> 00:35:51.390 align:middle line:90%
of the residual connection.

00:35:51.390 --> 00:35:52.960 align:middle line:90%
So we use it for ResNet.

00:35:52.960 --> 00:35:56.120 align:middle line:84%
We actually use the
transformer stack.

00:35:56.120 --> 00:35:59.120 align:middle line:84%
If you remember, it goes through
the self-attention layer.

00:35:59.120 --> 00:36:00.440 align:middle line:90%
It comes out the other end.

00:36:00.440 --> 00:36:03.400 align:middle line:84%
And then, we add the
input back to it.

00:36:03.400 --> 00:36:05.900 align:middle line:84%
And then, we send it
through LayerNorm.

00:36:05.900 --> 00:36:07.780 align:middle line:84%
So you will see that
this residual connection

00:36:07.780 --> 00:36:10.260 align:middle line:84%
is sitting in two different
places in a single transformer

00:36:10.260 --> 00:36:11.460 align:middle line:90%
block.

00:36:11.460 --> 00:36:13.300 align:middle line:90%
So it's extremely heavily used.

00:36:13.300 --> 00:36:16.000 align:middle line:84%
There is something called Deep
and Wide Network, if I remember,

00:36:16.000 --> 00:36:18.700 align:middle line:84%
or DenseNet, which
uses the same trick.

00:36:18.700 --> 00:36:23.200 align:middle line:84%
In fact, when you're working
with structured data, good, old,

00:36:23.200 --> 00:36:25.940 align:middle line:84%
say, linear regression, and
you've looked at your data

00:36:25.940 --> 00:36:28.780 align:middle line:84%
and you've come up with all
kinds of very clever features.

00:36:28.780 --> 00:36:30.940 align:middle line:84%
I'm going to look at
price per square foot.

00:36:30.940 --> 00:36:32.740 align:middle line:84%
You do a bunch of
feature engineering

00:36:32.740 --> 00:36:34.660 align:middle line:84%
and you have a bunch
of new features.

00:36:34.660 --> 00:36:38.540 align:middle line:84%
Well, you should take your old
features and your new features

00:36:38.540 --> 00:36:40.580 align:middle line:90%
and send both in.

00:36:40.580 --> 00:36:42.945 align:middle line:84%
Why send only the new stuff
that you have concocted?

00:36:42.945 --> 00:36:44.320 align:middle line:84%
Why can't you send
everything in?

00:36:44.320 --> 00:36:44.987 align:middle line:90%
That's the idea.

00:36:44.987 --> 00:36:47.420 align:middle line:90%


00:36:47.420 --> 00:36:50.360 align:middle line:84%
All right, so let's
come back here.

00:36:50.360 --> 00:36:53.090 align:middle line:90%


00:36:53.090 --> 00:36:56.010 align:middle line:84%
Now, we have seen how to
generate a good image.

00:36:56.010 --> 00:36:58.050 align:middle line:84%
Now, let's figure
out how to steer it

00:36:58.050 --> 00:37:00.396 align:middle line:84%
or condition it with a
text prompt because that's,

00:37:00.396 --> 00:37:02.730 align:middle line:90%
sort of, the Holy Grail.

00:37:02.730 --> 00:37:07.957 align:middle line:84%
So we want to take, so
here is some intuition.

00:37:07.957 --> 00:37:09.790 align:middle line:84%
We want to take the
text prompt into account

00:37:09.790 --> 00:37:11.248 align:middle line:84%
and obviously
generating the image.

00:37:11.248 --> 00:37:15.530 align:middle line:84%
Now, imagine if we had like a
rough image that corresponds

00:37:15.530 --> 00:37:16.770 align:middle line:90%
to that text prompt.

00:37:16.770 --> 00:37:17.850 align:middle line:90%
Just imagine.

00:37:17.850 --> 00:37:20.930 align:middle line:84%
So the text prompt is
cute Labrador Retriever.

00:37:20.930 --> 00:37:23.610 align:middle line:84%
And you have a very noisy
image of a Labrador Retriever

00:37:23.610 --> 00:37:25.070 align:middle line:90%
that just happens to be handy.

00:37:25.070 --> 00:37:26.170 align:middle line:90%
You have it.

00:37:26.170 --> 00:37:29.130 align:middle line:84%
Well, now, you're in good shape
because you just feed that in,

00:37:29.130 --> 00:37:31.370 align:middle line:84%
and your system will
denoise it for you.

00:37:31.370 --> 00:37:32.670 align:middle line:90%
You can get a better image.

00:37:32.670 --> 00:37:33.970 align:middle line:90%
That's pretty easy.

00:37:33.970 --> 00:37:36.303 align:middle line:84%
But obviously, in reality,
you don't have a rough image.

00:37:36.303 --> 00:37:37.803 align:middle line:84%
In fact, you're
trying to create one

00:37:37.803 --> 00:37:39.290 align:middle line:84%
of those things in
the first place.

00:37:39.290 --> 00:37:39.950 align:middle line:90%
We don't.

00:37:39.950 --> 00:37:44.770 align:middle line:84%
But what if we had an
embedding for the prompt that's

00:37:44.770 --> 00:37:47.290 align:middle line:84%
close to the embeddings
of all the images

00:37:47.290 --> 00:37:48.670 align:middle line:90%
that correspond to the prompt?

00:37:48.670 --> 00:37:50.880 align:middle line:84%
So let's take a
prompt, and let's

00:37:50.880 --> 00:37:54.960 align:middle line:84%
imagine all the images in the
universe that correspond to that

00:37:54.960 --> 00:37:55.640 align:middle line:90%
prompt.

00:37:55.640 --> 00:37:59.858 align:middle line:84%
And now, further imagine,
because everything is a vector,

00:37:59.858 --> 00:38:01.400 align:middle line:84%
everything is
embedding in our world,

00:38:01.400 --> 00:38:05.320 align:middle line:90%
that the image has an embedding.

00:38:05.320 --> 00:38:07.720 align:middle line:84%
Sorry, the text prompt
has an embedding.

00:38:07.720 --> 00:38:09.960 align:middle line:90%
Every image has an embedding.

00:38:09.960 --> 00:38:13.360 align:middle line:84%
And we have somehow
calculated these embeddings

00:38:13.360 --> 00:38:17.080 align:middle line:84%
so that the text prompt's
embedding is smack where

00:38:17.080 --> 00:38:20.040 align:middle line:90%
all the image embeddings are.

00:38:20.040 --> 00:38:22.700 align:middle line:84%
We will get to how we actually
do it in just a moment.

00:38:22.700 --> 00:38:25.700 align:middle line:84%
But conceptually, imagine
if we had an embedding,

00:38:25.700 --> 00:38:27.200 align:middle line:84%
if you could
calculate embeddings

00:38:27.200 --> 00:38:29.420 align:middle line:84%
for text and
embeddings for images,

00:38:29.420 --> 00:38:33.080 align:middle line:84%
so they all live
in the same space

00:38:33.080 --> 00:38:36.440 align:middle line:84%
So if we feed this
embedding to a denoising

00:38:36.440 --> 00:38:40.960 align:middle line:84%
model, because that text
embedding is sitting

00:38:40.960 --> 00:38:44.160 align:middle line:84%
in the same space as all
the image embeddings that it

00:38:44.160 --> 00:38:48.670 align:middle line:84%
corresponds to, maybe our model
can just denoise that embedding

00:38:48.670 --> 00:38:52.350 align:middle line:90%
and give you what you want.

00:38:52.350 --> 00:38:55.030 align:middle line:84%
So since this embedding is
already close to the embeddings

00:38:55.030 --> 00:38:56.450 align:middle line:84%
of the things we
want to generate,

00:38:56.450 --> 00:38:59.230 align:middle line:90%
maybe we'll just get it done.

00:38:59.230 --> 00:39:01.190 align:middle line:84%
So ultimately, we want
to generate an image.

00:39:01.190 --> 00:39:02.950 align:middle line:84%
And if we had an
embedding for that image,

00:39:02.950 --> 00:39:05.310 align:middle line:84%
we could generate the
image from the embedding.

00:39:05.310 --> 00:39:07.030 align:middle line:90%
And we use the text.

00:39:07.030 --> 00:39:09.295 align:middle line:84%
So we go from text
to embedding, which

00:39:09.295 --> 00:39:10.670 align:middle line:84%
happens to live
in the same space

00:39:10.670 --> 00:39:13.050 align:middle line:84%
as all the embeddings of
the images we care about.

00:39:13.050 --> 00:39:14.550 align:middle line:84%
And then, from that
image embedding,

00:39:14.550 --> 00:39:15.630 align:middle line:90%
we go to the final image.

00:39:15.630 --> 00:39:16.510 align:middle line:90%
OK.

00:39:16.510 --> 00:39:18.920 align:middle line:84%
This is a bunch of me
talking and hand waving.

00:39:18.920 --> 00:39:20.170 align:middle line:90%
It will all become very clear.

00:39:20.170 --> 00:39:22.296 align:middle line:90%
But that's the rough intuition.

00:39:22.296 --> 00:39:24.110 align:middle line:90%
OK.

00:39:24.110 --> 00:39:26.190 align:middle line:84%
So what we'll do is we'll
describe an approach

00:39:26.190 --> 00:39:29.590 align:middle line:84%
to calculate an embedding for
any text, any piece of text,

00:39:29.590 --> 00:39:33.430 align:middle line:84%
that is close to the embeddings
of the images that correspond

00:39:33.430 --> 00:39:35.255 align:middle line:90%
to that piece of text.

00:39:35.255 --> 00:39:37.130 align:middle line:84%
So this is the problem
you're going to solve.

00:39:37.130 --> 00:39:38.510 align:middle line:90%
There's a bunch of text.

00:39:38.510 --> 00:39:40.052 align:middle line:84%
Conceptually, there
are a whole bunch

00:39:40.052 --> 00:39:41.850 align:middle line:84%
of images that
describe the text.

00:39:41.850 --> 00:39:43.670 align:middle line:84%
And we're going to
now create embeddings

00:39:43.670 --> 00:39:48.020 align:middle line:84%
so that that is close to all
the embeddings of those images.

00:39:48.020 --> 00:39:50.620 align:middle line:84%
It feels, kind of, almost
impossible that you can actually

00:39:50.620 --> 00:39:52.380 align:middle line:90%
do something like this.

00:39:52.380 --> 00:39:57.620 align:middle line:84%
But there's a very clever idea
that OpenAI came up with that

00:39:57.620 --> 00:39:58.740 align:middle line:90%
tells you how to do it.

00:39:58.740 --> 00:40:00.300 align:middle line:84%
So here's what
we're going to do.

00:40:00.300 --> 00:40:03.100 align:middle line:84%
So let's say we have
an image and a caption.

00:40:03.100 --> 00:40:04.940 align:middle line:90%
So here is an image.

00:40:04.940 --> 00:40:06.100 align:middle line:90%
Here's a caption.

00:40:06.100 --> 00:40:10.162 align:middle line:84%
And we need some way to
take that piece of text

00:40:10.162 --> 00:40:12.620 align:middle line:84%
and run it through some network
and create a nice embedding

00:40:12.620 --> 00:40:13.780 align:middle line:90%
from it.

00:40:13.780 --> 00:40:15.800 align:middle line:84%
Similarly, we want
to take this image,

00:40:15.800 --> 00:40:18.667 align:middle line:84%
run it through some network and
create an embedding from it.

00:40:18.667 --> 00:40:19.500 align:middle line:90%
Now, first question.

00:40:19.500 --> 00:40:21.625 align:middle line:84%
How can we compute embeddings
from a piece of text?

00:40:21.625 --> 00:40:23.583 align:middle line:84%
First question, how can
we compute an embedding

00:40:23.583 --> 00:40:24.600 align:middle line:90%
from a piece of text?

00:40:24.600 --> 00:40:27.780 align:middle line:90%
You know the answer.

00:40:27.780 --> 00:40:29.700 align:middle line:90%
Run it through a transformer.

00:40:29.700 --> 00:40:30.360 align:middle line:90%
Piece of cake.

00:40:30.360 --> 00:40:31.326 align:middle line:90%
We know how to do that.

00:40:31.326 --> 00:40:34.420 align:middle line:90%


00:40:34.420 --> 00:40:36.740 align:middle line:84%
In particular, you can
do something like BERT.

00:40:36.740 --> 00:40:38.220 align:middle line:84%
And for an image
encoder, you just

00:40:38.220 --> 00:40:40.700 align:middle line:84%
run it through
something like ResNet.

00:40:40.700 --> 00:40:43.657 align:middle line:84%
The penultimate layer, one of
the final layers, the ResNet

00:40:43.657 --> 00:40:45.990 align:middle line:84%
is going to be a very good
representation of that image.

00:40:45.990 --> 00:40:47.850 align:middle line:90%
You get another embedding.

00:40:47.850 --> 00:40:49.970 align:middle line:84%
So using the building
blocks, we already know,

00:40:49.970 --> 00:40:53.570 align:middle line:84%
we can create embeddings very
quickly from these things.

00:40:53.570 --> 00:40:56.470 align:middle line:84%
But if you just take a piece of
text and run it through a BERT,

00:40:56.470 --> 00:40:58.470 align:middle line:84%
and you take an image and
run it through ResNet,

00:40:58.470 --> 00:40:59.970 align:middle line:84%
you're going to get
some embeddings.

00:40:59.970 --> 00:41:04.470 align:middle line:84%
But why the heck
should they be related?

00:41:04.470 --> 00:41:06.970 align:middle line:84%
They were not trained together,
so there's no basis for them

00:41:06.970 --> 00:41:08.210 align:middle line:90%
to be related.

00:41:08.210 --> 00:41:10.190 align:middle line:84%
They would just be
some two embeddings.

00:41:10.190 --> 00:41:11.678 align:middle line:90%
Maybe they are kind of similar.

00:41:11.678 --> 00:41:12.470 align:middle line:90%
Maybe they are not.

00:41:12.470 --> 00:41:13.070 align:middle line:90%
We don't know.

00:41:13.070 --> 00:41:16.470 align:middle line:84%
There's no reason to expect that
they're going to be similar.

00:41:16.470 --> 00:41:17.590 align:middle line:90%
They just do embeddings.

00:41:17.590 --> 00:41:20.250 align:middle line:90%


00:41:20.250 --> 00:41:23.367 align:middle line:84%
Now, what we want to do
is, once we have these,

00:41:23.367 --> 00:41:25.450 align:middle line:84%
we need to make sure the
embeddings that comes out

00:41:25.450 --> 00:41:28.710 align:middle line:84%
of these two things satisfy two
very important requirements.

00:41:28.710 --> 00:41:31.903 align:middle line:90%


00:41:31.903 --> 00:41:33.570 align:middle line:84%
We want to make sure
that if you give it

00:41:33.570 --> 00:41:38.660 align:middle line:84%
an image and a caption
that describes that image.

00:41:38.660 --> 00:41:40.410 align:middle line:84%
So you have an image
and a subcaption that

00:41:40.410 --> 00:41:42.480 align:middle line:84%
describes that image,
we want to make sure

00:41:42.480 --> 00:41:44.985 align:middle line:84%
that the embeddings that come
out of these two boxes, they

00:41:44.985 --> 00:41:46.610 align:middle line:84%
are as close to each
other as possible.

00:41:46.610 --> 00:41:49.260 align:middle line:90%


00:41:49.260 --> 00:41:51.300 align:middle line:84%
Given an image and a
caption that describes it,

00:41:51.300 --> 00:41:52.380 align:middle line:90%
that's the connection.

00:41:52.380 --> 00:41:53.880 align:middle line:84%
They have to be
close to each other.

00:41:53.880 --> 00:41:58.040 align:middle line:84%
And conversely, if you have
an image and a caption, that's

00:41:58.040 --> 00:42:01.960 align:middle line:84%
totally irrelevant, a
train rounding a bend

00:42:01.960 --> 00:42:05.380 align:middle line:84%
to the beautiful fall foliage
all around, clearly irrelevant,

00:42:05.380 --> 00:42:07.960 align:middle line:84%
those embeddings
should be far apart.

00:42:07.960 --> 00:42:11.000 align:middle line:90%
That intuitively makes sense.

00:42:11.000 --> 00:42:12.980 align:middle line:84%
Pairs of related things
should be together.

00:42:12.980 --> 00:42:14.920 align:middle line:84%
Irrelevant things
should be far apart.

00:42:14.920 --> 00:42:18.140 align:middle line:84%
So if you can find embeddings
that satisfy these two criteria,

00:42:18.140 --> 00:42:20.880 align:middle line:90%
maybe we will be in the game.

00:42:20.880 --> 00:42:24.920 align:middle line:84%
So now, this ensures that the
text embedding and the image

00:42:24.920 --> 00:42:27.880 align:middle line:84%
embedding are referring to
the same underlying concept.

00:42:27.880 --> 00:42:31.080 align:middle line:84%
These requirements
will enforce that.

00:42:31.080 --> 00:42:32.783 align:middle line:84%
And so the embedding
for any text prompt

00:42:32.783 --> 00:42:34.200 align:middle line:84%
is close to the
embeddings though,

00:42:34.200 --> 00:42:36.283 align:middle line:84%
for all the images that
correspond to that prompt.

00:42:36.283 --> 00:42:38.680 align:middle line:90%


00:42:38.680 --> 00:42:41.712 align:middle line:84%
So the question is,
how do we do this?

00:42:41.712 --> 00:42:44.170 align:middle line:84%
First of all, how can we tell
how close two embeddings are?

00:42:44.170 --> 00:42:45.337 align:middle line:90%
You know the answer to this.

00:42:45.337 --> 00:42:46.105 align:middle line:90%
What's the answer?

00:42:46.105 --> 00:42:47.230 align:middle line:90%
STUDENT: Cosine similarity.

00:42:47.230 --> 00:42:49.110 align:middle line:84%
PROFESSOR: Correct,
cosine similarity.

00:42:49.110 --> 00:42:52.390 align:middle line:84%
We use the cosine similarity
of the embeddings.

00:42:52.390 --> 00:42:53.890 align:middle line:84%
So we know how to
measure closeness.

00:42:53.890 --> 00:42:56.140 align:middle line:84%
So the question is, how can
we compute embeddings that

00:42:56.140 --> 00:42:57.470 align:middle line:90%
satisfy the two requirements.

00:42:57.470 --> 00:43:01.710 align:middle line:84%
And OpenAI built a model called
CLIP, which is very famous,

00:43:01.710 --> 00:43:02.810 align:middle line:90%
to solve this problem.

00:43:02.810 --> 00:43:07.190 align:middle line:84%
It stands for Contrastive
Language-Image Pretaining.

00:43:07.190 --> 00:43:09.430 align:middle line:84%
And this forms the basis
for a whole bunch of models

00:43:09.430 --> 00:43:11.910 align:middle line:84%
that have sprung up after
this called BLIP and BLIP 2

00:43:11.910 --> 00:43:15.790 align:middle line:84%
and so on and so forth, but
this is the fundamental idea.

00:43:15.790 --> 00:43:19.910 align:middle line:90%
So this is how clip works.

00:43:19.910 --> 00:43:25.470 align:middle line:84%
What they did is they
took a 12-block, 12-layer,

00:43:25.470 --> 00:43:31.642 align:middle line:84%
8-head Transformer Causal
Encoder Stack as a text encoder.

00:43:31.642 --> 00:43:33.150 align:middle line:90%
Now, you understand this, right?

00:43:33.150 --> 00:43:35.270 align:middle line:90%
That's what it is.

00:43:35.270 --> 00:43:38.820 align:middle line:84%
8-head, 12-layer Transformer
Causal Encoder, TCE Stack.

00:43:38.820 --> 00:43:40.520 align:middle line:90%
And that's a text encoder.

00:43:40.520 --> 00:43:42.500 align:middle line:84%
So we send any piece
of text through it.

00:43:42.500 --> 00:43:45.220 align:middle line:84%
You get the next word
prediction embedding.

00:43:45.220 --> 00:43:47.780 align:middle line:84%
And that's embedding
you're going to use.

00:43:47.780 --> 00:43:51.120 align:middle line:84%
And they took ResNet-50 and
made it the image encoder.

00:43:51.120 --> 00:43:54.020 align:middle line:84%
They took ResNet-50,
chopped off the top.

00:43:54.020 --> 00:43:57.220 align:middle line:84%
And whatever was left
is the image encoder.

00:43:57.220 --> 00:43:59.060 align:middle line:90%
OK.

00:43:59.060 --> 00:44:03.020 align:middle line:84%
Then, they initialized with
random weights, these things,

00:44:03.020 --> 00:44:06.760 align:middle line:84%
and then they grab a batch
of image caption pairs.

00:44:06.760 --> 00:44:09.820 align:middle line:84%
So in this example, let's say
that we have these three images,

00:44:09.820 --> 00:44:13.060 align:middle line:84%
and I have captions to
go with these images.

00:44:13.060 --> 00:44:14.860 align:middle line:90%
We have these three things.

00:44:14.860 --> 00:44:17.380 align:middle line:90%
And this is the key step.

00:44:17.380 --> 00:44:20.820 align:middle line:84%
They run the images through the
image encoder and the captions

00:44:20.820 --> 00:44:23.600 align:middle line:84%
through the text encoder
and get these embeddings.

00:44:23.600 --> 00:44:24.860 align:middle line:90%
It's a forward pass.

00:44:24.860 --> 00:44:29.580 align:middle line:84%
You send it through this
network, you get to embeddings.

00:44:29.580 --> 00:44:32.002 align:middle line:90%
And then, this is what they do.

00:44:32.002 --> 00:44:34.460 align:middle line:84%
With these embeddings, they
calculate the cosine similarity

00:44:34.460 --> 00:44:36.380 align:middle line:90%
for every image caption pair.

00:44:36.380 --> 00:44:37.060 align:middle line:90%
OK.

00:44:37.060 --> 00:44:39.340 align:middle line:84%
And so imagine
something like this.

00:44:39.340 --> 00:44:41.420 align:middle line:84%
So you have these
three captions.

00:44:41.420 --> 00:44:43.200 align:middle line:90%
You have these three images.

00:44:43.200 --> 00:44:45.540 align:middle line:90%
And those are the embeddings.

00:44:45.540 --> 00:44:48.140 align:middle line:84%
And then, they calculate
the cosine similarity

00:44:48.140 --> 00:44:50.870 align:middle line:90%
for every one of those things.

00:44:50.870 --> 00:44:52.620 align:middle line:84%
It took me like five
minutes or 10 minutes

00:44:52.620 --> 00:44:54.340 align:middle line:90%
to do this PowerPoint.

00:44:54.340 --> 00:44:55.581 align:middle line:90%
You're welcome.

00:44:55.581 --> 00:45:00.660 align:middle line:90%
[LAUGHS]

00:45:00.660 --> 00:45:02.580 align:middle line:84%
Particularly, trying to
get this comma to line

00:45:02.580 --> 00:45:05.060 align:middle line:90%
was as a real pain in the neck.

00:45:05.060 --> 00:45:06.180 align:middle line:90%
So all right.

00:45:06.180 --> 00:45:09.020 align:middle line:84%
So we have this
here OK, now what

00:45:09.020 --> 00:45:14.140 align:middle line:84%
we want to do is we want these
scores to be as high as possible

00:45:14.140 --> 00:45:17.020 align:middle line:84%
because the scores
in the diagonal

00:45:17.020 --> 00:45:20.050 align:middle line:84%
are the ones for the
matching picture and caption.

00:45:20.050 --> 00:45:24.282 align:middle line:90%


00:45:24.282 --> 00:45:27.185 align:middle line:84%
Those are the scores for the
matching pairs of embeddings.

00:45:27.185 --> 00:45:28.810 align:middle line:84%
We want them to be
as high as possible.

00:45:28.810 --> 00:45:33.420 align:middle line:90%


00:45:33.420 --> 00:45:36.410 align:middle line:84%
So we want to maximize the
sum of the green cells.

00:45:36.410 --> 00:45:39.490 align:middle line:84%
These are the green
cells, the diagonal.

00:45:39.490 --> 00:45:42.170 align:middle line:84%
So if you want to write
it as a loss function,

00:45:42.170 --> 00:45:44.270 align:middle line:84%
because the loss function
is always minimization,

00:45:44.270 --> 00:45:47.660 align:middle line:84%
we basically say minimize the
negative sum of the green cells.

00:45:47.660 --> 00:45:50.782 align:middle line:90%


00:45:50.782 --> 00:45:53.790 align:middle line:84%
So the question is, will this
loss function do the trick?

00:45:53.790 --> 00:45:58.770 align:middle line:90%


00:45:58.770 --> 00:46:00.330 align:middle line:90%
It seems reasonable.

00:46:00.330 --> 00:46:02.970 align:middle line:84%
You want to make sure that
related things are really close

00:46:02.970 --> 00:46:03.630 align:middle line:90%
together.

00:46:03.630 --> 00:46:04.870 align:middle line:90%
So you want to maximize.

00:46:04.870 --> 00:46:07.197 align:middle line:90%


00:46:07.197 --> 00:46:09.530 align:middle line:84%
STUDENT: If that was the only
part of the loss function,

00:46:09.530 --> 00:46:11.690 align:middle line:84%
wouldn't it just kind
of squish everything

00:46:11.690 --> 00:46:13.210 align:middle line:90%
to the same spot in space?

00:46:13.210 --> 00:46:14.650 align:middle line:90%
PROFESSOR: Correct.

00:46:14.650 --> 00:46:16.730 align:middle line:84%
What it's going to
do is it's going

00:46:16.730 --> 00:46:20.170 align:middle line:90%
to basically ignore the input.

00:46:20.170 --> 00:46:22.230 align:middle line:84%
The optimizer can
simply ignore the input,

00:46:22.230 --> 00:46:24.450 align:middle line:84%
make all the
embeddings the same.

00:46:24.450 --> 00:46:28.410 align:middle line:84%
For example, it can just
make all the embeddings 0.

00:46:28.410 --> 00:46:28.950 align:middle line:90%
That's it.

00:46:28.950 --> 00:46:31.690 align:middle line:84%
And then, now, we have a
perfect cosine similarity

00:46:31.690 --> 00:46:33.160 align:middle line:90%
for everything.

00:46:33.160 --> 00:46:35.460 align:middle line:84%
For any pair of
image and captions,

00:46:35.460 --> 00:46:37.240 align:middle line:84%
the cosine similarity
is going to be 1.

00:46:37.240 --> 00:46:38.400 align:middle line:90%
It's perfect.

00:46:38.400 --> 00:46:40.640 align:middle line:90%
So clearly, that's not enough.

00:46:40.640 --> 00:46:43.240 align:middle line:84%
This is, by the way, is
called model collapse.

00:46:43.240 --> 00:46:45.980 align:middle line:84%
So to prevent it
from doing that,

00:46:45.980 --> 00:46:48.320 align:middle line:84%
we need to do one more
thing to the loss function.

00:46:48.320 --> 00:46:48.950 align:middle line:90%
Any guesses?

00:46:48.950 --> 00:46:51.560 align:middle line:90%


00:46:51.560 --> 00:46:53.360 align:middle line:90%
Yeah.

00:46:53.360 --> 00:46:55.960 align:middle line:84%
STUDENT: Make the images
that aren't related

00:46:55.960 --> 00:46:58.098 align:middle line:90%
not have a cosine similarity.

00:46:58.098 --> 00:46:59.140 align:middle line:90%
PROFESSOR: Exactly right.

00:46:59.140 --> 00:47:00.200 align:middle line:90%
Exactly right.

00:47:00.200 --> 00:47:04.280 align:middle line:84%
So what we want to do is we
want the scores of the red stuff

00:47:04.280 --> 00:47:07.240 align:middle line:90%
to be as small as possible.

00:47:07.240 --> 00:47:08.840 align:middle line:84%
We want the green
stuff to be as much

00:47:08.840 --> 00:47:12.560 align:middle line:84%
as possible and the red stuff
to be as small as possible.

00:47:12.560 --> 00:47:14.420 align:middle line:84%
Together, it'll
get the job done.

00:47:14.420 --> 00:47:19.478 align:middle line:90%


00:47:19.478 --> 00:47:21.520 align:middle line:84%
So we want to maximize
the sum of the green cells

00:47:21.520 --> 00:47:23.360 align:middle line:84%
and minimize the sum
of the red cells,

00:47:23.360 --> 00:47:26.400 align:middle line:84%
so the equivalent loss
function is minimize

00:47:26.400 --> 00:47:29.440 align:middle line:84%
the sum of the red cells and the
negative sum of the green cells.

00:47:29.440 --> 00:47:30.830 align:middle line:90%
That's it.

00:47:30.830 --> 00:47:35.750 align:middle line:84%
So all CLIP does is that it just
grabs a batch of image caption

00:47:35.750 --> 00:47:38.470 align:middle line:84%
pairs, runs it
through the networks,

00:47:38.470 --> 00:47:41.550 align:middle line:84%
calculates the embeddings and
calculates some of the stuff

00:47:41.550 --> 00:47:42.570 align:middle line:90%
here.

00:47:42.570 --> 00:47:43.910 align:middle line:90%
And that is your loss.

00:47:43.910 --> 00:47:46.190 align:middle line:84%
And then, back propagate
through the network.

00:47:46.190 --> 00:47:47.006 align:middle line:90%
Boom.

00:47:47.006 --> 00:47:48.230 align:middle line:90%
Batch, batch, batch.

00:47:48.230 --> 00:47:49.710 align:middle line:90%
Do it a whole bunch of times.

00:47:49.710 --> 00:47:52.310 align:middle line:84%
And OpenAI did
this with, oh, this

00:47:52.310 --> 00:47:55.390 align:middle line:84%
is the official picture
from the paper, which

00:47:55.390 --> 00:47:56.870 align:middle line:90%
is worth reading, by the way.

00:47:56.870 --> 00:47:59.010 align:middle line:90%
It comes in, text encoder.

00:47:59.010 --> 00:48:02.110 align:middle line:84%
You get these embedding
vectors, image encoder.

00:48:02.110 --> 00:48:05.250 align:middle line:84%
And then, boom, the
diagonal is maximized

00:48:05.250 --> 00:48:07.790 align:middle line:84%
and the off diagonals
are minimized.

00:48:07.790 --> 00:48:11.230 align:middle line:84%
And they did it with 400
million image caption pairs

00:48:11.230 --> 00:48:14.430 align:middle line:90%
scraped from the internet.

00:48:14.430 --> 00:48:16.430 align:middle line:90%
400 million.

00:48:16.430 --> 00:48:18.750 align:middle line:84%
By the way, folks who
work in this space

00:48:18.750 --> 00:48:21.630 align:middle line:84%
may know this really
well, but one very

00:48:21.630 --> 00:48:25.190 align:middle line:84%
easy way to get a
caption for an image.

00:48:25.190 --> 00:48:27.970 align:middle line:84%
We see images, but where do you
think the captions come from?

00:48:27.970 --> 00:48:29.455 align:middle line:84%
Where did they get
those captions?

00:48:29.455 --> 00:48:31.580 align:middle line:84%
Obviously, they didn't ask
people to manually label

00:48:31.580 --> 00:48:32.860 align:middle line:90%
each image with the caption.

00:48:32.860 --> 00:48:35.020 align:middle line:84%
Where do you think
they got it from?

00:48:35.020 --> 00:48:36.460 align:middle line:90%
STUDENT: Google Search?

00:48:36.460 --> 00:48:39.340 align:middle line:84%
PROFESSOR: Google Search
can help, but why does

00:48:39.340 --> 00:48:41.160 align:middle line:84%
Google Search actually
find the caption?

00:48:41.160 --> 00:48:42.868 align:middle line:84%
How does it, because
Google Search is not

00:48:42.868 --> 00:48:45.318 align:middle line:90%
creating the caption.

00:48:45.318 --> 00:48:47.360 align:middle line:84%
STUDENT: Take it from the
alt text on the images.

00:48:47.360 --> 00:48:49.460 align:middle line:90%
PROFESSOR: Correct, alt text.

00:48:49.460 --> 00:48:51.480 align:middle line:84%
So a lot of folks, for
accessibility reasons,

00:48:51.480 --> 00:48:54.320 align:middle line:84%
they have alt text on
the images they create.

00:48:54.320 --> 00:48:56.220 align:middle line:84%
A lot of people have
alt text in their images

00:48:56.220 --> 00:48:59.220 align:middle line:84%
that they have published on the
web, and that's what we use.

00:48:59.220 --> 00:49:01.780 align:middle line:84%
And the alt text
actually, ends up

00:49:01.780 --> 00:49:04.620 align:middle line:84%
being a more verbose
description of the image

00:49:04.620 --> 00:49:07.420 align:middle line:84%
than a typical caption, which
tends to be much briefer.

00:49:07.420 --> 00:49:10.660 align:middle line:84%
And for us, more verbose,
longer the better,

00:49:10.660 --> 00:49:14.540 align:middle line:84%
because there's more stuff
for the model to learn from.

00:49:14.540 --> 00:49:17.180 align:middle line:90%
So that's why they built CLIP.

00:49:17.180 --> 00:49:20.300 align:middle line:84%
And so now, what we do
is we can use CLIP's text

00:49:20.300 --> 00:49:21.980 align:middle line:90%
encoder by itself.

00:49:21.980 --> 00:49:25.100 align:middle line:84%
We can send in any text
and get an embedding that

00:49:25.100 --> 00:49:27.690 align:middle line:84%
is close to the embedding
of any image that's

00:49:27.690 --> 00:49:31.010 align:middle line:90%
described by the text.

00:49:31.010 --> 00:49:31.890 align:middle line:90%
OK.

00:49:31.890 --> 00:49:35.650 align:middle line:84%
Now, by the way, CLIP can
be used for zero-shot image

00:49:35.650 --> 00:49:37.410 align:middle line:90%
classification.

00:49:37.410 --> 00:49:39.830 align:middle line:84%
And what I mean by zero-shot
image classification,

00:49:39.830 --> 00:49:41.850 align:middle line:84%
and I'll walk through
the picture in a second,

00:49:41.850 --> 00:49:45.150 align:middle line:84%
is that typically when you want
to build an image classifier,

00:49:45.150 --> 00:49:47.770 align:middle line:84%
you get a whole bunch of
training data of images

00:49:47.770 --> 00:49:50.270 align:middle line:84%
and their labels, and
then we train them.

00:49:50.270 --> 00:49:54.010 align:middle line:84%
Maybe you take something like
ResNet, chop off the top,

00:49:54.010 --> 00:49:56.320 align:middle line:84%
attach our own output head
and train, train, train.

00:49:56.320 --> 00:49:56.820 align:middle line:90%
Boom.

00:49:56.820 --> 00:49:58.190 align:middle line:90%
You have a classifier.

00:49:58.190 --> 00:50:00.890 align:middle line:84%
But the only problem with that
is, let's say that tomorrow--

00:50:00.890 --> 00:50:04.268 align:middle line:84%
so today, for example, you had
five classes in your problem.

00:50:04.268 --> 00:50:06.810 align:middle line:84%
And tomorrow, somebody comes
along and says, oh, actually, we

00:50:06.810 --> 00:50:09.357 align:middle line:90%
have a sixth category.

00:50:09.357 --> 00:50:10.190 align:middle line:90%
What do you do then?

00:50:10.190 --> 00:50:11.690 align:middle line:84%
Well, you have to go
back to the drawing board

00:50:11.690 --> 00:50:13.990 align:middle line:84%
and retrain the whole
thing with six labels now,

00:50:13.990 --> 00:50:16.450 align:middle line:84%
not five, because your
problem has changed.

00:50:16.450 --> 00:50:19.330 align:middle line:84%
Wouldn't it be great if you
had a classifier where you just

00:50:19.330 --> 00:50:21.288 align:middle line:84%
come to it and say,
here's an image,

00:50:21.288 --> 00:50:23.830 align:middle line:84%
and here are the six possible
labels I want you to pick from.

00:50:23.830 --> 00:50:25.380 align:middle line:90%
Pick one for me.

00:50:25.380 --> 00:50:27.880 align:middle line:84%
And you want to be able to give
it a different set of labels

00:50:27.880 --> 00:50:29.560 align:middle line:90%
each time.

00:50:29.560 --> 00:50:32.360 align:middle line:84%
And it will just use the labels
you're giving it, and the image,

00:50:32.360 --> 00:50:34.040 align:middle line:84%
and figures out which
label corresponds

00:50:34.040 --> 00:50:35.880 align:middle line:90%
to the image you just fed it.

00:50:35.880 --> 00:50:38.920 align:middle line:84%
That would be an insanely
flexible image classification

00:50:38.920 --> 00:50:40.440 align:middle line:90%
system.

00:50:40.440 --> 00:50:43.560 align:middle line:84%
And that's what I mean by
zero-shot image classification.

00:50:43.560 --> 00:50:48.523 align:middle line:84%
And you can use CLIP to do
zero-shot image classification.

00:50:48.523 --> 00:50:50.440 align:middle line:84%
Now, how you do it is
actually in the picture,

00:50:50.440 --> 00:50:52.280 align:middle line:90%
though not very clearly done.

00:50:52.280 --> 00:50:53.460 align:middle line:90%
Anyone wants to guess?

00:50:53.460 --> 00:50:58.120 align:middle line:90%


00:50:58.120 --> 00:51:02.320 align:middle line:84%
How can you use CLIP to build
an infinitely flexible image

00:51:02.320 --> 00:51:03.100 align:middle line:90%
classifier?

00:51:03.100 --> 00:51:12.560 align:middle line:90%


00:51:12.560 --> 00:51:15.480 align:middle line:84%
STUDENT: I mean, the text
input was trained BERT, right?

00:51:15.480 --> 00:51:17.360 align:middle line:84%
So in the same way
BERT can handle

00:51:17.360 --> 00:51:20.787 align:middle line:84%
words it's never seen before,
does it essentially do that?

00:51:20.787 --> 00:51:22.870 align:middle line:84%
PROFESSOR: Sorry, say that
again, the second part.

00:51:22.870 --> 00:51:25.110 align:middle line:84%
STUDENT: You're saying it sees
a text input with something

00:51:25.110 --> 00:51:26.270 align:middle line:90%
it's never seen before, right?

00:51:26.270 --> 00:51:26.550 align:middle line:90%
PROFESSOR: Yeah.

00:51:26.550 --> 00:51:27.130 align:middle line:90%
STUDENT: OK.

00:51:27.130 --> 00:51:30.230 align:middle line:84%
So in the BERT model, which
is where it came from,

00:51:30.230 --> 00:51:31.370 align:middle line:90%
in the text encoding.

00:51:31.370 --> 00:51:32.870 align:middle line:84%
In the BERT model,
I think we talked

00:51:32.870 --> 00:51:35.550 align:middle line:84%
about when it sees a
word it doesn't know,

00:51:35.550 --> 00:51:39.270 align:middle line:84%
it's never seen before,
it can use the context

00:51:39.270 --> 00:51:40.692 align:middle line:90%
words around it to try to--

00:51:40.692 --> 00:51:41.650 align:middle line:90%
PROFESSOR: Right track.

00:51:41.650 --> 00:51:45.470 align:middle line:84%
But here, just to
be clear, I want

00:51:45.470 --> 00:51:47.990 align:middle line:84%
you to use CLIP
that we just built.

00:51:47.990 --> 00:51:51.030 align:middle line:84%
And assume CLIP
knows all the words

00:51:51.030 --> 00:51:52.993 align:middle line:84%
because it's been trained
on a big vocabulary.

00:51:52.993 --> 00:51:54.410 align:middle line:84%
You can give it
any text you want.

00:51:54.410 --> 00:51:56.870 align:middle line:84%
It will create an
embedding from it.

00:51:56.870 --> 00:51:58.290 align:middle line:90%
That's the key capability.

00:51:58.290 --> 00:52:02.230 align:middle line:90%


00:52:02.230 --> 00:52:05.843 align:middle line:84%
STUDENT: So it creates a text
embedding for a text compute.

00:52:05.843 --> 00:52:06.510 align:middle line:90%
PROFESSOR: Yeah.

00:52:06.510 --> 00:52:08.468 align:middle line:84%
STUDENT: But because
there are things, and then

00:52:08.468 --> 00:52:11.190 align:middle line:90%
for image [INAUDIBLE].

00:52:11.190 --> 00:52:14.790 align:middle line:84%
So for comparing the similarity
scores between the two,

00:52:14.790 --> 00:52:16.850 align:middle line:84%
image is complete but
the text is not complete.

00:52:16.850 --> 00:52:18.210 align:middle line:90%
So there will be missing spaces.

00:52:18.210 --> 00:52:20.970 align:middle line:84%
And then, it makes a
prediction using this method.

00:52:20.970 --> 00:52:24.060 align:middle line:84%
PROFESSOR: But why is there
a missing piece in the text?

00:52:24.060 --> 00:52:28.420 align:middle line:84%
STUDENT: Because
the image, the text,

00:52:28.420 --> 00:52:31.780 align:middle line:84%
the text does not
contain the class.

00:52:31.780 --> 00:52:35.140 align:middle line:84%
And then, well, for the
image, the way it was trained,

00:52:35.140 --> 00:52:38.280 align:middle line:84%
it was trained with pairs
with class included.

00:52:38.280 --> 00:52:41.040 align:middle line:84%
PROFESSOR: Right, but we
actually know the class now.

00:52:41.040 --> 00:52:44.620 align:middle line:84%
So the use case is that I come
to you with an image and I say,

00:52:44.620 --> 00:52:48.060 align:middle line:84%
here are the seven possible
labels for this image.

00:52:48.060 --> 00:52:51.500 align:middle line:84%
And each label is
a piece of text.

00:52:51.500 --> 00:52:55.180 align:middle line:84%
So you actually have seven
pieces of text and an image.

00:52:55.180 --> 00:52:58.920 align:middle line:84%
And all I want CLIP to do is
to tell me, OK, the seventh,

00:52:58.920 --> 00:53:03.310 align:middle line:84%
the fourth label is the
right one for this image,

00:53:03.310 --> 00:53:04.560 align:middle line:90%
but you're on the right track.

00:53:04.560 --> 00:53:07.780 align:middle line:90%


00:53:07.780 --> 00:53:10.280 align:middle line:84%
Once you see how it's done,
you'll be like, yeah, of course.

00:53:10.280 --> 00:53:13.123 align:middle line:90%


00:53:13.123 --> 00:53:15.040 align:middle line:84%
STUDENT: I may not be
understanding something,

00:53:15.040 --> 00:53:16.998 align:middle line:84%
but wouldn't you just
pick the embedding that's

00:53:16.998 --> 00:53:19.772 align:middle line:84%
the closest to the, like,
the text embedding that's

00:53:19.772 --> 00:53:21.230 align:middle line:84%
the closest to the
image embedding?

00:53:21.230 --> 00:53:22.170 align:middle line:90%
PROFESSOR: Correct.

00:53:22.170 --> 00:53:23.070 align:middle line:90%
You're not missing anything.

00:53:23.070 --> 00:53:24.130 align:middle line:90%
That's the right answer.

00:53:24.130 --> 00:53:25.250 align:middle line:90%
Well done.

00:53:25.250 --> 00:53:26.910 align:middle line:90%
[LAUGHS] Come on, people.

00:53:26.910 --> 00:53:29.030 align:middle line:90%
Can we applaud our fellow peer?

00:53:29.030 --> 00:53:30.210 align:middle line:90%
[APPLAUSE]

00:53:30.210 --> 00:53:32.810 align:middle line:90%
You folks are hard to impress.

00:53:32.810 --> 00:53:34.170 align:middle line:90%
That's exactly what we do.

00:53:34.170 --> 00:53:40.073 align:middle line:84%
So here, the key
thing to remember,

00:53:40.073 --> 00:53:41.490 align:middle line:84%
the key thing to
keep in your head

00:53:41.490 --> 00:53:47.310 align:middle line:84%
is that when a label is just
text, dog, cat, it's just text.

00:53:47.310 --> 00:53:50.890 align:middle line:84%
So you can just imagine taking
each label, which in this case

00:53:50.890 --> 00:53:53.770 align:middle line:90%
is plane, car, dog, whatever.

00:53:53.770 --> 00:53:55.930 align:middle line:84%
For each one of them,
you create an embedding.

00:53:55.930 --> 00:53:58.690 align:middle line:84%
You get T1 through whatever,
if you have n labels.

00:53:58.690 --> 00:54:01.047 align:middle line:84%
For the image, you just
have one embedding, I.

00:54:01.047 --> 00:54:02.630 align:middle line:84%
And then, you just
create the cosines,

00:54:02.630 --> 00:54:04.050 align:middle line:90%
calculate the cosine similarity.

00:54:04.050 --> 00:54:06.790 align:middle line:84%
And whichever is the highest
number, you say, OK, it's a dog.

00:54:06.790 --> 00:54:09.250 align:middle line:90%
That's it.

00:54:09.250 --> 00:54:12.070 align:middle line:84%
It's super, just imagine the
level of flexibility here.

00:54:12.070 --> 00:54:15.250 align:middle line:90%


00:54:15.250 --> 00:54:19.313 align:middle line:84%
So that's a side use of CLIP
unrelated to diffusion models,

00:54:19.313 --> 00:54:20.980 align:middle line:84%
but I just thought
it was really clever,

00:54:20.980 --> 00:54:22.063 align:middle line:90%
so I wanted to share that.

00:54:22.063 --> 00:54:22.903 align:middle line:90%
OK, good.

00:54:22.903 --> 00:54:24.320 align:middle line:84%
Now, let's see how
we can actually

00:54:24.320 --> 00:54:28.120 align:middle line:84%
use this entire capability to go
to solve the original problem we

00:54:28.120 --> 00:54:31.000 align:middle line:84%
set out to solve, which is
can we steer the diffusion

00:54:31.000 --> 00:54:33.840 align:middle line:84%
model to create an image based
on a particular prompt we give

00:54:33.840 --> 00:54:35.240 align:middle line:90%
it?

00:54:35.240 --> 00:54:38.520 align:middle line:84%
So now, remember, if we
go back to how we did it,

00:54:38.520 --> 00:54:40.800 align:middle line:84%
we created all these
training pairs of x and y

00:54:40.800 --> 00:54:44.240 align:middle line:90%
based on denoising the image.

00:54:44.240 --> 00:54:47.320 align:middle line:84%
X is the image, y is the less
noisy version of the image.

00:54:47.320 --> 00:54:51.640 align:middle line:84%
So what we can simply
do is we can actually

00:54:51.640 --> 00:54:54.920 align:middle line:84%
change the input so it
becomes the image, and then

00:54:54.920 --> 00:54:59.287 align:middle line:84%
the clip text embedding of
the caption for that image.

00:54:59.287 --> 00:55:01.120 align:middle line:84%
So you have an image
and you have a caption.

00:55:01.120 --> 00:55:03.440 align:middle line:84%
You take the caption,
run it through CLIP.

00:55:03.440 --> 00:55:04.760 align:middle line:90%
You get an embedding.

00:55:04.760 --> 00:55:07.960 align:middle line:84%
By definition, that
embedding lives

00:55:07.960 --> 00:55:10.160 align:middle line:84%
in the same space as
all the images that

00:55:10.160 --> 00:55:11.293 align:middle line:90%
correspond to that caption.

00:55:11.293 --> 00:55:14.000 align:middle line:90%


00:55:14.000 --> 00:55:17.150 align:middle line:84%
So you just attach, you
concatenate the embedding

00:55:17.150 --> 00:55:20.890 align:middle line:84%
of the CLIP output of a
caption along with the image.

00:55:20.890 --> 00:55:23.230 align:middle line:90%
You make that the new input now.

00:55:23.230 --> 00:55:25.773 align:middle line:84%
Why continues to be the less
noisy version of the image?

00:55:25.773 --> 00:55:28.190 align:middle line:84%
Or as we saw earlier, it could
be just the noise component

00:55:28.190 --> 00:55:29.750 align:middle line:90%
of that image.

00:55:29.750 --> 00:55:33.310 align:middle line:84%
OK, this is the new x,
y pair that we have.

00:55:33.310 --> 00:55:37.310 align:middle line:84%
And so now, the model is you
send the CLIP text embedding,

00:55:37.310 --> 00:55:38.230 align:middle line:90%
the image x.

00:55:38.230 --> 00:55:40.410 align:middle line:84%
Send it through this noisy
version of the image.

00:55:40.410 --> 00:55:42.550 align:middle line:84%
And you keep on
training it for a while.

00:55:42.550 --> 00:55:44.550 align:middle line:84%
Once your model is
trained for when

00:55:44.550 --> 00:55:47.590 align:middle line:84%
you want to use it for
inference, for a new prompt,

00:55:47.590 --> 00:55:51.890 align:middle line:84%
you just give it, Killian Court
at MIT during the spring time,

00:55:51.890 --> 00:55:54.270 align:middle line:90%
along with a bunch of noise.

00:55:54.270 --> 00:55:56.610 align:middle line:90%
Goes in, it starts denoising it.

00:55:56.610 --> 00:55:59.370 align:middle line:84%
But because this
embedding of this thing,

00:55:59.370 --> 00:56:01.830 align:middle line:84%
thanks to CLIP, lives
in the same image space

00:56:01.830 --> 00:56:05.120 align:middle line:84%
as all Killian Court embeddings,
Killian Court images,

00:56:05.120 --> 00:56:06.820 align:middle line:84%
you keep on doing
it for a while.

00:56:06.820 --> 00:56:08.570 align:middle line:84%
At some point, you
will get Killian Court.

00:56:08.570 --> 00:56:11.253 align:middle line:90%


00:56:11.253 --> 00:56:12.170 align:middle line:90%
That's how they do it.

00:56:12.170 --> 00:56:14.620 align:middle line:90%
That's how they steer the image.

00:56:14.620 --> 00:56:15.760 align:middle line:90%
It's a two-step process.

00:56:15.760 --> 00:56:18.220 align:middle line:84%
You create all these
clip embeddings,

00:56:18.220 --> 00:56:20.560 align:middle line:84%
which CLIP was a
breakthrough, in my opinion,

00:56:20.560 --> 00:56:22.842 align:middle line:84%
because it was one of the,
maybe the first example.

00:56:22.842 --> 00:56:24.300 align:middle line:84%
I don't know if
it's the very first

00:56:24.300 --> 00:56:26.580 align:middle line:84%
but one of the early
examples of saying

00:56:26.580 --> 00:56:29.200 align:middle line:84%
we have different kinds
of data, we have images,

00:56:29.200 --> 00:56:31.000 align:middle line:90%
we have captions, we have text.

00:56:31.000 --> 00:56:33.140 align:middle line:84%
How do we create
embeddings for every one

00:56:33.140 --> 00:56:35.020 align:middle line:84%
of these very
different data types

00:56:35.020 --> 00:56:37.820 align:middle line:84%
that all happen to live in the
same space, the same concept

00:56:37.820 --> 00:56:38.320 align:middle line:90%
space?

00:56:38.320 --> 00:56:39.780 align:middle line:90%
That was the key idea.

00:56:39.780 --> 00:56:42.420 align:middle line:84%
And if you look at the modern
multimodal large language

00:56:42.420 --> 00:56:46.260 align:middle line:84%
models, they are all based
on the same exact idea.

00:56:46.260 --> 00:56:49.400 align:middle line:84%
So it's very powerful,
this approach.

00:56:49.400 --> 00:56:50.900 align:middle line:84%
STUDENT: Yeah, now
I understand this

00:56:50.900 --> 00:56:54.700 align:middle line:84%
for images, but for video
generation models like Sora,

00:56:54.700 --> 00:56:58.420 align:middle line:84%
do they have some sort of
underlying physics structure,

00:56:58.420 --> 00:57:00.880 align:middle line:84%
or do they learn the
physical representations?

00:57:00.880 --> 00:57:02.297 align:middle line:84%
PROFESSOR: There's
a lot of debate

00:57:02.297 --> 00:57:03.963 align:middle line:84%
on the internet
about this stuff.

00:57:03.963 --> 00:57:05.380 align:middle line:84%
They haven't
published the results

00:57:05.380 --> 00:57:08.060 align:middle line:84%
of the full technical report
yet, so we don't know for sure.

00:57:08.060 --> 00:57:10.460 align:middle line:84%
But the consensus
seems to be no, they

00:57:10.460 --> 00:57:12.100 align:middle line:90%
are not using a physics engine.

00:57:12.100 --> 00:57:14.902 align:middle line:84%
What they have done, and
again, this may be wrong.

00:57:14.902 --> 00:57:16.860 align:middle line:84%
Once the report comes
out, we'll know for sure.

00:57:16.860 --> 00:57:20.320 align:middle line:84%
But what people are saying,
computer vision experts,

00:57:20.320 --> 00:57:26.180 align:middle line:84%
is that it has been trained
on a lot of video game data,

00:57:26.180 --> 00:57:29.060 align:middle line:84%
along with actual
videos and so on.

00:57:29.060 --> 00:57:31.460 align:middle line:84%
And the corpus of
training is so massive

00:57:31.460 --> 00:57:34.900 align:middle line:84%
that it has basically learned
to mimic certain physics aspects

00:57:34.900 --> 00:57:36.600 align:middle line:90%
to it just as a side effect.

00:57:36.600 --> 00:57:39.720 align:middle line:84%
Much like LLMs, you train them
on a large amount of text data,

00:57:39.720 --> 00:57:42.860 align:middle line:84%
they begin to start to do things
which you didn't anticipate

00:57:42.860 --> 00:57:44.560 align:middle line:90%
that they'll do.

00:57:44.560 --> 00:57:46.320 align:middle line:84%
So for example, I
read this, I thought

00:57:46.320 --> 00:57:49.460 align:middle line:84%
it was a really great
example of what is surprising

00:57:49.460 --> 00:57:52.900 align:middle line:84%
about large language models
is not that you train them

00:57:52.900 --> 00:57:54.593 align:middle line:84%
on a bunch of high
school math problems,

00:57:54.593 --> 00:57:56.760 align:middle line:84%
and then you give it a new
high school math problem.

00:57:56.760 --> 00:57:57.802 align:middle line:90%
It can actually solve it.

00:57:57.802 --> 00:57:58.825 align:middle line:90%
That's not surprising.

00:57:58.825 --> 00:58:00.700 align:middle line:84%
You give it a whole
bunch of high school math

00:58:00.700 --> 00:58:02.900 align:middle line:84%
problems in English,
then you ask

00:58:02.900 --> 00:58:04.867 align:middle line:84%
it to read a bunch
of French literature,

00:58:04.867 --> 00:58:06.700 align:middle line:84%
and then you give it
French high school math

00:58:06.700 --> 00:58:08.040 align:middle line:90%
problems to solve it.

00:58:08.040 --> 00:58:10.010 align:middle line:90%
That is the new news.

00:58:10.010 --> 00:58:13.090 align:middle line:84%
So similarly here, I
think the expectation

00:58:13.090 --> 00:58:15.757 align:middle line:84%
is that it's not actually using
a physics engine under the hood.

00:58:15.757 --> 00:58:17.965 align:middle line:84%
It may have used a physics
engine to actually come up

00:58:17.965 --> 00:58:19.690 align:middle line:84%
with the videos and
renderings, but there

00:58:19.690 --> 00:58:22.440 align:middle line:84%
are no physics constraints
in the model itself.

00:58:22.440 --> 00:58:24.190 align:middle line:84%
It just comes out of
the training process.

00:58:24.190 --> 00:58:25.690 align:middle line:90%
That's the current view.

00:58:25.690 --> 00:58:27.190 align:middle line:84%
Once the technical
report comes out,

00:58:27.190 --> 00:58:29.230 align:middle line:84%
we'll know for sure
what they actually did.

00:58:29.230 --> 00:58:33.730 align:middle line:90%


00:58:33.730 --> 00:58:36.030 align:middle line:84%
STUDENT: So one quick
question about stability.

00:58:36.030 --> 00:58:38.930 align:middle line:84%
It's claiming to be a little
bit more real time in our image

00:58:38.930 --> 00:58:40.048 align:middle line:90%
generation.

00:58:40.048 --> 00:58:41.590 align:middle line:84%
PROFESSOR: You mean
Stable Diffusion?

00:58:41.590 --> 00:58:41.890 align:middle line:90%
STUDENT: Yeah.

00:58:41.890 --> 00:58:42.170 align:middle line:90%
PROFESSOR: Yeah, yeah.

00:58:42.170 --> 00:58:42.990 align:middle line:90%
STUDENT: For Stable Diffusion.

00:58:42.990 --> 00:58:45.130 align:middle line:84%
So are they jumping through
the noise more quickly,

00:58:45.130 --> 00:58:47.590 align:middle line:84%
or are they kind of preprompting
it, and kind of trick--

00:58:47.590 --> 00:58:49.390 align:middle line:84%
PROFESSOR: Very good question,
and there's a very key trick.

00:58:49.390 --> 00:58:50.450 align:middle line:90%
It's coming.

00:58:50.450 --> 00:58:52.770 align:middle line:90%
You had a question.

00:58:52.770 --> 00:58:54.890 align:middle line:84%
STUDENT: So here, the
example of the noise

00:58:54.890 --> 00:58:55.943 align:middle line:90%
is normal distribution.

00:58:55.943 --> 00:58:56.610 align:middle line:90%
PROFESSOR: Yeah.

00:58:56.610 --> 00:59:00.090 align:middle line:84%
STUDENT: However, if we have
changed the noise distribution,

00:59:00.090 --> 00:59:01.215 align:middle line:90%
would it change the result?

00:59:01.215 --> 00:59:02.882 align:middle line:84%
PROFESSOR: Oh, you
mean if you change it

00:59:02.882 --> 00:59:04.770 align:middle line:84%
to a Poisson or some
other distribution?

00:59:04.770 --> 00:59:06.353 align:middle line:84%
It will definitely
change the results.

00:59:06.353 --> 00:59:08.440 align:middle line:84%
Because if you look
at the underlying

00:59:08.440 --> 00:59:10.680 align:middle line:84%
math of why this
works, it heavily

00:59:10.680 --> 00:59:13.280 align:middle line:84%
depends on the
Gaussian assumption.

00:59:13.280 --> 00:59:14.800 align:middle line:90%
Yeah.

00:59:14.800 --> 00:59:18.260 align:middle line:84%
There was another
question somewhere here.

00:59:18.260 --> 00:59:19.760 align:middle line:84%
STUDENT: You may
not know the answer

00:59:19.760 --> 00:59:21.220 align:middle line:84%
because the technical
report is not out,

00:59:21.220 --> 00:59:23.080 align:middle line:84%
but could it be in terms
of video generation

00:59:23.080 --> 00:59:26.960 align:middle line:84%
that's analogous to going
from one noisy image

00:59:26.960 --> 00:59:30.600 align:middle line:84%
to another, like you're almost
doing a series of still images

00:59:30.600 --> 00:59:31.620 align:middle line:90%
and learning how to--

00:59:31.620 --> 00:59:33.520 align:middle line:84%
PROFESSOR: No, that,
I think, people

00:59:33.520 --> 00:59:34.900 align:middle line:90%
are sure is how it's done.

00:59:34.900 --> 00:59:36.400 align:middle line:84%
So basically, you
think of the video

00:59:36.400 --> 00:59:40.080 align:middle line:84%
as just a series of frames,
and each frame is an image.

00:59:40.080 --> 00:59:42.460 align:middle line:84%
And there is a
sequentiality to it,

00:59:42.460 --> 00:59:44.960 align:middle line:84%
which is where the transformer
stack will come in because it

00:59:44.960 --> 00:59:46.400 align:middle line:90%
handles sequentiality.

00:59:46.400 --> 00:59:49.640 align:middle line:84%
So in general, video
stuff typically

00:59:49.640 --> 00:59:53.040 align:middle line:84%
operates on frame by frame,
which is just an image.

00:59:53.040 --> 00:59:54.260 align:middle line:90%
So that is definitely there.

00:59:54.260 --> 00:59:56.880 align:middle line:84%
What we don't know
is if they also

00:59:56.880 --> 00:59:59.080 align:middle line:84%
used some understanding
of the fact

00:59:59.080 --> 01:00:02.020 align:middle line:84%
that, for example, that
if an object is dropped,

01:00:02.020 --> 01:00:04.800 align:middle line:84%
it has to fall to the
Earth in a certain rate.

01:00:04.800 --> 01:00:07.250 align:middle line:84%
Or if an object goes
behind another object,

01:00:07.250 --> 01:00:08.870 align:middle line:84%
you can't see the
object anymore.

01:00:08.870 --> 01:00:12.070 align:middle line:84%
Things like that, which
we take for granted.

01:00:12.070 --> 01:00:13.730 align:middle line:84%
The question is,
are they using it?

01:00:13.730 --> 01:00:16.368 align:middle line:84%
And the consensus seems
to be, in the absence

01:00:16.368 --> 01:00:18.910 align:middle line:84%
of an actual technical report,
that no, they're not doing it.

01:00:18.910 --> 01:00:20.785 align:middle line:84%
Because there are lots
of examples on Twitter

01:00:20.785 --> 01:00:23.470 align:middle line:84%
where people will show
a Sora video in which

01:00:23.470 --> 01:00:25.350 align:middle line:84%
it's not obeying
the laws of physics.

01:00:25.350 --> 01:00:28.370 align:middle line:84%
So you take like a beach chair,
and then put it in the sand.

01:00:28.370 --> 01:00:31.807 align:middle line:84%
You see the sand come through
the base of the beach chair.

01:00:31.807 --> 01:00:33.890 align:middle line:84%
Or you take an object and
put it behind an object,

01:00:33.890 --> 01:00:35.550 align:middle line:84%
you can still see the object,
even though the original object

01:00:35.550 --> 01:00:36.438 align:middle line:90%
is opaque.

01:00:36.438 --> 01:00:38.730 align:middle line:84%
So you'll be seeing some
evidence that no, no, it's not

01:00:38.730 --> 01:00:39.610 align:middle line:90%
obeying the laws of physics.

01:00:39.610 --> 01:00:41.270 align:middle line:84%
What you're seeing
is just an amazing--

01:00:41.270 --> 01:00:45.010 align:middle line:84%
you just learn to fake the laws
of physics in some context.

01:00:45.010 --> 01:00:46.510 align:middle line:84%
Much like you can
fake five fingers

01:00:46.510 --> 01:00:48.677 align:middle line:84%
without knowing that it has
to be only five fingers.

01:00:48.677 --> 01:00:51.470 align:middle line:90%


01:00:51.470 --> 01:00:52.590 align:middle line:90%
OK.

01:00:52.590 --> 01:00:53.330 align:middle line:90%
All right.

01:00:53.330 --> 01:00:55.590 align:middle line:90%
So let's keep going.

01:00:55.590 --> 01:00:59.928 align:middle line:84%
Now, so there was
another paper afterwards,

01:00:59.928 --> 01:01:01.470 align:middle line:84%
and this is the
original paper, which

01:01:01.470 --> 01:01:04.070 align:middle line:84%
took that idea of
the diffusion model.

01:01:04.070 --> 01:01:06.400 align:middle line:84%
And then, diffusion models
are very slow, as Olivia,

01:01:06.400 --> 01:01:07.422 align:middle line:90%
you pointed out.

01:01:07.422 --> 01:01:09.380 align:middle line:84%
So the question is, can
we make it much faster?

01:01:09.380 --> 01:01:11.023 align:middle line:84%
So what they did,
and I'm not going

01:01:11.023 --> 01:01:12.440 align:middle line:84%
to get into this
whole thing here.

01:01:12.440 --> 01:01:15.060 align:middle line:84%
I just want to highlight
a couple of things.

01:01:15.060 --> 01:01:18.720 align:middle line:84%
The first one is
that, first of all,

01:01:18.720 --> 01:01:21.240 align:middle line:90%
notice that you see U-Net here.

01:01:21.240 --> 01:01:25.740 align:middle line:84%
So they are using a U-Net,
to go from image-to-image.

01:01:25.740 --> 01:01:29.020 align:middle line:84%
The second thing is that the
CLIP embedding of the text

01:01:29.020 --> 01:01:32.140 align:middle line:84%
prompt is basically,
is woven, meaning

01:01:32.140 --> 01:01:36.420 align:middle line:84%
it's incorporated into the
U-Net through an attention

01:01:36.420 --> 01:01:38.440 align:middle line:84%
mechanism, a
transformer mechanism.

01:01:38.440 --> 01:01:40.820 align:middle line:84%
And you can see the
QKV business here,

01:01:40.820 --> 01:01:42.700 align:middle line:84%
which should be
familiar at this point.

01:01:42.700 --> 01:01:44.660 align:middle line:84%
So it is integrated
into the transformer

01:01:44.660 --> 01:01:47.940 align:middle line:84%
stack directly that input the
CLIP embedding, the second thing

01:01:47.940 --> 01:01:48.880 align:middle line:90%
I want to point out.

01:01:48.880 --> 01:01:52.620 align:middle line:84%
And then, thirdly, and this
is where the speedup comes.

01:01:52.620 --> 01:01:55.040 align:middle line:84%
So what you do is instead
of taking the image,

01:01:55.040 --> 01:01:56.540 align:middle line:84%
running it through
the whole network

01:01:56.540 --> 01:01:58.900 align:middle line:84%
and creating a slightly less
noisy version of the image,

01:01:58.900 --> 01:02:01.300 align:middle line:84%
here, what you do is
you take the image,

01:02:01.300 --> 01:02:04.130 align:middle line:84%
you run it through an image
encoder, you get an embedding.

01:02:04.130 --> 01:02:06.610 align:middle line:84%
And now, you only work
with the embedding.

01:02:06.610 --> 01:02:09.090 align:middle line:84%
You take the embedding and
create a slightly less noisy

01:02:09.090 --> 01:02:09.990 align:middle line:90%
version embedding.

01:02:09.990 --> 01:02:10.698 align:middle line:90%
Keep on doing it.

01:02:10.698 --> 01:02:13.240 align:middle line:84%
And these embeddings are much
smaller than images, therefore,

01:02:13.240 --> 01:02:14.750 align:middle line:90%
they are much faster to process.

01:02:14.750 --> 01:02:17.090 align:middle line:84%
And once you've done it
1,000 times, you get a very,

01:02:17.090 --> 01:02:20.150 align:middle line:84%
sort of, almost pure noiseless
version of the embedding.

01:02:20.150 --> 01:02:24.050 align:middle line:84%
Now, you run it through an image
decoder to get the image out.

01:02:24.050 --> 01:02:30.623 align:middle line:84%
So the idea here is that you
operate in the latent space,

01:02:30.623 --> 01:02:31.790 align:middle line:90%
meaning the embedding space.

01:02:31.790 --> 01:02:34.413 align:middle line:84%
And hence, it's called the
latent diffusion model.

01:02:34.413 --> 01:02:35.830 align:middle line:84%
So that's where
the speedup comes.

01:02:35.830 --> 01:02:38.130 align:middle line:84%
But research continues
to be very strong,

01:02:38.130 --> 01:02:39.138 align:middle line:90%
to make it even faster.

01:02:39.138 --> 01:02:40.930 align:middle line:84%
Because for a lot of
consumer applications,

01:02:40.930 --> 01:02:42.890 align:middle line:84%
people are obviously not
going to wait around for,

01:02:42.890 --> 01:02:45.010 align:middle line:84%
I mean, who wants to wait
for ten seconds, right?

01:02:45.010 --> 01:02:47.860 align:middle line:84%
And so there's a lot of
pressure to make it even faster.

01:02:47.860 --> 01:02:52.170 align:middle line:90%


01:02:52.170 --> 01:02:52.710 align:middle line:90%
All right.

01:02:52.710 --> 01:02:53.790 align:middle line:90%
So that's what we have.

01:02:53.790 --> 01:02:57.730 align:middle line:84%
Obviously, these models are
transforming everything.

01:02:57.730 --> 01:03:00.210 align:middle line:84%
And by the way, this
site here, lexica.art,

01:03:00.210 --> 01:03:01.480 align:middle line:90%
you can go check it out.

01:03:01.480 --> 01:03:03.760 align:middle line:84%
It has a whole bunch of
very interesting images

01:03:03.760 --> 01:03:05.680 align:middle line:84%
and prompts that
created the images.

01:03:05.680 --> 01:03:07.380 align:middle line:84%
So if you're working
in this space,

01:03:07.380 --> 01:03:09.060 align:middle line:84%
it gives you a lot
of interesting ideas.

01:03:09.060 --> 01:03:12.640 align:middle line:84%
But it's not just for
consumer fun applications.

01:03:12.640 --> 01:03:16.580 align:middle line:84%
These models are being used to
actually-- you know, AlphaFold,

01:03:16.580 --> 01:03:19.300 align:middle line:84%
if you'll recall, if you give
it an amino acid sequence,

01:03:19.300 --> 01:03:21.540 align:middle line:84%
it actually create
the 3D structure.

01:03:21.540 --> 01:03:25.720 align:middle line:84%
So that's an example of, I
don't think they use a diffusion

01:03:25.720 --> 01:03:28.560 align:middle line:84%
model, but you can imagine using
a diffusion model to create

01:03:28.560 --> 01:03:30.800 align:middle line:90%
these complicated objects.

01:03:30.800 --> 01:03:34.760 align:middle line:84%
Meaning, the objects you
create don't have to be images.

01:03:34.760 --> 01:03:37.360 align:middle line:84%
They can be arbitrarily
complicated things.

01:03:37.360 --> 01:03:39.720 align:middle line:84%
As long as you have enough
data about such things

01:03:39.720 --> 01:03:43.440 align:middle line:84%
to use for training, and the
notion of noising the input

01:03:43.440 --> 01:03:45.600 align:middle line:84%
is meaningful, you can
create some very interesting

01:03:45.600 --> 01:03:46.460 align:middle line:90%
structures.

01:03:46.460 --> 01:03:49.623 align:middle line:84%
You can create 3D things
and protein structures.

01:03:49.623 --> 01:03:52.040 align:middle line:84%
And there's a whole bunch of
very interesting applications

01:03:52.040 --> 01:03:54.320 align:middle line:90%
in biomedical sciences.

01:03:54.320 --> 01:03:56.820 align:middle line:84%
So this is really just
the tip of the iceberg.

01:03:56.820 --> 01:03:59.390 align:middle line:84%
And now, there are
these things, there

01:03:59.390 --> 01:04:02.310 align:middle line:84%
are ways in which you
can use diffusion models

01:04:02.310 --> 01:04:04.510 align:middle line:84%
to do large language
modeling as well.

01:04:04.510 --> 01:04:07.350 align:middle line:84%
So there's a lot of overlap
and blending and so on going on

01:04:07.350 --> 01:04:08.430 align:middle line:90%
in the space.

01:04:08.430 --> 01:04:10.733 align:middle line:90%
So I'm going to do a quick demo.

01:04:10.733 --> 01:04:12.150 align:middle line:84%
If you look at
Hugging Face, there

01:04:12.150 --> 01:04:14.150 align:middle line:84%
is something called the
diffusers library, which

01:04:14.150 --> 01:04:16.550 align:middle line:84%
is like, as the
name suggests, it's

01:04:16.550 --> 01:04:19.950 align:middle line:84%
a library for a lot
of diffusion models.

01:04:19.950 --> 01:04:22.010 align:middle line:90%
And let's take a quick look.

01:04:22.010 --> 01:04:25.790 align:middle line:90%


01:04:25.790 --> 01:04:27.827 align:middle line:84%
All right, so the
diffusers library

01:04:27.827 --> 01:04:29.410 align:middle line:84%
has a whole bunch
of diffusion models.

01:04:29.410 --> 01:04:31.577 align:middle line:84%
We are going to work with
Stable Diffusion, which is

01:04:31.577 --> 01:04:34.270 align:middle line:90%
one of the better known models.

01:04:34.270 --> 01:04:36.490 align:middle line:90%
So let's install diffusers.

01:04:36.490 --> 01:04:39.670 align:middle line:90%


01:04:39.670 --> 01:04:42.030 align:middle line:84%
You will recall, when I
did the quick lightning

01:04:42.030 --> 01:04:45.310 align:middle line:84%
tour of the Hugging Face
ecosystem for language,

01:04:45.310 --> 01:04:49.070 align:middle line:84%
Hugging Face has a whole bunch
of capabilities built out

01:04:49.070 --> 01:04:50.010 align:middle line:90%
of the box.

01:04:50.010 --> 01:04:52.550 align:middle line:84%
And you use this thing called
the pipeline function to very

01:04:52.550 --> 01:04:54.670 align:middle line:90%
quickly use any model you want.

01:04:54.670 --> 01:04:56.890 align:middle line:84%
The same exact
philosophy applies here.

01:04:56.890 --> 01:04:58.740 align:middle line:90%
You still use the pipeline.

01:04:58.740 --> 01:05:00.730 align:middle line:84%
So I'm going to import
a bunch of stuff.

01:05:00.730 --> 01:05:09.340 align:middle line:90%


01:05:09.340 --> 01:05:10.260 align:middle line:90%
All right, so.

01:05:10.260 --> 01:05:11.520 align:middle line:90%
Oh, I see, I to do this thing.

01:05:11.520 --> 01:05:12.020 align:middle line:90%
OK.

01:05:12.020 --> 01:05:15.980 align:middle line:90%


01:05:15.980 --> 01:05:16.740 align:middle line:90%
Great.

01:05:16.740 --> 01:05:17.430 align:middle line:90%
Hugging Face.

01:05:17.430 --> 01:05:23.020 align:middle line:90%


01:05:23.020 --> 01:05:24.880 align:middle line:84%
All right, now
that we have here.

01:05:24.880 --> 01:05:27.760 align:middle line:84%
So you remember that
when we worked with text,

01:05:27.760 --> 01:05:31.100 align:middle line:84%
we would grab a pretrained
model and then we actually

01:05:31.100 --> 01:05:32.400 align:middle line:90%
run it through a pipeline.

01:05:32.400 --> 01:05:34.560 align:middle line:84%
And we can do all the
inference we want on it.

01:05:34.560 --> 01:05:38.520 align:middle line:84%
The same exact
philosophy applies here.

01:05:38.520 --> 01:05:42.500 align:middle line:84%
And this is very similar to what
we did in lecture 8 for NLP.

01:05:42.500 --> 01:05:44.260 align:middle line:84%
So what we're going
to do is, we use

01:05:44.260 --> 01:05:46.380 align:middle line:84%
this command, the
StableDiffusionPipeline

01:05:46.380 --> 01:05:47.440 align:middle line:90%
from pretrained.

01:05:47.440 --> 01:05:51.140 align:middle line:84%
And we use this version,
1.4 Stable Diffusion model.

01:05:51.140 --> 01:05:52.820 align:middle line:84%
So let's just
create the pipeline.

01:05:52.820 --> 01:05:56.450 align:middle line:90%


01:05:56.450 --> 01:05:58.950 align:middle line:84%
And obviously, we have used
TensorFlow, not PyTorch here,

01:05:58.950 --> 01:06:01.370 align:middle line:84%
but a lot of these
models, unfortunately,

01:06:01.370 --> 01:06:02.750 align:middle line:90%
happen to be in PyTorch.

01:06:02.750 --> 01:06:05.170 align:middle line:84%
So knowing a little
bit of PyTorch

01:06:05.170 --> 01:06:08.450 align:middle line:84%
is actually very helpful to be
able to work with these things.

01:06:08.450 --> 01:06:12.170 align:middle line:84%
And what we're doing here,
while it's downloading,

01:06:12.170 --> 01:06:18.710 align:middle line:84%
we are using this FP16 storage
format for the model weights,

01:06:18.710 --> 01:06:20.970 align:middle line:84%
because it's going to be a
little smaller than using

01:06:20.970 --> 01:06:22.050 align:middle line:90%
32 bits.

01:06:22.050 --> 01:06:23.617 align:middle line:90%
So it will download faster.

01:06:23.617 --> 01:06:24.950 align:middle line:90%
So that's what's happening here.

01:06:24.950 --> 01:06:25.533 align:middle line:90%
So, all right.

01:06:25.533 --> 01:06:27.390 align:middle line:90%
It's downloaded fine.

01:06:27.390 --> 01:06:28.870 align:middle line:84%
So now, we just
give it a prompt.

01:06:28.870 --> 01:06:31.610 align:middle line:84%
And this is actually one
of the original famous meme

01:06:31.610 --> 01:06:35.050 align:middle line:84%
prompts, a photograph of an
astronaut riding a horse.

01:06:35.050 --> 01:06:37.810 align:middle line:84%
And so once we have
the pipeline set up,

01:06:37.810 --> 01:06:39.550 align:middle line:84%
I'll just set a seed
for reproducibility.

01:06:39.550 --> 01:06:42.250 align:middle line:84%
And then, literally,
I do pipe a prompt.

01:06:42.250 --> 01:06:46.410 align:middle line:84%
And then, it's actually,
you can see here, 50.

01:06:46.410 --> 01:06:49.050 align:middle line:84%
So it's going through
50 denoising steps.

01:06:49.050 --> 01:06:50.410 align:middle line:90%
OK.

01:06:50.410 --> 01:06:52.865 align:middle line:84%
And you come up with an
astronaut riding a horse.

01:06:52.865 --> 01:06:54.640 align:middle line:90%
So that's that.

01:06:54.640 --> 01:06:56.340 align:middle line:84%
You can actually
change the seed,

01:06:56.340 --> 01:06:59.000 align:middle line:90%
and you can get a different--

01:06:59.000 --> 01:07:01.600 align:middle line:84%
the seed is basically, sets
the random starting point

01:07:01.600 --> 01:07:02.460 align:middle line:90%
for the image.

01:07:02.460 --> 01:07:04.760 align:middle line:84%
So therefore, you would
expect a different astronaut.

01:07:04.760 --> 01:07:05.360 align:middle line:90%
Yep.

01:07:05.360 --> 01:07:07.400 align:middle line:84%
This is an astronaut
riding another horse.

01:07:07.400 --> 01:07:10.480 align:middle line:84%
So I think people came up with
these kinds of fun examples

01:07:10.480 --> 01:07:14.105 align:middle line:84%
because it's guaranteed not
to be in the training data.

01:07:14.105 --> 01:07:16.280 align:middle line:84%
So whatever the
model is doing, it's

01:07:16.280 --> 01:07:18.760 align:middle line:84%
not regurgitating what
it has already seen.

01:07:18.760 --> 01:07:19.572 align:middle line:90%
All right.

01:07:19.572 --> 01:07:20.280 align:middle line:90%
Give me a prompt.

01:07:20.280 --> 01:07:26.546 align:middle line:90%


01:07:26.546 --> 01:07:27.330 align:middle line:90%
Prompts, anyone?

01:07:27.330 --> 01:07:29.840 align:middle line:90%


01:07:29.840 --> 01:07:30.880 align:middle line:90%
Wow.

01:07:30.880 --> 01:07:32.680 align:middle line:84%
STUDENT: MIT professor
riding a horse.

01:07:32.680 --> 01:07:34.490 align:middle line:90%
PROFESSOR: [LAUGHS] Ooh.

01:07:34.490 --> 01:07:34.990 align:middle line:90%
OK.

01:07:34.990 --> 01:07:38.600 align:middle line:90%


01:07:38.600 --> 01:07:40.306 align:middle line:90%
That might be a--

01:07:40.306 --> 01:07:43.859 align:middle line:90%
all right, riding a horse.

01:07:43.859 --> 01:07:48.790 align:middle line:90%
[LAUGHS]

01:07:48.790 --> 01:07:49.290 align:middle line:90%
All right.

01:07:49.290 --> 01:07:54.750 align:middle line:90%


01:07:54.750 --> 01:07:56.630 align:middle line:90%
[LAUGHS]

01:07:56.630 --> 01:07:58.110 align:middle line:90%
There are two of them.

01:07:58.110 --> 01:08:01.696 align:middle line:84%
And clearly, MIT professors
don't have, really--

01:08:01.696 --> 01:08:03.510 align:middle line:90%
[LAUGHS]

01:08:03.510 --> 01:08:05.030 align:middle line:90%
Yeah.

01:08:05.030 --> 01:08:07.270 align:middle line:90%
Moving on.

01:08:07.270 --> 01:08:10.670 align:middle line:84%
So by the way, you
should spend some time

01:08:10.670 --> 01:08:11.890 align:middle line:90%
with the diffusers library.

01:08:11.890 --> 01:08:14.070 align:middle line:84%
They have a bunch of tutorials,
which are really interesting.

01:08:14.070 --> 01:08:16.229 align:middle line:84%
Because this core capability
of giving a prompt

01:08:16.229 --> 01:08:18.189 align:middle line:84%
and getting an image
out can actually

01:08:18.189 --> 01:08:21.649 align:middle line:84%
be manipulated for all sorts
of very interesting use cases.

01:08:21.649 --> 01:08:24.670 align:middle line:84%
So for example, there is this
thing called negative prompting.

01:08:24.670 --> 01:08:26.470 align:middle line:84%
And the idea of
negative prompting

01:08:26.470 --> 01:08:29.510 align:middle line:84%
is that you can
give it two prompts

01:08:29.510 --> 01:08:33.069 align:middle line:84%
and say, create an image which
embodies the first prompt

01:08:33.069 --> 01:08:34.790 align:middle line:90%
but not the second prompt.

01:08:34.790 --> 01:08:37.689 align:middle line:84%
Essentially, subtract the second
prompt from the first one.

01:08:37.689 --> 01:08:39.069 align:middle line:84%
That's called
negative prompting.

01:08:39.069 --> 01:08:41.470 align:middle line:84%
And you might be wondering,
what use is that?

01:08:41.470 --> 01:08:43.350 align:middle line:90%
There are lots of fun uses.

01:08:43.350 --> 01:08:46.550 align:middle line:84%
So here, the prompt
is going to be,

01:08:46.550 --> 01:08:48.430 align:middle line:84%
Labrador in the
style of Vermeer.

01:08:48.430 --> 01:08:51.100 align:middle line:90%
That's the first prompt.

01:08:51.100 --> 01:08:53.620 align:middle line:90%
50 steps.

01:08:53.620 --> 01:08:54.500 align:middle line:90%
Look at that.

01:08:54.500 --> 01:08:56.819 align:middle line:90%
Amazing, right?

01:08:56.819 --> 01:09:00.540 align:middle line:84%
But maybe you don't
care for the blue scarf.

01:09:00.540 --> 01:09:02.899 align:middle line:84%
So you basically give
it a negative prompt.

01:09:02.899 --> 01:09:05.500 align:middle line:84%
And you basically, the
negative prompt is blue,

01:09:05.500 --> 01:09:08.020 align:middle line:84%
meaning remove
everything that's blue.

01:09:08.020 --> 01:09:09.000 align:middle line:90%
I don't like this.

01:09:09.000 --> 01:09:11.140 align:middle line:84%
Otherwise, keep the
Labrador thing going.

01:09:11.140 --> 01:09:12.479 align:middle line:90%
So you run it.

01:09:12.479 --> 01:09:16.420 align:middle line:90%


01:09:16.420 --> 01:09:17.240 align:middle line:90%
Look at that.

01:09:17.240 --> 01:09:18.080 align:middle line:90%
The blue is gone.

01:09:18.080 --> 01:09:19.460 align:middle line:90%
Negative prompting.

01:09:19.460 --> 01:09:19.979 align:middle line:90%
OK.

01:09:19.979 --> 01:09:22.380 align:middle line:90%
Yeah?

01:09:22.380 --> 01:09:26.859 align:middle line:84%
STUDENT: If you change
that from 50 to 1,000,

01:09:26.859 --> 01:09:30.140 align:middle line:84%
will it become less pixelated
or will it eventually just keep

01:09:30.140 --> 01:09:31.380 align:middle line:90%
going and iterating?

01:09:31.380 --> 01:09:33.040 align:middle line:84%
PROFESSOR: No, typically, if
you do more of these things,

01:09:33.040 --> 01:09:33.840 align:middle line:90%
it gets better.

01:09:33.840 --> 01:09:35.140 align:middle line:90%
The quality is much better.

01:09:35.140 --> 01:09:38.100 align:middle line:84%
Because each step will
denoise it very slightly,

01:09:38.100 --> 01:09:40.540 align:middle line:84%
so errors won't accumulate
and things like that.

01:09:40.540 --> 01:09:42.899 align:middle line:84%
And the diffuse library
gives you lots of controls

01:09:42.899 --> 01:09:46.060 align:middle line:84%
for fiddling around
with all these things.

01:09:46.060 --> 01:09:48.939 align:middle line:90%
OK, so that's what we had.

01:09:48.939 --> 01:09:50.460 align:middle line:90%
9:49.

01:09:50.460 --> 01:09:52.660 align:middle line:84%
OK, so check out this
tutorial if you're

01:09:52.660 --> 01:09:54.820 align:middle line:84%
curious about how
this stuff works.

01:09:54.820 --> 01:09:57.540 align:middle line:84%
And I'm going to do one
other thing, because I

01:09:57.540 --> 01:09:59.980 align:middle line:90%
didn't get to do it earlier on.

01:09:59.980 --> 01:10:03.480 align:middle line:84%
So we spent some time
with the HuggingFace Hub,

01:10:03.480 --> 01:10:05.860 align:middle line:84%
and I walked you through
a few use cases for text

01:10:05.860 --> 01:10:09.868 align:middle line:84%
where you can take a text model
and use it for classification,

01:10:09.868 --> 01:10:12.160 align:middle line:84%
things like that, summarization
and so on and so forth.

01:10:12.160 --> 01:10:14.940 align:middle line:84%
You can do the same thing
for computer vision models.

01:10:14.940 --> 01:10:17.100 align:middle line:84%
So if you have a computer
vision problem that

01:10:17.100 --> 01:10:20.040 align:middle line:84%
just maps to a standard
computer vision task,

01:10:20.040 --> 01:10:22.540 align:middle line:84%
you can just use the
HuggingFace Hub as well.

01:10:22.540 --> 01:10:26.980 align:middle line:84%
So let me just show you very
quickly, the same kind of thing

01:10:26.980 --> 01:10:27.830 align:middle line:90%
actually works here.

01:10:27.830 --> 01:10:32.460 align:middle line:90%


01:10:32.460 --> 01:10:35.275 align:middle line:90%
All right.

01:10:35.275 --> 01:10:37.680 align:middle line:84%
So let's say that you want
to classify something.

01:10:37.680 --> 01:10:40.580 align:middle line:84%
You just import the
pipeline as before.

01:10:40.580 --> 01:10:43.580 align:middle line:84%
And once you import it,
you can just literally

01:10:43.580 --> 01:10:45.680 align:middle line:84%
give it the standard
task that you care about,

01:10:45.680 --> 01:10:48.690 align:middle line:90%
like image classification.

01:10:48.690 --> 01:10:53.390 align:middle line:84%
And then, you can start using
it right from that point on.

01:10:53.390 --> 01:10:53.890 align:middle line:90%
OK.

01:10:53.890 --> 01:10:59.770 align:middle line:90%


01:10:59.770 --> 01:11:00.400 align:middle line:90%
All right.

01:11:00.400 --> 01:11:00.900 align:middle line:90%
OK.

01:11:00.900 --> 01:11:03.850 align:middle line:84%
So now, I'm going to
just get this image.

01:11:03.850 --> 01:11:06.090 align:middle line:90%
So it's a very famous image.

01:11:06.090 --> 01:11:08.290 align:middle line:84%
And we're going to ask it
to classify this image.

01:11:08.290 --> 01:11:12.290 align:middle line:84%
So we just literally run
it through the pipeline.

01:11:12.290 --> 01:11:16.210 align:middle line:84%
And it says the most likely
label is 94% probability.

01:11:16.210 --> 01:11:18.450 align:middle line:90%
It's an Egyptian cat.

01:11:18.450 --> 01:11:19.610 align:middle line:90%
Seems reasonable.

01:11:19.610 --> 01:11:21.530 align:middle line:84%
I mean, it's a tough
picture because there

01:11:21.530 --> 01:11:23.363 align:middle line:84%
are lots of things going
on in that picture.

01:11:23.363 --> 01:11:25.970 align:middle line:84%
It's not like, one
image, one object.

01:11:25.970 --> 01:11:28.630 align:middle line:84%
So you don't have to
use the default model.

01:11:28.630 --> 01:11:30.910 align:middle line:84%
You can actually give it
your own model that you want.

01:11:30.910 --> 01:11:35.650 align:middle line:84%
So for example,
you can go, you can

01:11:35.650 --> 01:11:39.730 align:middle line:84%
go to the Hub, HuggingFace
Hub, and you can go in there

01:11:39.730 --> 01:11:42.290 align:middle line:84%
and say, all right I want
image classification.

01:11:42.290 --> 01:11:45.180 align:middle line:84%
These are all the
models, 10,487 models.

01:11:45.180 --> 01:11:48.880 align:middle line:84%
They're sorted by, I
don't know, most downloads

01:11:48.880 --> 01:11:51.960 align:middle line:90%
or maybe most likes.

01:11:51.960 --> 01:11:53.340 align:middle line:90%
And you have all these models.

01:11:53.340 --> 01:11:54.300 align:middle line:90%
You can pick any one of them.

01:11:54.300 --> 01:11:55.240 align:middle line:84%
So for example,
let's say you want

01:11:55.240 --> 01:11:57.220 align:middle line:84%
to pick Microsoft
ResNet as your one.

01:11:57.220 --> 01:11:58.560 align:middle line:90%
That's what I tried here.

01:11:58.560 --> 01:11:59.900 align:middle line:90%
So I have Microsoft ResNet.

01:11:59.900 --> 01:12:03.520 align:middle line:84%
You just do model
equals that, run it,

01:12:03.520 --> 01:12:06.200 align:middle line:84%
and it takes care of all
the tokenization, this, that

01:12:06.200 --> 01:12:06.820 align:middle line:90%
and whatnot.

01:12:06.820 --> 01:12:08.280 align:middle line:90%
It's really very handy.

01:12:08.280 --> 01:12:10.320 align:middle line:84%
And then, you run it
through the pipeline again,

01:12:10.320 --> 01:12:11.920 align:middle line:90%
and it says tiger cat.

01:12:11.920 --> 01:12:15.440 align:middle line:84%
94% probability,
according to ResNet.

01:12:15.440 --> 01:12:17.020 align:middle line:90%
So yeah, that's how you do it.

01:12:17.020 --> 01:12:18.600 align:middle line:84%
Now, let's actually try a
more interesting example

01:12:18.600 --> 01:12:21.183 align:middle line:84%
where you want to detect all the
objects in the picture, which

01:12:21.183 --> 01:12:23.220 align:middle line:84%
we didn't talk about in
class, object detection.

01:12:23.220 --> 01:12:27.400 align:middle line:84%
So just create an object
detection pipeline.

01:12:27.400 --> 01:12:28.880 align:middle line:90%
Same thing as before.

01:12:28.880 --> 01:12:30.500 align:middle line:84%
When you actually
run this command,

01:12:30.500 --> 01:12:32.680 align:middle line:84%
an astonishing some amount
of complicated stuff

01:12:32.680 --> 01:12:34.700 align:middle line:90%
is going on under the hood.

01:12:34.700 --> 01:12:37.560 align:middle line:84%
And we are all the beneficiaries
of that, so thank you.

01:12:37.560 --> 01:12:38.100 align:middle line:90%
So yeah.

01:12:38.100 --> 01:12:42.307 align:middle line:84%
So we have this here, and then
we run it through the pipeline.

01:12:42.307 --> 01:12:44.390 align:middle line:84%
It's looking at all the
possible things that might

01:12:44.390 --> 01:12:45.515 align:middle line:90%
be sitting in the pipeline.

01:12:45.515 --> 01:12:49.950 align:middle line:84%
The results are hard to read, so
let's actually visualize them.

01:12:49.950 --> 01:12:51.750 align:middle line:84%
And I got some nice
code from this site,

01:12:51.750 --> 01:12:52.930 align:middle line:90%
for how to visualize them.

01:12:52.930 --> 01:12:54.830 align:middle line:90%
Let's just reuse it.

01:12:54.830 --> 01:12:55.530 align:middle line:90%
So yeah.

01:12:55.530 --> 01:12:59.410 align:middle line:84%
So if you plot the
results, look at that.

01:12:59.410 --> 01:13:04.150 align:middle line:90%


01:13:04.150 --> 01:13:07.830 align:middle line:84%
So it has picked up the cat,
100% probability, I guess.

01:13:07.830 --> 01:13:12.570 align:middle line:84%
The remote, the couch, the
other remote, and then the cat.

01:13:12.570 --> 01:13:13.630 align:middle line:90%
Pretty good, right?

01:13:13.630 --> 01:13:16.170 align:middle line:90%
Off the shelf, ready to go.

01:13:16.170 --> 01:13:17.790 align:middle line:90%
No heavy lifting required.

01:13:17.790 --> 01:13:19.550 align:middle line:84%
Now, in this case,
we are actually

01:13:19.550 --> 01:13:21.350 align:middle line:84%
putting these boxes,
called bounding boxes,

01:13:21.350 --> 01:13:22.245 align:middle line:90%
around each picture.

01:13:22.245 --> 01:13:24.370 align:middle line:84%
But what if you actually
don't want a bounding box?

01:13:24.370 --> 01:13:26.910 align:middle line:84%
What if you want to actually
find the exact contour

01:13:26.910 --> 01:13:28.850 align:middle line:90%
of that cat or the remote?

01:13:28.850 --> 01:13:29.690 align:middle line:90%
No problem.

01:13:29.690 --> 01:13:31.910 align:middle line:84%
We do something called
image segmentation.

01:13:31.910 --> 01:13:36.510 align:middle line:84%
So let's do an image
segmentation pipeline.

01:13:36.510 --> 01:13:37.320 align:middle line:90%
And run it through.

01:13:37.320 --> 01:13:42.900 align:middle line:90%


01:13:42.900 --> 01:13:45.780 align:middle line:90%
It takes some time.

01:13:45.780 --> 01:13:46.660 align:middle line:90%
All right.

01:13:46.660 --> 01:13:47.160 align:middle line:90%
All right.

01:13:47.160 --> 01:13:49.080 align:middle line:90%
Let's visualize it.

01:13:49.080 --> 01:13:51.840 align:middle line:84%
So each object it finds,
it gives you a mask.

01:13:51.840 --> 01:13:53.940 align:middle line:84%
It basically tells
you for each object

01:13:53.940 --> 01:13:56.260 align:middle line:84%
what object it
is, and then which

01:13:56.260 --> 01:13:59.640 align:middle line:84%
pixels are on for that object
and off for everything else.

01:13:59.640 --> 01:14:00.280 align:middle line:90%
It's a mask.

01:14:00.280 --> 01:14:01.680 align:middle line:90%
It tells you where it stands.

01:14:01.680 --> 01:14:05.020 align:middle line:84%
And you can see here, it has
the first object that's found,

01:14:05.020 --> 01:14:06.160 align:middle line:90%
is this thing here.

01:14:06.160 --> 01:14:08.300 align:middle line:90%
And it's perfectly delineated.

01:14:08.300 --> 01:14:09.420 align:middle line:90%
It's pretty amazing.

01:14:09.420 --> 01:14:11.820 align:middle line:84%
So we can overlay this
on the original image

01:14:11.820 --> 01:14:14.770 align:middle line:90%
and see that it's found that.

01:14:14.770 --> 01:14:16.755 align:middle line:90%
And let's look at other objects.

01:14:16.755 --> 01:14:17.880 align:middle line:90%
Oh, it is found the remote.

01:14:17.880 --> 01:14:20.460 align:middle line:90%
That's the second object.

01:14:20.460 --> 01:14:21.970 align:middle line:90%
And the third remote.

01:14:21.970 --> 01:14:24.580 align:middle line:90%


01:14:24.580 --> 01:14:26.160 align:middle line:90%
And the fourth.

01:14:26.160 --> 01:14:27.988 align:middle line:84%
You think any other
objects are remaining?

01:14:27.988 --> 01:14:28.780 align:middle line:90%
STUDENT: The couch.

01:14:28.780 --> 01:14:30.780 align:middle line:90%
PROFESSOR: Couch, good.

01:14:30.780 --> 01:14:33.620 align:middle line:90%
All right, let's find the couch.

01:14:33.620 --> 01:14:36.240 align:middle line:84%
And look, the couch
is pretty good,

01:14:36.240 --> 01:14:39.650 align:middle line:84%
except that the middle
part has gotten confused.

01:14:39.650 --> 01:14:40.150 align:middle line:90%
All right.

01:14:40.150 --> 01:14:41.567 align:middle line:84%
But it's still
pretty good, right?

01:14:41.567 --> 01:14:44.090 align:middle line:90%
So, yeah.

01:14:44.090 --> 01:14:46.050 align:middle line:84%
So Hugging Face is
all these things,

01:14:46.050 --> 01:14:48.810 align:middle line:84%
and so you should definitely
check it out if you're not

01:14:48.810 --> 01:14:50.190 align:middle line:90%
already very familiar with it.

01:14:50.190 --> 01:14:51.710 align:middle line:90%
So we have one minute left.

01:14:51.710 --> 01:14:52.470 align:middle line:90%
Any questions?

01:14:52.470 --> 01:14:58.170 align:middle line:90%


01:14:58.170 --> 01:14:59.050 align:middle line:90%
No questions?

01:14:59.050 --> 01:14:59.550 align:middle line:90%
OK.

01:14:59.550 --> 01:15:00.270 align:middle line:90%
All right, folks.

01:15:00.270 --> 01:15:01.145 align:middle line:90%
See you on Wednesday.

01:15:01.145 --> 01:15:02.540 align:middle line:90%
Thanks.

01:15:02.540 --> 01:15:38.000 align:middle line:90%