WEBVTT

00:00:00.000 --> 00:00:02.435
[SQUEAKING]

00:00:02.435 --> 00:00:03.870
[RUSTLING]

00:00:03.870 --> 00:00:07.307
[CLICKING]

00:00:16.516 --> 00:00:18.184
All right, so today's lecture.

00:00:18.184 --> 00:00:21.021
Introduction to neural networks
and deep learning.

00:00:21.021 --> 00:00:22.455
So we'll start
with a very quick intro

00:00:22.455 --> 00:00:25.325
to these things, and then
we'll switch and dive deep

00:00:25.325 --> 00:00:26.526
into neural networks.

00:00:26.526 --> 00:00:29.462
All right, so the field
of AI originated in 1956.

00:00:29.462 --> 00:00:30.864
Sadly, it didn't
originate at MIT.

00:00:30.864 --> 00:00:33.133
It originated at Dartmouth
because all these people

00:00:33.133 --> 00:00:34.300
got together at Dartmouth--

00:00:34.300 --> 00:00:37.504
I guess it's got a nice quad or
whatever-- they got together,

00:00:37.504 --> 00:00:38.905
they defined the field.

00:00:38.905 --> 00:00:42.208
But fortunately for us, MIT
was very well represented.

00:00:42.208 --> 00:00:46.012
So we have Marvin Minsky,
who founded the MIT AI Lab--

00:00:46.012 --> 00:00:50.016
John McCarthy, who invented
LISP and then later defected

00:00:50.016 --> 00:00:51.184
to the West Coast--

00:00:51.184 --> 00:00:54.087
and then Claude Shannon, who
invented information theory, who

00:00:54.087 --> 00:00:55.488
was a professor at MIT.

00:00:55.488 --> 00:00:57.123
So MIT was well represented.

00:00:57.123 --> 00:00:58.658
These folks founded the field.

00:00:58.658 --> 00:01:01.928
And they were so bright, they
thought that AI was going to be

00:01:01.928 --> 00:01:06.900
substantially solved, quote
unquote, "by that fall."

00:01:06.900 --> 00:01:09.269
Now, obviously, it turned
out a bit differently

00:01:09.269 --> 00:01:10.870
than what they expected.

00:01:10.870 --> 00:01:14.007
So it's been, whatever, 67,
68 years since its founding.

00:01:14.007 --> 00:01:15.742
So it's gone
through, essentially,

00:01:15.742 --> 00:01:18.545
in my opinion, three
seminal breakthroughs,

00:01:18.545 --> 00:01:21.114
starting with the traditional
approach, then machine learning,

00:01:21.114 --> 00:01:22.482
deep learning,
and generative AI.

00:01:22.482 --> 00:01:25.118
So let's take a very quick look
at each of these breakthroughs

00:01:25.118 --> 00:01:26.686
and what motivated them.

00:01:26.686 --> 00:01:29.589
So let's start with the
traditional approach to AI.

00:01:29.589 --> 00:01:30.857
And so what is AI?

00:01:30.857 --> 00:01:34.027
AI, informally, is the
ability to imbue computers

00:01:34.027 --> 00:01:37.764
with the ability to do things
that only humans can typically

00:01:37.764 --> 00:01:40.233
do-- cognitive tasks, thinking
tasks, and things like that.

00:01:40.233 --> 00:01:42.368
And so the most
common, sensical way

00:01:42.368 --> 00:01:45.171
to do that is to say, well,
if I want the computer to do

00:01:45.171 --> 00:01:46.906
something complicated,
like play chess,

00:01:46.906 --> 00:01:49.843
I'm just going to sit down
with a few chess grandmasters,

00:01:49.843 --> 00:01:51.678
show them a whole
bunch of board moves,

00:01:51.678 --> 00:01:54.013
and ask them how they
figure out how to respond,

00:01:54.013 --> 00:01:55.115
how to play the next move.

00:01:55.115 --> 00:01:57.250
I'm going to sit down and
talk to all these people,

00:01:57.250 --> 00:01:59.719
and then I'm going to write
down a whole bunch of rules.

00:01:59.719 --> 00:02:01.554
If this is the board
position, move this.

00:02:01.554 --> 00:02:02.622
If this is the
board position, move

00:02:02.622 --> 00:02:03.823
this, and so on and so forth.

00:02:03.823 --> 00:02:05.558
Or I might sit down
with a cardiologist

00:02:05.558 --> 00:02:08.194
and tell them, OK, how do you
actually interpret an ECG?

00:02:08.194 --> 00:02:10.196
They will give me
all this-- similarly,

00:02:10.196 --> 00:02:11.297
a bunch of if-then rules.

00:02:11.297 --> 00:02:12.198
I will take all
these rules, I'll

00:02:12.198 --> 00:02:13.733
put them into the
computer, and boom,

00:02:13.733 --> 00:02:16.436
I have a system that can
do what a human can do.

00:02:16.436 --> 00:02:19.005
Now, this approach, even
though it's commonsensical,

00:02:19.005 --> 00:02:20.306
it kind of makes sense.

00:02:20.306 --> 00:02:22.976
It had success in
only a few areas.

00:02:22.976 --> 00:02:25.478
And so the interesting
question is, why was it

00:02:25.478 --> 00:02:28.114
not pervasively successful?

00:02:28.114 --> 00:02:29.716
Why was it not
pervasively successful?

00:02:29.716 --> 00:02:31.351
It seems like a pretty
good idea to me.

00:02:31.351 --> 00:02:33.920
And the people who came up with
these things are smart people.

00:02:33.920 --> 00:02:35.021
They're not dumb people.

00:02:35.021 --> 00:02:36.256
They know what they're doing.

00:02:36.256 --> 00:02:37.524
So why did it not work?

00:02:40.727 --> 00:02:42.829
AUDIENCE: Because
it's time-intensive,

00:02:42.829 --> 00:02:45.465
so in the case that you have to
run through all these scenarios

00:02:45.465 --> 00:02:48.301
that can ever exist, and still
some new scenarios can come up

00:02:48.301 --> 00:02:50.436
that you didn't cater
for [INAUDIBLE. ]

00:02:50.436 --> 00:02:51.471
RAMA RAMAKRISHNAN: Right.

00:02:51.471 --> 00:02:53.239
So there are two aspects
to what you said,

00:02:53.239 --> 00:02:55.308
which is the first aspect
is it's time-intensive.

00:02:55.308 --> 00:02:56.976
That, as it turns
out, is not a big deal

00:02:56.976 --> 00:02:59.779
because computers are
getting faster and faster.

00:02:59.779 --> 00:03:01.548
The second thing is
actually the key thing,

00:03:01.548 --> 00:03:04.250
which is that it doesn't
generalize to new situations

00:03:04.250 --> 00:03:06.152
very well.

00:03:06.152 --> 00:03:08.421
The problem is there are an
infinite number of things

00:03:08.421 --> 00:03:09.789
that you're going
to see when you

00:03:09.789 --> 00:03:11.457
deploy these systems
in the real world.

00:03:11.457 --> 00:03:12.892
By definition, what
you're training

00:03:12.892 --> 00:03:15.028
it on is a small
sample of rules.

00:03:15.028 --> 00:03:16.663
So these rules are very brittle.

00:03:16.663 --> 00:03:19.499
But there's actually an even
more interesting reason,

00:03:19.499 --> 00:03:23.836
and that reason is that we
know more than we can tell.

00:03:23.836 --> 00:03:25.572
This is called
"Polanyi's paradox."

00:03:25.572 --> 00:03:28.575
So the idea is that if I come
to you and say, hey, here's

00:03:28.575 --> 00:03:30.610
a picture, is it a dog or a cat?

00:03:30.610 --> 00:03:32.178
You will tell me within--

00:03:32.178 --> 00:03:33.546
I believe they
measured it, like,

00:03:33.546 --> 00:03:34.747
20 milliseconds or something.

00:03:34.747 --> 00:03:36.516
You know if it's a dog or a cat.

00:03:36.516 --> 00:03:39.185
And then if I ask you to
explain to me exactly how you

00:03:39.185 --> 00:03:41.788
figured that out, you'll come
up with a bunch of reasons--

00:03:41.788 --> 00:03:43.056
alleged reasons.

00:03:43.056 --> 00:03:45.892
Oh, if it has whiskers, I
think it's a cat, or whatever.

00:03:45.892 --> 00:03:48.228
But the problem is that
you actually, first of all,

00:03:48.228 --> 00:03:50.029
can't really articulate
what's going on in your head,

00:03:50.029 --> 00:03:51.064
how you do these things.

00:03:51.064 --> 00:03:53.233
And number 2, even
if you articulate it,

00:03:53.233 --> 00:03:55.401
oftentimes your articulation
has no correspondence

00:03:55.401 --> 00:03:57.937
with how your brain
actually does it.

00:03:57.937 --> 00:04:01.140
So you're incomplete underlier.

00:04:01.140 --> 00:04:02.909
So this is Polanyi's paradox.

00:04:02.909 --> 00:04:06.980
So if you can't even tell me how
you do something, how the heck

00:04:06.980 --> 00:04:09.148
am I supposed to take it
and put it into a computer?

00:04:09.148 --> 00:04:09.849
Doesn't work.

00:04:09.849 --> 00:04:12.318
And second is the fact that we
can't write down these rules

00:04:12.318 --> 00:04:14.554
for all possible situations--

00:04:14.554 --> 00:04:16.723
edge cases, corner cases,
cetera-- and the world

00:04:16.723 --> 00:04:18.625
is full of edge cases.

00:04:18.625 --> 00:04:21.527
So for these reasons,
this approach didn't work.

00:04:21.527 --> 00:04:23.429
And so a different
approach was developed,

00:04:23.429 --> 00:04:25.898
and this approach was,
well, basically said, hey,

00:04:25.898 --> 00:04:27.867
instead of explicitly
telling the computer what

00:04:27.867 --> 00:04:29.869
to do, why don't we
simply give it lots

00:04:29.869 --> 00:04:32.538
of examples of
inputs and outputs?

00:04:32.538 --> 00:04:34.507
Chess positions-- next move.

00:04:34.507 --> 00:04:37.810
ECG-- diagnosis--
inputs and outputs.

00:04:37.810 --> 00:04:40.313
And then why don't we just use
some statistical techniques

00:04:40.313 --> 00:04:43.549
to learn a mapping a function
that can go from the input

00:04:43.549 --> 00:04:44.450
to the output?

00:04:44.450 --> 00:04:48.521
That was the idea, and this
idea is machine learning.

00:04:48.521 --> 00:04:50.556
So machine learning is
basically just a fancy way

00:04:50.556 --> 00:04:52.992
of saying learn from
input-output examples

00:04:52.992 --> 00:04:55.828
using statistical techniques.

00:04:55.828 --> 00:04:56.362
Good.

00:04:56.362 --> 00:04:59.966
All right, so now
there are numerous ways

00:04:59.966 --> 00:05:01.367
to create machine
learning models.

00:05:01.367 --> 00:05:03.202
And if you have ever
done linear regression,

00:05:03.202 --> 00:05:06.773
congratulations, you've
been doing machine learning.

00:05:06.773 --> 00:05:09.575
And only one of those methods
happens to be something called

00:05:09.575 --> 00:05:10.843
"neural networks."

00:05:10.843 --> 00:05:13.179
There are many other methods,
and, in fact, you probably

00:05:13.179 --> 00:05:14.947
have done these
other methods if you

00:05:14.947 --> 00:05:17.283
have done a course like The
Analytics Edge or something

00:05:17.283 --> 00:05:19.619
similar.

00:05:19.619 --> 00:05:22.255
So machine learning has
got tremendous impact

00:05:22.255 --> 00:05:23.356
around the world.

00:05:23.356 --> 00:05:26.793
It's like, at this point,
it's widely accepted.

00:05:26.793 --> 00:05:29.362
It's a very, very
successful technology.

00:05:29.362 --> 00:05:32.565
And in fact, whenever people
are actually talking about AI,

00:05:32.565 --> 00:05:35.535
chances are they're actually
talking about machine learning.

00:05:35.535 --> 00:05:38.871
It's just that AI sounds cooler.

00:05:38.871 --> 00:05:41.607
The only problem is for machine
learning to work really well,

00:05:41.607 --> 00:05:44.143
the input data has
to be structured.

00:05:44.143 --> 00:05:47.280
And what I mean by that is
data that can essentially

00:05:47.280 --> 00:05:50.049
be numericalized and
stuck into the columns

00:05:50.049 --> 00:05:52.218
and rows of a spreadsheet.

00:05:52.218 --> 00:05:54.020
So for example,
here, let's say I

00:05:54.020 --> 00:05:58.725
want to put together a dataset
of patients, their symptoms,

00:05:58.725 --> 00:06:01.394
and their characteristics, and
then in the following year,

00:06:01.394 --> 00:06:03.396
after they showed up
at the doctor's office,

00:06:03.396 --> 00:06:05.331
whether they had a
cardiac event or not.

00:06:05.331 --> 00:06:06.833
I might create a
dataset like this

00:06:06.833 --> 00:06:10.103
with age, smoking status, yes-no
exercise, blah, blah, blah,

00:06:10.103 --> 00:06:11.237
blah, blah.

00:06:11.237 --> 00:06:13.473
And so either these
numbers are numbers--

00:06:13.473 --> 00:06:15.508
they're numerical-- or if
they are not numerical,

00:06:15.508 --> 00:06:17.577
they're categorical.

00:06:17.577 --> 00:06:20.446
Yes-no smoking, yes-no--
things like that.

00:06:20.446 --> 00:06:22.582
Which means that if you
have categorical variables,

00:06:22.582 --> 00:06:25.118
you can just numericalize
them pretty easily.

00:06:25.118 --> 00:06:27.153
You folks have done some
machine learning before,

00:06:27.153 --> 00:06:28.955
so you know things like
one-hot encoding and stuff

00:06:28.955 --> 00:06:30.990
like that can be done to
make them all numerical.

00:06:30.990 --> 00:06:32.792
So the point is,
you can just render

00:06:32.792 --> 00:06:35.928
the data into the columns
and rows of a spreadsheet

00:06:35.928 --> 00:06:36.963
pretty easily.

00:06:36.963 --> 00:06:39.665
That's what I mean
by structured data.

00:06:39.665 --> 00:06:41.134
But the situation
is very different

00:06:41.134 --> 00:06:42.535
if you have unstructured data.

00:06:42.535 --> 00:06:45.438
So if you have an
image of a cute puppy--

00:06:45.438 --> 00:06:48.808
this is my puppy, by the
way, from many years ago.

00:06:48.808 --> 00:06:52.278
Sadly, he's no more,
but his name was Google.

00:06:52.278 --> 00:06:54.747
[LAUGHTER]

00:06:54.747 --> 00:06:58.384
So, yeah, anyway, my DMD
alums know Google well.

00:06:58.384 --> 00:06:59.485
So this Google.

00:06:59.485 --> 00:07:01.421
If you want to take Google--

00:07:01.421 --> 00:07:04.390
this picture-- and figure
out how to numericalize it,

00:07:04.390 --> 00:07:06.359
the first thing you want
to-- need to understand

00:07:06.359 --> 00:07:10.229
is that if you actually look at
how this picture is represented

00:07:10.229 --> 00:07:12.799
digitally in the computer,
basically every picture

00:07:12.799 --> 00:07:16.702
like this is depicted using
three tables of numbers.

00:07:16.702 --> 00:07:18.838
And we'll get to what these
numbers mean later on,

00:07:18.838 --> 00:07:22.575
but the point I'm making is that
each number basically represents

00:07:22.575 --> 00:07:25.478
the amount of light--

00:07:25.478 --> 00:07:27.647
on a scale of 0
to 255, the amount

00:07:27.647 --> 00:07:29.749
of light in that
location, in that pixel.

00:07:29.749 --> 00:07:31.317
That's all-- the
amount of light.

00:07:31.317 --> 00:07:34.187
So basically, this
table is the amount of--

00:07:34.187 --> 00:07:37.323
sorry-- this table is
amount of red light,

00:07:37.323 --> 00:07:39.859
amount of green light,
amount of blue light.

00:07:39.859 --> 00:07:42.428
Now, you will agree with me
that if you, for example, look

00:07:42.428 --> 00:07:45.998
at something like this and
say, OK, 251 at this location--

00:07:45.998 --> 00:07:48.367
there is a lot of blue
light because it's

00:07:48.367 --> 00:07:51.003
251 out of a possible 255.

00:07:51.003 --> 00:07:52.839
Maybe a lot of blue
light somewhere here.

00:07:52.839 --> 00:07:55.608
There's a lot of blue here.

00:07:55.608 --> 00:07:58.878
Whether that area
is blue because

00:07:58.878 --> 00:08:03.549
of a piece of sky, some water,
or a bunch of blue paint--

00:08:03.549 --> 00:08:06.219
could be anything--
it's going to say 251.

00:08:06.219 --> 00:08:08.888
So the underlying reality,
the underlying object

00:08:08.888 --> 00:08:12.925
that's being described, has
nothing to do with the 251,

00:08:12.925 --> 00:08:14.427
so that's the whole problem.

00:08:14.427 --> 00:08:17.263
The raw form of the data
has no intrinsic meaning

00:08:17.263 --> 00:08:18.798
with the underlying thing.

00:08:18.798 --> 00:08:21.067
So given that there's no
connection between the number

00:08:21.067 --> 00:08:23.169
and what it's describing,
how the heck can

00:08:23.169 --> 00:08:25.037
any algorithm do
anything with it?

00:08:25.037 --> 00:08:25.538
It can't.

00:08:28.374 --> 00:08:31.777
So what you have to do is
something called "feature

00:08:31.777 --> 00:08:34.080
engineering," or
"feature extraction,"

00:08:34.080 --> 00:08:37.450
where you have to manually take
all these things and create,

00:08:37.450 --> 00:08:39.051
essentially, a
spreadsheet from them.

00:08:39.051 --> 00:08:42.054
So basically, let's say that
you have a bunch of birds

00:08:42.054 --> 00:08:44.457
and you're trying to build a
bird classifier to figure out

00:08:44.457 --> 00:08:46.158
what kind of bird species it is.

00:08:46.158 --> 00:08:48.094
You might actually have
to take this picture,

00:08:48.094 --> 00:08:51.564
and then you have to measure
the beak length, the wingspan,

00:08:51.564 --> 00:08:54.467
the primary color, and
so on and so forth.

00:08:54.467 --> 00:08:59.372
So you basically structuring
the unstructured data manually.

00:08:59.372 --> 00:09:03.509
And this process of
structuring unstructured data

00:09:03.509 --> 00:09:06.145
is basically called--

00:09:06.145 --> 00:09:07.780
we use the word
"representation."

00:09:07.780 --> 00:09:10.316
We take the raw data,
and we represent

00:09:10.316 --> 00:09:12.151
the data in a different form.

00:09:12.151 --> 00:09:15.821
And the reason why I'm
focusing on the use of the word

00:09:15.821 --> 00:09:18.257
"representation" is
because it becomes really,

00:09:18.257 --> 00:09:21.694
really important a bit later on
when we get to deep learning.

00:09:21.694 --> 00:09:24.297
So we have to represent the
data in a different way for it

00:09:24.297 --> 00:09:24.830
to work.

00:09:24.830 --> 00:09:26.465
That's the basic idea.

00:09:26.465 --> 00:09:31.037
All right, so what that means is
that historically, researchers

00:09:31.037 --> 00:09:33.506
would manually develop
these representations.

00:09:33.506 --> 00:09:36.342
And once you develop them, once
you have the representations,

00:09:36.342 --> 00:09:38.277
you can just use traditional
linear regression

00:09:38.277 --> 00:09:40.479
or logistic regression
and get the job done.

00:09:40.479 --> 00:09:42.782
So the whole name of the
game is the representations.

00:09:42.782 --> 00:09:46.185
So in fact, people doing PhDs,
for example, in computer vision,

00:09:46.185 --> 00:09:48.487
would spend, like,
four years developing

00:09:48.487 --> 00:09:50.222
amazing representations
for solving

00:09:50.222 --> 00:09:52.391
one particular little problem.

00:09:52.391 --> 00:09:54.260
We have a bunch
of, say, CAT scans,

00:09:54.260 --> 00:09:56.228
and we need to take the
CAT scan and figure out

00:09:56.228 --> 00:09:57.930
whether a particular
kind of stroke--

00:09:57.930 --> 00:09:59.799
there is evidence for
it in the CAT scan.

00:09:59.799 --> 00:10:01.400
They might actually
sit and develop

00:10:01.400 --> 00:10:03.603
all kinds of representations
and test it and so on,

00:10:03.603 --> 00:10:05.905
and then they'll finally
declare victory and say, yay,

00:10:05.905 --> 00:10:06.806
I'm done with my PhD.

00:10:06.806 --> 00:10:08.541
Here is this amazing
representation,

00:10:08.541 --> 00:10:10.309
and you can build a
classifier with it

00:10:10.309 --> 00:10:14.513
to predict a particular kind
of stroke with high accuracy.

00:10:14.513 --> 00:10:18.484
So that's where the world was.

00:10:18.484 --> 00:10:21.220
Now, as you can imagine,
developing representations--

00:10:21.220 --> 00:10:24.290
because it's so manual-- is
this massive human bottleneck,

00:10:24.290 --> 00:10:27.660
and this sharply limited
the reach and applicability

00:10:27.660 --> 00:10:31.897
of machine learning
as you would expect.

00:10:31.897 --> 00:10:34.667
To address this problem, a
different approach came about,

00:10:34.667 --> 00:10:35.701
and that's deep learning.

00:10:35.701 --> 00:10:38.738
So deep learning sits
inside machine learning.

00:10:38.738 --> 00:10:43.242
And deep learning can handle
unstructured input data

00:10:43.242 --> 00:10:46.879
without upfront
manual processing,

00:10:46.879 --> 00:10:51.083
meaning it will automatically
learn the right representations

00:10:51.083 --> 00:10:52.585
from the raw input.

00:10:52.585 --> 00:10:53.986
"Automatically" is the keyword.

00:10:53.986 --> 00:10:55.655
Automatically learn
representations,

00:10:55.655 --> 00:10:57.790
which means that you could
give it structured data,

00:10:57.790 --> 00:10:59.725
you can give it pictures,
you can give it text,

00:10:59.725 --> 00:11:03.462
you can give it anything
you want, and just learn it.

00:11:03.462 --> 00:11:06.399
It can automatically extract
these representations.

00:11:06.399 --> 00:11:08.901
And since it's being
automatically extracted,

00:11:08.901 --> 00:11:11.604
you can imagine a pipeline
where the raw data comes

00:11:11.604 --> 00:11:13.706
in, you have a bunch of
stuff in the middle that's

00:11:13.706 --> 00:11:15.541
learning these
representations automatically

00:11:15.541 --> 00:11:17.243
without your help,
and then boom,

00:11:17.243 --> 00:11:19.078
you just attach a
little linear regression

00:11:19.078 --> 00:11:20.479
or logistic
regression at the end.

00:11:20.479 --> 00:11:22.915
Problem solved.

00:11:22.915 --> 00:11:24.984
That, in a nutshell,
is deep learning--

00:11:24.984 --> 00:11:26.686
input, a whole bunch
of representations

00:11:26.686 --> 00:11:29.288
being learned, and then piped
into a linear or logistic

00:11:29.288 --> 00:11:32.625
regression model.

00:11:32.625 --> 00:11:37.596
So the amazing thing
is this simple idea

00:11:37.596 --> 00:11:40.132
is just incredibly powerful.

00:11:40.132 --> 00:11:44.136
That idea has led to ChatGPT,
has led to AlphaGo, AlphaFold,

00:11:44.136 --> 00:11:45.438
and so on and so forth.

00:11:45.438 --> 00:11:49.975
And I kid you not,
I've been doing

00:11:49.975 --> 00:11:51.844
deep learning for
about 10 years now,

00:11:51.844 --> 00:11:53.913
and every time I look
at it, I literally

00:11:53.913 --> 00:11:56.215
get goosebumps every so often--

00:11:56.215 --> 00:11:59.251
that something so simple
could be so powerful.

00:11:59.251 --> 00:12:01.220
It's really, like,
boggles the mind.

00:12:01.220 --> 00:12:02.855
I'm like, I'm just
so lucky to be alive

00:12:02.855 --> 00:12:04.123
and working during this period.

00:12:06.358 --> 00:12:08.527
And coming from people who
have been in the industry

00:12:08.527 --> 00:12:10.696
a long time, this
breathless exclamation

00:12:10.696 --> 00:12:14.834
is not very rare, particularly
because I'm not in marketing.

00:12:14.834 --> 00:12:19.605
I actually mean it, with all due
apologies to various marketing

00:12:19.605 --> 00:12:20.106
folks.

00:12:20.106 --> 00:12:21.440
[LAUGHTER]

00:12:21.440 --> 00:12:24.710
Just realized it's
being taped, so.

00:12:24.710 --> 00:12:26.445
So this demolished
the human bottleneck

00:12:26.445 --> 00:12:29.682
for using machine learning
with unstructured data.

00:12:29.682 --> 00:12:32.818
And so it comes from the
confluence of three forces--

00:12:32.818 --> 00:12:35.688
new algorithmic
ideas, a lot of data,

00:12:35.688 --> 00:12:37.389
and then, very
importantly, the fact

00:12:37.389 --> 00:12:40.459
that we have access to parallel
computing hardware in the form

00:12:40.459 --> 00:12:41.761
of these things called GPUs--

00:12:41.761 --> 00:12:43.596
graphics processing units.

00:12:43.596 --> 00:12:45.231
And these three
forces came together,

00:12:45.231 --> 00:12:47.833
and they were applied to an old
idea called "neural networks,"

00:12:47.833 --> 00:12:49.034
and that's basically
deep learning.

00:12:49.034 --> 00:12:50.803
And I'll go through it very
quickly, because obviously we're

00:12:50.803 --> 00:12:52.471
going to spend half
the semester looking

00:12:52.471 --> 00:12:54.440
into this thing in detail.

00:12:54.440 --> 00:12:57.409
So what's the
immediate application

00:12:57.409 --> 00:13:01.514
of the ability to automatically
handle unstructured data?

00:13:01.514 --> 00:13:03.549
What is, like, the
no-brainer application?

00:13:10.756 --> 00:13:11.857
It's OK if it's obvious.

00:13:11.857 --> 00:13:14.126
Tell me.

00:13:14.126 --> 00:13:16.228
Sorry.

00:13:16.228 --> 00:13:17.496
AUDIENCE: Image classification.

00:13:17.496 --> 00:13:17.930
RAMA RAMAKRISHNAN: Right.

00:13:17.930 --> 00:13:19.398
So image classification, yes.

00:13:19.398 --> 00:13:20.499
So you can take an image--

00:13:20.499 --> 00:13:22.034
a good example of
unstructured data--

00:13:22.034 --> 00:13:23.669
you can do some
classification on it.

00:13:23.669 --> 00:13:27.606
But more generally,
what I'm getting at

00:13:27.606 --> 00:13:31.410
is that every
sensor in the world

00:13:31.410 --> 00:13:34.847
can be given the ability to
detect, recognize, and classify

00:13:34.847 --> 00:13:35.848
what it's sensing--

00:13:35.848 --> 00:13:36.682
every sensor.

00:13:36.682 --> 00:13:39.852
Because remember,
what does a sensor do?

00:13:39.852 --> 00:13:43.189
A sensor is just a receptacle
for unstructured data.

00:13:43.189 --> 00:13:46.025
A camera is a receptacle
for unstructured video

00:13:46.025 --> 00:13:48.060
or unstructured still images.

00:13:48.060 --> 00:13:49.829
Microphone-- unstructured audio.

00:13:49.829 --> 00:13:53.966
So every sensor-- you can
imagine taking a sensor

00:13:53.966 --> 00:13:56.802
and sticking a little deep
learning system behind it.

00:13:56.802 --> 00:13:59.171
And now, suddenly,
what comes out

00:13:59.171 --> 00:14:01.540
of the sensor of the deep
learning system, you can count,

00:14:01.540 --> 00:14:03.008
you can classify,
you can detect,

00:14:03.008 --> 00:14:04.510
you can do all kinds of stuff.

00:14:04.510 --> 00:14:08.681
In short, you can analyze,
and you can predict.

00:14:08.681 --> 00:14:12.051
And the way I'm
describing it right now,

00:14:12.051 --> 00:14:15.621
you'll be like,
yeah, duh, obviously.

00:14:15.621 --> 00:14:16.388
But you know what?

00:14:16.388 --> 00:14:19.825
This "obviously" thing is
actually not at all obvious

00:14:19.825 --> 00:14:22.228
in terms of whether it will
help you find interesting

00:14:22.228 --> 00:14:24.463
applications or not.

00:14:24.463 --> 00:14:30.102
So here's something I
literally saw last week.

00:14:30.102 --> 00:14:31.871
Actually, I have another
slide before that,

00:14:31.871 --> 00:14:32.972
but we are coming to that.

00:14:32.972 --> 00:14:37.109
So for instance, every time you
use Face ID, unlock your phone,

00:14:37.109 --> 00:14:38.744
this is the basic
principle at work.

00:14:38.744 --> 00:14:40.713
The camera and the
iPhone is the sensor,

00:14:40.713 --> 00:14:42.681
and they stuck a deep
learning system behind it

00:14:42.681 --> 00:14:44.750
to do image classification--
drama, non-drama.

00:14:44.750 --> 00:14:47.353
That's what it's classifying.

00:14:47.353 --> 00:14:50.022
And so here, you have
a breast cancer--

00:14:50.022 --> 00:14:51.390
this is a breast
cancer detection

00:14:51.390 --> 00:14:53.192
system from a mammogram.

00:14:53.192 --> 00:14:56.795
By the way, this picture is
a very interesting picture.

00:14:56.795 --> 00:15:01.033
So there's a professor
in EECS, Regina Barzilay,

00:15:01.033 --> 00:15:03.535
who's a very well-known
expert in this field.

00:15:03.535 --> 00:15:07.106
And she actually has built
a breast cancer detection

00:15:07.106 --> 00:15:10.209
system, which has been deployed
at Mass General Hospital.

00:15:10.209 --> 00:15:13.312
And it turns out, she's actually
a breast cancer survivor,

00:15:13.312 --> 00:15:16.015
and she was--

00:15:16.015 --> 00:15:17.750
--she's good now, all good--

00:15:17.750 --> 00:15:20.986
but after she
built her system, I

00:15:20.986 --> 00:15:25.257
heard that she actually ran that
system against the mammograms

00:15:25.257 --> 00:15:29.728
from many years prior, when
she went for a mammogram

00:15:29.728 --> 00:15:32.398
and was told that
everything is fine.

00:15:32.398 --> 00:15:34.333
She ran the system
on that mammogram,

00:15:34.333 --> 00:15:37.136
and it came back and
said, here's a problem.

00:15:37.136 --> 00:15:39.271
So a very interesting
example where a deep learning

00:15:39.271 --> 00:15:42.474
system picked up something
that a radiologist could not.

00:15:42.474 --> 00:15:45.577
So these things can
be quite powerful.

00:15:45.577 --> 00:15:48.948
Obviously, any self-driving
system has numerous deep

00:15:48.948 --> 00:15:51.250
learning algorithms
running under the hood--

00:15:51.250 --> 00:15:54.086
pedestrian detection,
stoplight detection,

00:15:54.086 --> 00:15:57.189
zebra crossing detection,
so on and so forth.

00:15:57.189 --> 00:16:00.559
It's being very heavily used in
visual inspection manufacturing.

00:16:00.559 --> 00:16:02.027
You have various
cameras, and it'll

00:16:02.027 --> 00:16:03.896
show people looking at--
saying, OK, there is a dent

00:16:03.896 --> 00:16:04.964
or there's a scratch.

00:16:04.964 --> 00:16:07.466
They have a little system, which
is a dent detector, scratch

00:16:07.466 --> 00:16:08.400
detector, and so on.

00:16:08.400 --> 00:16:09.735
That's going on right now.

00:16:09.735 --> 00:16:13.605
And now I come to the example
I saw last week, which is--

00:16:13.605 --> 00:16:15.240
so this is an example
of you can create

00:16:15.240 --> 00:16:17.309
dramatically better
products if you really

00:16:17.309 --> 00:16:19.912
internalize this idea of--

00:16:19.912 --> 00:16:22.348
it's almost like you're looking
at the world and saying,

00:16:22.348 --> 00:16:24.750
oh, there's a sensor-- can I
attach a DL thing behind it?

00:16:24.750 --> 00:16:26.251
That's the way you
should be looking

00:16:26.251 --> 00:16:27.920
at the world for startup ideas.

00:16:27.920 --> 00:16:30.122
So here's an example.

00:16:30.122 --> 00:16:32.858
These apparently are the
world's first smart binoculars.

00:16:35.527 --> 00:16:42.267
This is the binocular two weeks
ago, where you look at the bird,

00:16:42.267 --> 00:16:47.339
and now it tells you what kind
of bird it is right there.

00:16:47.339 --> 00:16:51.710
It's a simple idea, but
imagine you are the first out

00:16:51.710 --> 00:16:53.212
of the gate with this feature.

00:16:53.212 --> 00:16:54.747
You will have a
little bit of an edge

00:16:54.747 --> 00:16:57.683
till everybody catches up,
like, three months later.

00:16:57.683 --> 00:16:59.985
Let's be very clear-- there
are no long-term monopoly

00:16:59.985 --> 00:17:01.253
windows in the world.

00:17:01.253 --> 00:17:02.654
There are only
short-term windows,

00:17:02.654 --> 00:17:06.759
so the hunt is always on for
a little monopoly window.

00:17:06.759 --> 00:17:09.461
So here's an example of that.

00:17:09.461 --> 00:17:13.132
So I encourage you to always
think about the world as--

00:17:13.132 --> 00:17:14.967
where are the sensors here?

00:17:14.967 --> 00:17:16.802
And can I attach something
behind the sensor

00:17:16.802 --> 00:17:19.071
to do something useful with it?

00:17:19.071 --> 00:17:19.638
All right.

00:17:24.843 --> 00:17:27.179
Now let's turn our
attention to the output.

00:17:27.179 --> 00:17:29.148
We have been talking
about in structured data,

00:17:29.148 --> 00:17:32.684
unstructured data, and how
deep learning has unlocked

00:17:32.684 --> 00:17:34.486
the ability to work
with unstructured data,

00:17:34.486 --> 00:17:37.489
but you've been neglecting the
output side of the equation.

00:17:37.489 --> 00:17:41.026
So traditionally, we could
predict single numbers

00:17:41.026 --> 00:17:43.095
or a few numbers pretty easily.

00:17:43.095 --> 00:17:46.331
So you've all done
the canonical--

00:17:46.331 --> 00:17:48.467
should this person be
given a loan application

00:17:48.467 --> 00:17:49.635
in machine learning?

00:17:49.635 --> 00:17:51.437
So it just predicts
the probability

00:17:51.437 --> 00:17:52.838
that a borrower
will repay a loan

00:17:52.838 --> 00:17:56.075
based on a whole bunch of
data or a supply chain.

00:17:56.075 --> 00:17:58.043
You predict the demand
for a product next week,

00:17:58.043 --> 00:17:59.611
or it can predict
a bunch of numbers.

00:17:59.611 --> 00:18:02.247
So given a picture,
you can say, OK,

00:18:02.247 --> 00:18:04.616
which one of 10 kinds
of furniture is it?

00:18:04.616 --> 00:18:07.386
You can predict 10 numbers, 10
probabilities that add up to 1.

00:18:07.386 --> 00:18:08.754
You can predict a
whole bunch of numbers

00:18:08.754 --> 00:18:10.722
that don't have to add
up to 1, such as the GPS

00:18:10.722 --> 00:18:12.724
coordinates of an Uber.

00:18:12.724 --> 00:18:15.327
So these are all simple,
unstructured-- sorry,

00:18:15.327 --> 00:18:17.796
simple structured output--
just a few numbers.

00:18:17.796 --> 00:18:20.899
What we could not do very
easily was to actually generate

00:18:20.899 --> 00:18:23.435
pictures like this.

00:18:23.435 --> 00:18:25.437
We could not generate
unstructured data.

00:18:25.437 --> 00:18:28.407
We could only consume
unstructured data.

00:18:28.407 --> 00:18:30.976
You can generate text, you
can generate pictures, and so

00:18:30.976 --> 00:18:32.878
on-- and audio, and
so on and so forth.

00:18:32.878 --> 00:18:36.115
So with generative AI,
that problem is gone.

00:18:36.115 --> 00:18:38.617
So generative AI is the
ability to actually create

00:18:38.617 --> 00:18:41.019
unstructured data,
and therefore, it

00:18:41.019 --> 00:18:42.654
sits within deep learning.

00:18:42.654 --> 00:18:44.389
It still runs on deep
learning, but it's

00:18:44.389 --> 00:18:47.493
just one kind of deep learning.

00:18:47.493 --> 00:18:49.528
There's plenty of stuff
going on in deep learning

00:18:49.528 --> 00:18:51.730
that's got nothing to
do with generative AI.

00:18:51.730 --> 00:18:54.833
Nowadays, of course, if you're a
self-respecting entrepreneur who

00:18:54.833 --> 00:18:57.469
wants to ride this
craze, you'll probably

00:18:57.469 --> 00:19:01.206
declare whatever you're
doing as generative AI,

00:19:01.206 --> 00:19:03.208
and some VCs may actually
be ready to fund you.

00:19:03.208 --> 00:19:04.042
Who knows?

00:19:04.042 --> 00:19:06.111
But the point is, there's
plenty of stuff going on

00:19:06.111 --> 00:19:07.579
in deep learning
that's got nothing

00:19:07.579 --> 00:19:10.449
to do with generative AI, but
this is the overall picture.

00:19:10.449 --> 00:19:13.719
Now, here we can produce
unstructured outputs

00:19:13.719 --> 00:19:14.953
like pictures.

00:19:14.953 --> 00:19:16.488
You can take this
thing, and then you

00:19:16.488 --> 00:19:19.158
can actually come up with a nice
picture-- description of it.

00:19:19.158 --> 00:19:21.293
This actually is a very
famous picture, by the way,

00:19:21.293 --> 00:19:22.628
in the world of computer vision.

00:19:22.628 --> 00:19:25.230
So we are actually going to be
analyzing this picture a little

00:19:25.230 --> 00:19:28.400
later on in the semester.

00:19:28.400 --> 00:19:31.870
You can obviously go from a very
complicated caption to an image.

00:19:31.870 --> 00:19:33.338
You can go from text to music.

00:19:36.141 --> 00:19:36.942
Can people hear it?

00:19:36.942 --> 00:19:37.242
OK.

00:19:37.242 --> 00:19:37.576
AUDIENCE: Yeah.

00:19:37.576 --> 00:19:38.810
RAMA RAMAKRISHNAN: Yeah, yeah.

00:19:38.810 --> 00:19:40.746
All right, and, of
course, we go from text

00:19:40.746 --> 00:19:42.848
to text, i.e., ChatGPT.

00:19:42.848 --> 00:19:45.184
And then as of a few
months ago, things

00:19:45.184 --> 00:19:46.585
have gotten even
more interesting

00:19:46.585 --> 00:19:48.487
where you can actually go--

00:19:48.487 --> 00:19:51.823
you can send text and an image
in, and you can get text out.

00:19:51.823 --> 00:19:53.725
And in fact, as of
a few weeks ago,

00:19:53.725 --> 00:19:55.994
you can send text,
image, text, image, text,

00:19:55.994 --> 00:19:59.498
image in an arbitrary
sequence into the system,

00:19:59.498 --> 00:20:02.568
and it can actually come back
to you with text and image.

00:20:02.568 --> 00:20:04.469
So things are
becoming multimodal.

00:20:04.469 --> 00:20:06.805
And I just want to share
with you a really fun example

00:20:06.805 --> 00:20:08.807
I saw recently.

00:20:08.807 --> 00:20:12.144
So this person
sends this picture.

00:20:12.144 --> 00:20:13.879
Can folks see this?

00:20:13.879 --> 00:20:17.182
This is very complicated
parking sign, apparently

00:20:17.182 --> 00:20:18.850
in San Francisco.

00:20:18.850 --> 00:20:20.719
And they are like, it's
Wednesday at 4:00 PM.

00:20:20.719 --> 00:20:21.486
Can I park here?

00:20:21.486 --> 00:20:22.321
[LAUGHTER]

00:20:22.321 --> 00:20:23.989
Tell me in one line--
because you really

00:20:23.989 --> 00:20:26.959
didn't want GPT-4 to be giving
you a big essay about this.

00:20:26.959 --> 00:20:29.027
You literally want to park.

00:20:29.027 --> 00:20:32.064
So GPT-4 comes
back and says, yes,

00:20:32.064 --> 00:20:35.200
you can park here for up to
1 hour, starting at 4:00 PM.

00:20:35.200 --> 00:20:36.835
And folks, I
double-checked this thing.

00:20:36.835 --> 00:20:38.470
It's correct.

00:20:38.470 --> 00:20:40.138
We all know these
things hallucinate.

00:20:40.138 --> 00:20:41.340
Can you imagine getting
a parking ticket

00:20:41.340 --> 00:20:42.708
and telling the
judge, I'm sorry,

00:20:42.708 --> 00:20:45.210
I didn't realize it
was hallucinating?

00:20:45.210 --> 00:20:46.745
So you have to double-check it.

00:20:46.745 --> 00:20:49.748
So, yeah, so things are getting
multimodal very quickly.

00:20:49.748 --> 00:20:52.985
And so the picture here
is that within GenAI, we

00:20:52.985 --> 00:20:55.320
used to have these separate
circles-- text to text, text

00:20:55.320 --> 00:20:57.723
to image, text to music,
text to this, text to that,

00:20:57.723 --> 00:20:58.790
and so on and so forth.

00:20:58.790 --> 00:21:00.926
Those are all beginning
to merge now inside GenAI

00:21:00.926 --> 00:21:02.661
because multimodal
models are going

00:21:02.661 --> 00:21:04.496
to become the norm this year.

00:21:04.496 --> 00:21:06.832
We already have really
good closed models.

00:21:06.832 --> 00:21:08.267
We actually already
have very good

00:21:08.267 --> 00:21:11.503
open-source multimodal
models, and so my feeling

00:21:11.503 --> 00:21:13.805
is that by the end
of the year, the idea

00:21:13.805 --> 00:21:18.010
of using a text-only model
is going to be like, really?

00:21:18.010 --> 00:21:19.311
You do that still?

00:21:19.311 --> 00:21:21.647
It's going to become like a
quaint, old-fashioned thing.

00:21:21.647 --> 00:21:23.982
I think multimodality is
going to become the norm.

00:21:23.982 --> 00:21:26.318
So that's where the world is,
and this is the landscape.

00:21:26.318 --> 00:21:28.854
So any questions
on the landscape

00:21:28.854 --> 00:21:30.555
before we actually
start doing some math?

00:21:35.460 --> 00:21:37.729
OK.

00:21:37.729 --> 00:21:38.764
Yeah?

00:21:38.764 --> 00:21:40.632
AUDIENCE: It's not going
to be the landscape,

00:21:40.632 --> 00:21:43.502
but it's more related to
the breast cancer brush.

00:21:43.502 --> 00:21:49.574
So how did she know that
it was evaluating, like,

00:21:49.574 --> 00:21:52.344
the training that you do, and
you were [INAUDIBLE], which

00:21:52.344 --> 00:21:55.080
is based on the existing
breast cancer [INAUDIBLE].

00:21:55.080 --> 00:21:57.182
But here, she was
taking the mammograms

00:21:57.182 --> 00:21:59.985
from six weeks
past, [INAUDIBLE],

00:21:59.985 --> 00:22:02.187
which is probably from
a smaller version which

00:22:02.187 --> 00:22:04.656
was not there earlier, right?

00:22:04.656 --> 00:22:05.390
And [INAUDIBLE]--

00:22:05.390 --> 00:22:07.059
RAMA RAMAKRISHNAN:
You mean the evidence

00:22:07.059 --> 00:22:09.294
of there being a problem
would have been smaller?

00:22:09.294 --> 00:22:09.528
AUDIENCE: Yeah.

00:22:09.528 --> 00:22:10.529
RAMA RAMAKRISHNAN: Yeah.

00:22:10.529 --> 00:22:13.065
AUDIENCE: So how do know
your model is trained enough

00:22:13.065 --> 00:22:16.168
that it done [INAUDIBLE]?

00:22:16.168 --> 00:22:17.169
RAMA RAMAKRISHNAN: Yeah.

00:22:17.169 --> 00:22:18.670
So the question is
that, in general,

00:22:18.670 --> 00:22:20.672
how do you train your
model so that it gives you

00:22:20.672 --> 00:22:23.108
the right answers, given that
over the passage of time,

00:22:23.108 --> 00:22:24.609
the amount of
evidence in this data

00:22:24.609 --> 00:22:26.478
could be very highly variable?

00:22:26.478 --> 00:22:31.016
So in this particular case of
the professor I talked about,

00:22:31.016 --> 00:22:32.417
everything at that
point was going

00:22:32.417 --> 00:22:34.486
through an expert radiologist.

00:22:34.486 --> 00:22:36.388
So five years ago,
this mammogram

00:22:36.388 --> 00:22:38.190
was seen by a radiologist,
and that person

00:22:38.190 --> 00:22:39.958
concluded there is no problem.

00:22:39.958 --> 00:22:41.393
So that was the training label--

00:22:41.393 --> 00:22:43.095
the wrong training label.

00:22:43.095 --> 00:22:46.264
So typically what happens is
that training labels could

00:22:46.264 --> 00:22:48.433
be wrong some small
fraction of the time,

00:22:48.433 --> 00:22:50.369
so you need to have
systems that are robust.

00:22:50.369 --> 00:22:52.371
So your data needs
to be complete,

00:22:52.371 --> 00:22:55.874
it needs to be comprehensive, it
needs to be have correct labels.

00:22:55.874 --> 00:22:58.310
If these ideas are
not met, your systems

00:22:58.310 --> 00:22:59.578
are not going to be that good.

00:22:59.578 --> 00:23:01.780
But as it turns out,
with neural networks,

00:23:01.780 --> 00:23:04.316
even with some amount
of noise in the labels,

00:23:04.316 --> 00:23:06.251
they still do a pretty good job.

00:23:06.251 --> 00:23:08.520
So that's the general idea.

00:23:08.520 --> 00:23:10.689
AUDIENCE: But there has to
be some verification that

00:23:10.689 --> 00:23:11.189
[INAUDIBLE].

00:23:11.189 --> 00:23:12.657
RAMA RAMAKRISHNAN:
The verification

00:23:12.657 --> 00:23:13.692
comes from the human.

00:23:13.692 --> 00:23:17.629
So remember, when we
look at radiology data,

00:23:17.629 --> 00:23:20.031
the data we are working
with is-- the input is,

00:23:20.031 --> 00:23:23.168
let's say, an image, like
a mammogram or something.

00:23:23.168 --> 00:23:26.037
And then a human radiologist,
or a set of radiologists,

00:23:26.037 --> 00:23:29.341
have said, this has a problem
or does not have a problem.

00:23:29.341 --> 00:23:31.243
So that is called
"ground truth."

00:23:31.243 --> 00:23:33.979
So it is this ground
truth image and label--

00:23:33.979 --> 00:23:36.348
this combination that's being
used to train these models.

00:23:39.518 --> 00:23:40.352
Yeah?

00:23:40.352 --> 00:23:43.155
AUDIENCE: Will you be covering,
like, embodiment [INAUDIBLE]?

00:23:43.155 --> 00:23:43.588
RAMA RAMAKRISHNAN: Embodiment?

00:23:43.588 --> 00:23:44.289
AUDIENCE: Yeah.

00:23:44.289 --> 00:23:46.558
RAMA RAMAKRISHNAN: So are we
going to cover embodiment?

00:23:46.558 --> 00:23:49.060
So the embodiment here
refers to the fact

00:23:49.060 --> 00:23:54.166
that if you have robots,
they need to actually operate

00:23:54.166 --> 00:23:55.000
in the real world.

00:23:55.000 --> 00:23:57.436
And so robots are an example
of what's called "embodied

00:23:57.436 --> 00:23:58.470
intelligence."

00:23:58.470 --> 00:24:00.639
So unfortunately, due to
the constraints of time,

00:24:00.639 --> 00:24:02.674
we're not going to get
into robotics at all.

00:24:02.674 --> 00:24:04.609
But I will say that a
lot of the deep learning

00:24:04.609 --> 00:24:05.777
stuff we're going
to talk about, those

00:24:05.777 --> 00:24:07.245
are all fundamental
building blocks

00:24:07.245 --> 00:24:09.848
in modern robotic systems.

00:24:09.848 --> 00:24:15.220
All right, so in summary,
X and Y can be anything,

00:24:15.220 --> 00:24:17.789
and it can be multimodal.

00:24:17.789 --> 00:24:19.291
I literally could
not have put up

00:24:19.291 --> 00:24:21.593
this slide maybe two years ago.

00:24:21.593 --> 00:24:25.697
So it's very simple in how it
looks, but it's very profound.

00:24:25.697 --> 00:24:27.732
You can learn a
mapping from anything

00:24:27.732 --> 00:24:29.634
to anything at this point
very easily, as long

00:24:29.634 --> 00:24:32.204
as you have enough data.

00:24:32.204 --> 00:24:36.508
So now, note that all this
excitement that we see around us

00:24:36.508 --> 00:24:41.279
is everything stems
from deep learning.

00:24:41.279 --> 00:24:43.014
Everything depends
on deep learning,

00:24:43.014 --> 00:24:44.716
and so if you understand
deep learning,

00:24:44.716 --> 00:24:46.785
a lot of interesting
things become possible.

00:24:46.785 --> 00:24:48.253
So let's get going, all right?

00:24:48.253 --> 00:24:49.988
So we'll start with
the very basics--

00:24:49.988 --> 00:24:52.157
what's a neural network?

00:24:52.157 --> 00:24:55.961
Now, recall logistic regression
from back in the day.

00:24:55.961 --> 00:24:57.796
So what is logistic regression?

00:24:57.796 --> 00:24:59.564
You send in a bunch of numbers--

00:24:59.564 --> 00:25:00.866
a vector of numbers--

00:25:00.866 --> 00:25:02.834
and you usually get
a probability out--

00:25:02.834 --> 00:25:04.603
between 0 and 1, what
is the probability

00:25:04.603 --> 00:25:06.571
of something or the other?

00:25:06.571 --> 00:25:09.040
And so this logistic
regression model

00:25:09.040 --> 00:25:12.677
is also represented in this
form, if you will recall.

00:25:12.677 --> 00:25:17.449
So basically, what we do is
we take all these numbers,

00:25:17.449 --> 00:25:19.818
we run it through a linear
function, you get a number,

00:25:19.818 --> 00:25:21.419
and then we take
that thing and run it

00:25:21.419 --> 00:25:24.923
through 1 divided by 1 plus
e raised to minus that.

00:25:24.923 --> 00:25:27.392
And that's guaranteed to give
you a number between 0 and 1,

00:25:27.392 --> 00:25:29.327
which can be interpreted
as a probability,

00:25:29.327 --> 00:25:31.396
and that's logistic regression.

00:25:31.396 --> 00:25:35.166
And the canonical loan
approvals, things like that,

00:25:35.166 --> 00:25:38.870
all fall under this
convenient bucket.

00:25:38.870 --> 00:25:40.705
So this should be
super familiar.

00:25:44.309 --> 00:25:46.711
All right, now we're
going to actually look

00:25:46.711 --> 00:25:52.450
at this simple, modest,
humble little operation using

00:25:52.450 --> 00:25:55.020
the lens of a network of
mathematical operations,

00:25:55.020 --> 00:25:57.789
and the reason why we do it
will become clear a bit later.

00:25:57.789 --> 00:26:01.293
So we'll take this very
simple example where we have,

00:26:01.293 --> 00:26:03.094
let's say, two variables--

00:26:03.094 --> 00:26:05.096
GPA and experience.

00:26:05.096 --> 00:26:08.066
This is the GPA of some
graduates, a number

00:26:08.066 --> 00:26:09.467
of years of work experience.

00:26:09.467 --> 00:26:11.836
And then this is the
dependent variable,

00:26:11.836 --> 00:26:13.471
which is either 0 or 1.

00:26:13.471 --> 00:26:16.641
And 0 If they don't get called
for an interview, 1 if they get

00:26:16.641 --> 00:26:18.143
called for an interview.

00:26:18.143 --> 00:26:21.880
It's a two-input-variable,
one-output-variable problem,

00:26:21.880 --> 00:26:23.348
and it's a
classification problem

00:26:23.348 --> 00:26:25.584
because we are classifying
people into-- will they

00:26:25.584 --> 00:26:26.751
get called for an interview?

00:26:26.751 --> 00:26:29.287
Yes or no.

00:26:29.287 --> 00:26:33.091
And so that's the
setup for this problem.

00:26:33.091 --> 00:26:37.429
And let's say that we
actually run it through any--

00:26:37.429 --> 00:26:40.765
and we actually try to fit a
logistic regression model to it.

00:26:40.765 --> 00:26:42.767
So if you're familiar
with R, for example,

00:26:42.767 --> 00:26:46.137
you would use something
like GLM to fit this model.

00:26:46.137 --> 00:26:49.040
If you use something like
statsmodels in Python,

00:26:49.040 --> 00:26:50.642
there's a similar
function for it.

00:26:50.642 --> 00:26:52.844
Scikit-learn-- there's
another function for it.

00:26:52.844 --> 00:26:55.046
You get the idea.

00:26:55.046 --> 00:26:57.048
You can use whatever
favorite methods

00:26:57.048 --> 00:27:00.018
you have for logistic regression
modeling to get this job done.

00:27:00.018 --> 00:27:02.087
And if you do that with
this little dataset,

00:27:02.087 --> 00:27:04.723
you're going to get
these coefficients.

00:27:04.723 --> 00:27:07.659
The 0.4 is the intercept, 0.2
is a coefficient of a GPA,

00:27:07.659 --> 00:27:12.497
0.5 for experience, and that is
the resulting sigmoid function.

00:27:12.497 --> 00:27:13.565
All right, cool.

00:27:13.565 --> 00:27:16.001
So now let's actually
rewrite this formula

00:27:16.001 --> 00:27:18.003
as a network in
the following way.

00:27:18.003 --> 00:27:20.772
So first, what we'll do is
we'll take GP and experience

00:27:20.772 --> 00:27:22.540
and stick it here
on the left side,

00:27:22.540 --> 00:27:24.576
and we'll put little
circles next to them,

00:27:24.576 --> 00:27:27.178
and we'll call them
the input nodes.

00:27:27.178 --> 00:27:30.815
And so imagine that somebody
writes a GPA into the circle--

00:27:30.815 --> 00:27:34.085
3.5, or years' experience, 2.0--

00:27:34.085 --> 00:27:36.621
and then it flows
through this arrow.

00:27:36.621 --> 00:27:38.423
And as it flows through,
it gets multiplied

00:27:38.423 --> 00:27:40.258
by its coefficient, 0.2.

00:27:40.258 --> 00:27:42.727
The 0.2 is coming from here.

00:27:42.727 --> 00:27:45.230
Similarly, experience
gets multiplied by 0.5.

00:27:45.230 --> 00:27:46.364
It comes in here.

00:27:46.364 --> 00:27:48.266
And this node, as
the plus indicates,

00:27:48.266 --> 00:27:50.368
is adding everything
that's coming into it.

00:27:50.368 --> 00:27:52.904
So it's adding 0.2
times GPA, 0.5 times

00:27:52.904 --> 00:27:54.572
experience, plus the
intercept, which is

00:27:54.572 --> 00:27:55.940
the green arrow coming from--

00:27:55.940 --> 00:27:57.108
on its own.

00:27:57.108 --> 00:27:58.043
It comes through here.

00:27:58.043 --> 00:28:01.179
And what comes out of-- this
is just a single number.

00:28:01.179 --> 00:28:04.683
And that number goes
into this little circle,

00:28:04.683 --> 00:28:08.586
and then out pops a probability.

00:28:08.586 --> 00:28:14.292
So I've sort of done this
ridiculously long-winded way

00:28:14.292 --> 00:28:17.362
of writing a simple
function, and the reason

00:28:17.362 --> 00:28:21.499
why I'm doing it is will
become clear in a second.

00:28:21.499 --> 00:28:24.135
So this is a little
network of operations

00:28:24.135 --> 00:28:25.737
for the simple function.

00:28:25.737 --> 00:28:27.539
And so, for instance,
how you would

00:28:27.539 --> 00:28:29.407
use it is, to make
a prediction, you

00:28:29.407 --> 00:28:31.176
will-- let's say
someone has a 3.8 GPA

00:28:31.176 --> 00:28:32.911
and 1.2 years of experience.

00:28:32.911 --> 00:28:36.347
You just plug it in here,
do the math, you get 0.76.

00:28:36.347 --> 00:28:37.315
Same thing here.

00:28:37.315 --> 00:28:39.651
Comes in here, add them
all up, you get 1.76.

00:28:39.651 --> 00:28:42.387
You'd run 1.76 through
the sigmoid, you get 0.85,

00:28:42.387 --> 00:28:45.023
and that is the probability that
that particular individual may

00:28:45.023 --> 00:28:46.991
get called for an interview.

00:28:46.991 --> 00:28:49.160
At this point, we're just
doing logistic regression.

00:28:49.160 --> 00:28:52.030
Nothing more complicated.

00:28:52.030 --> 00:28:55.600
So now, if you have many
variables, not two variables,

00:28:55.600 --> 00:28:59.270
like x1 through xk,
the same logic applies.

00:28:59.270 --> 00:29:01.906
Each one has some coefficient,
and then there's an intercept.

00:29:01.906 --> 00:29:04.042
They all get added up here,
run through a sigmoid,

00:29:04.042 --> 00:29:05.877
and out pops this number.

00:29:05.877 --> 00:29:07.879
Notice how the data
flows from left to right.

00:29:10.482 --> 00:29:12.283
All right, any
questions on this?

00:29:15.019 --> 00:29:16.421
All right, good.

00:29:16.421 --> 00:29:18.857
So now a terminology.

00:29:18.857 --> 00:29:20.425
So you will actually--
you'll discover

00:29:20.425 --> 00:29:22.560
that the world of neural
networks and deep learning

00:29:22.560 --> 00:29:23.728
has its own terminology.

00:29:23.728 --> 00:29:25.663
They have their own ways
of referring to things

00:29:25.663 --> 00:29:28.199
that the rest of the world has
been referring using something

00:29:28.199 --> 00:29:29.868
else for the longest time.

00:29:29.868 --> 00:29:32.504
It's kind of annoying sometimes,
but it's the way it is.

00:29:32.504 --> 00:29:37.876
So remember, in regression, we
used to call those numbers next

00:29:37.876 --> 00:29:39.644
to each variable
as "coefficients"

00:29:39.644 --> 00:29:41.379
and the constant thing
as an "intercept."

00:29:41.379 --> 00:29:42.080
Well, guess what?

00:29:42.080 --> 00:29:44.149
In this world,
those coefficients

00:29:44.149 --> 00:29:49.120
are actually called weights, and
the intercept are called biases.

00:29:49.120 --> 00:29:50.989
So in the neural
network world, these

00:29:50.989 --> 00:29:52.557
are called weights and biases.

00:29:52.557 --> 00:29:54.159
And sometimes if
you're a little lazy,

00:29:54.159 --> 00:29:56.828
you may just call the
whole thing as weights.

00:29:56.828 --> 00:29:59.931
So when you see in the
newspaper that, oh my god,

00:29:59.931 --> 00:30:04.235
this amazing model's weights
have been leaked on the internet

00:30:04.235 --> 00:30:06.905
or on BitTorrent or something,
that's what's going on.

00:30:06.905 --> 00:30:09.007
All these coefficients
have been leaked

00:30:09.007 --> 00:30:10.942
because once you know
what the coefficients are

00:30:10.942 --> 00:30:12.343
and what the
architecture is, you

00:30:12.343 --> 00:30:14.946
can just reconstruct the model.

00:30:14.946 --> 00:30:17.549
All right, so that's
what's going on here.

00:30:17.549 --> 00:30:20.084
Now, why did we do
this network business?

00:30:20.084 --> 00:30:23.121
Why did we write
it as a network?

00:30:23.121 --> 00:30:24.255
Yeah, what is advantage?

00:30:24.255 --> 00:30:24.789
Any guesses?

00:30:33.865 --> 00:30:35.967
AUDIENCE: When you have
multiple functions flowing

00:30:35.967 --> 00:30:39.470
through the network, it's just
easier to see it that way.

00:30:39.470 --> 00:30:40.505
RAMA RAMAKRISHNAN: Right.

00:30:40.505 --> 00:30:43.474
If you have lots of things
going on, it's easier to see it

00:30:43.474 --> 00:30:45.310
if you actually write
it in graphical form.

00:30:45.310 --> 00:30:46.911
Yes, correct.

00:30:46.911 --> 00:30:51.749
But so is it only a
usability advantage?

00:30:51.749 --> 00:30:54.118
AUDIENCE: The other thing is
you want different functions

00:30:54.118 --> 00:30:56.287
for different layers
of the [INAUDIBLE].

00:30:56.287 --> 00:30:57.989
RAMA RAMAKRISHNAN:
Ah, OK, so maybe

00:30:57.989 --> 00:31:00.692
we want to use different
functions in different layers.

00:31:00.692 --> 00:31:04.462
But I think there's actually
even a larger, more basic point,

00:31:04.462 --> 00:31:07.298
which is that the moment
you write it down,

00:31:07.298 --> 00:31:09.634
you suddenly realize
that I could have

00:31:09.634 --> 00:31:12.437
lots of things in the middle.

00:31:12.437 --> 00:31:14.839
I don't have to go from the
input to the output directly.

00:31:14.839 --> 00:31:16.875
I could do lots of
things in the middle.

00:31:16.875 --> 00:31:18.443
That's some of the key idea.

00:31:18.443 --> 00:31:20.578
So what you do is--

00:31:20.578 --> 00:31:23.214
so remember the notion of
learning representations

00:31:23.214 --> 00:31:25.683
of unstructured data,
where you take a picture

00:31:25.683 --> 00:31:27.685
and say beak length
and things like that.

00:31:27.685 --> 00:31:30.355
And remember, I said deep
learning actually automatically

00:31:30.355 --> 00:31:31.556
learns these things.

00:31:31.556 --> 00:31:34.792
Where is that automatic
learning coming from?

00:31:34.792 --> 00:31:36.628
Well, this is where
it's coming from.

00:31:36.628 --> 00:31:38.997
So what we do is we
take this thing--

00:31:38.997 --> 00:31:42.734
this just a logistic
regression model-- inputs get

00:31:42.734 --> 00:31:45.603
added up as a linear function
or run through a sigmoid.

00:31:45.603 --> 00:31:49.874
And then we are like, hmm, if
we want to learn representations

00:31:49.874 --> 00:31:53.077
of the raw input, we better be
doing something in the middle

00:31:53.077 --> 00:31:56.814
here because the
output is the output.

00:31:56.814 --> 00:31:59.117
That's not going to change.

00:31:59.117 --> 00:32:00.518
It's either a dog or a cat.

00:32:00.518 --> 00:32:03.688
You don't have any
choice as to what it is.

00:32:03.688 --> 00:32:07.191
The only agency you
have at this point

00:32:07.191 --> 00:32:09.093
is you can take the
raw input and do things

00:32:09.093 --> 00:32:10.995
in the middle with it.

00:32:10.995 --> 00:32:12.630
You can do a lot of
stuff in the middle

00:32:12.630 --> 00:32:15.700
and then run it through
something to get the output.

00:32:15.700 --> 00:32:20.605
So in any mathematical
discipline,

00:32:20.605 --> 00:32:23.608
if someone comes to you and
says, here's a bunch of data--

00:32:23.608 --> 00:32:26.210
I want you to do
something with it--

00:32:26.210 --> 00:32:28.980
what is the most basic
first thing you should do?

00:32:31.649 --> 00:32:34.452
Run it through a
linear function.

00:32:34.452 --> 00:32:36.988
The most basic thing in
math is a linear function.

00:32:36.988 --> 00:32:39.290
So given anything, just run
it through linear function,

00:32:39.290 --> 00:32:39.991
see what happens.

00:32:39.991 --> 00:32:41.859
So that's exactly
what we can do.

00:32:41.859 --> 00:32:44.228
So the simplest thing
we can do here--

00:32:44.228 --> 00:32:46.564
we can insert a bunch
of linear functions.

00:32:46.564 --> 00:32:50.335
So what we do is we take all
this input and we just run it--

00:32:50.335 --> 00:32:51.569
we do a linear function on it.

00:32:51.569 --> 00:32:55.506
So think of this as x1 times 2,
plus x3 times 4, and all the way

00:32:55.506 --> 00:32:58.443
to xk times 9, plus some
intercept, and boom,

00:32:58.443 --> 00:32:59.477
it goes to the other end.

00:32:59.477 --> 00:33:05.183
So this little circle here
with a plus in it is just--

00:33:05.183 --> 00:33:08.152
thank you-- this
is just a linear--

00:33:08.152 --> 00:33:10.221
it's a shorthand for
a linear function.

00:33:10.221 --> 00:33:11.923
So whenever you see
a circle with a plus,

00:33:11.923 --> 00:33:14.258
it's just a shorthand
for a linear function.

00:33:14.258 --> 00:33:15.927
So you can take this
whole thing and run

00:33:15.927 --> 00:33:17.862
through a linear function,
and when you do it,

00:33:17.862 --> 00:33:19.330
you'll get some
number right there.

00:33:19.330 --> 00:33:20.365
You'll get some number.

00:33:20.365 --> 00:33:23.401
So you've taken these k numbers
and you've compressed them

00:33:23.401 --> 00:33:26.804
in some way into one number.

00:33:26.804 --> 00:33:28.906
But you don't have to
stop at one number.

00:33:28.906 --> 00:33:30.008
You can do more.

00:33:30.008 --> 00:33:33.978
So we can have a stack of
linear functions in the middle.

00:33:33.978 --> 00:33:36.114
There's a linear function
here, another one here,

00:33:36.114 --> 00:33:36.981
another one here.

00:33:36.981 --> 00:33:40.118
At this point, the
k numbers you have--

00:33:40.118 --> 00:33:42.387
k could be, for example, 1,000.

00:33:42.387 --> 00:33:44.122
It's just the size
of your input data.

00:33:44.122 --> 00:33:45.690
You have taken these
k things, and you

00:33:45.690 --> 00:33:50.328
have compressed them into
three numbers at this point.

00:33:50.328 --> 00:33:53.297
So maybe 4 is the right number,
maybe 10 is the right number.

00:33:53.297 --> 00:33:55.099
We don't know, and
we'll get to how

00:33:55.099 --> 00:33:58.069
do we know what the
right number is later on.

00:33:58.069 --> 00:34:01.139
So we can stack as many
linear functions we want.

00:34:01.139 --> 00:34:02.673
So we have transformed
this k thing

00:34:02.673 --> 00:34:04.242
into a three-dimensional vector.

00:34:04.242 --> 00:34:08.146
k numbers become three
numbers, and now we

00:34:08.146 --> 00:34:11.449
can flow these three
numbers through

00:34:11.449 --> 00:34:12.550
some other little function.

00:34:16.421 --> 00:34:18.056
And as you will see
in a few minutes,

00:34:18.056 --> 00:34:20.691
that function is called
an activation function,

00:34:20.691 --> 00:34:23.294
and it's chosen to be
a nonlinear function

00:34:23.294 --> 00:34:25.863
because if you don't choose
it to be a nonlinear function,

00:34:25.863 --> 00:34:27.899
all the effort
we're doing is going

00:34:27.899 --> 00:34:31.069
to be a total waste of time.

00:34:31.069 --> 00:34:34.072
For now, just take
it on faith that you

00:34:34.072 --> 00:34:36.174
need to have nonlinear
functions here,

00:34:36.174 --> 00:34:40.211
but note that the three numbers
here are still three numbers.

00:34:40.211 --> 00:34:41.579
They are three
different numbers.

00:34:41.579 --> 00:34:43.347
We did three numbers.

00:34:43.347 --> 00:34:45.550
And once we do this, we'll
be like, you know what?

00:34:45.550 --> 00:34:46.150
This was fun.

00:34:46.150 --> 00:34:48.619
Let's do it again.

00:34:48.619 --> 00:34:53.458
So you can do it again, and
you can keep on doing it.

00:34:53.458 --> 00:34:55.660
You can keep it 100
times if you want.

00:34:55.660 --> 00:34:57.495
And the key thing is
that every time you

00:34:57.495 --> 00:35:01.232
do it, you're giving this
network some ability,

00:35:01.232 --> 00:35:05.103
some capacity to learn something
interesting from the data,

00:35:05.103 --> 00:35:07.672
to learn an interesting
representation.

00:35:07.672 --> 00:35:09.273
Now, of course,
you're thinking, well,

00:35:09.273 --> 00:35:10.508
how do we know it's interesting?

00:35:10.508 --> 00:35:11.976
How do we know it's
a useful thing?

00:35:11.976 --> 00:35:14.178
And we'll come to
all that later on.

00:35:14.178 --> 00:35:16.280
We're just giving it the
capacity, the potential

00:35:16.280 --> 00:35:18.049
to learn interesting
things from the data.

00:35:18.049 --> 00:35:19.951
Whether it actually lives
up to its potential,

00:35:19.951 --> 00:35:21.419
we don't know yet.

00:35:21.419 --> 00:35:24.555
We'll give it the potential
because the more transformations

00:35:24.555 --> 00:35:26.891
of the input data you
make, the more opportunity

00:35:26.891 --> 00:35:28.659
you have to do interesting
things with it.

00:35:28.659 --> 00:35:30.328
If I don't even give
you the opportunity

00:35:30.328 --> 00:35:32.797
to transform it once, you
don't have any opportunity.

00:35:32.797 --> 00:35:35.299
If I give you 10 chances
to transform things,

00:35:35.299 --> 00:35:38.069
you have 10 shots at
doing something useful.

00:35:38.069 --> 00:35:40.271
So you can do this repeatedly.

00:35:40.271 --> 00:35:43.307
And once we are done doing
these transformations,

00:35:43.307 --> 00:35:46.277
we just pipe it through to our
good old logistic regression

00:35:46.277 --> 00:35:48.212
sigmoid here, and we are done.

00:35:51.315 --> 00:35:53.951
So this is the basic idea.

00:35:53.951 --> 00:35:55.953
And so just to
contrast it, this was

00:35:55.953 --> 00:35:59.190
good old logistic regression,
where we take the input,

00:35:59.190 --> 00:36:02.460
we run it through a linear
function and pop out a number--

00:36:02.460 --> 00:36:03.828
a probability number.

00:36:03.828 --> 00:36:06.697
But after we do all this stuff,
the input stays the same,

00:36:06.697 --> 00:36:08.599
the output stays the
same, but in the middle,

00:36:08.599 --> 00:36:10.101
we just run it
through a whole bunch

00:36:10.101 --> 00:36:12.403
of these functions, these
layers-- boop, boop, boop,

00:36:12.403 --> 00:36:15.139
boop-- and then
we get the output.

00:36:15.139 --> 00:36:19.477
That's all we have done, and
this is a neural network.

00:36:19.477 --> 00:36:24.048
A neural network is nothing
more than repeatedly transformed

00:36:24.048 --> 00:36:27.685
inputs which are finally fed to
a linear or logistic regression

00:36:27.685 --> 00:36:28.186
model.

00:36:35.393 --> 00:36:38.129
Any questions?

00:36:38.129 --> 00:36:40.198
Could you use the thing
so that everyone can hear?

00:36:40.198 --> 00:36:41.365
Yeah.

00:36:41.365 --> 00:36:42.767
AUDIENCE: There
are two questions.

00:36:42.767 --> 00:36:44.702
Firstly, so can we
say that there is

00:36:44.702 --> 00:36:46.404
a challenge of explainability?

00:36:46.404 --> 00:36:50.441
Is it that we don't know
which arrow it went through?

00:36:50.441 --> 00:36:51.509
That's one.

00:36:51.509 --> 00:36:55.646
Second, who's controlling
the number of iterations

00:36:55.646 --> 00:36:57.181
or the number of functions?

00:36:57.181 --> 00:36:58.216
That's up to us?

00:36:58.216 --> 00:36:58.482
RAMA RAMAKRISHNAN: Yeah.

00:36:58.482 --> 00:36:59.850
AUDIENCE: Or, how
does that work?

00:36:59.850 --> 00:37:02.420
RAMA RAMAKRISHNAN: So, yeah--
so the first question--

00:37:02.420 --> 00:37:07.358
explainability-- we actually
know exactly for any given input

00:37:07.358 --> 00:37:08.226
data point--

00:37:08.226 --> 00:37:10.261
we know exactly how it
flows through the network,

00:37:10.261 --> 00:37:12.029
so there is no problem there.

00:37:12.029 --> 00:37:16.033
The problem is in
ascribing, OK, we

00:37:16.033 --> 00:37:17.768
think this person
is going to be--

00:37:17.768 --> 00:37:21.639
repay the loan because of
this particular attribute.

00:37:21.639 --> 00:37:24.141
We don't know that because
those attributes all

00:37:24.141 --> 00:37:26.944
get enmeshed together and goes
to this complicated thing.

00:37:26.944 --> 00:37:28.379
So we know exactly
what happens--

00:37:28.379 --> 00:37:31.482
we just can't give credit to
any one thing very easily.

00:37:31.482 --> 00:37:33.684
And again, I'm just
standing on the brink

00:37:33.684 --> 00:37:35.586
of this vast ocean
of something called

00:37:35.586 --> 00:37:37.588
explainability and
interpretability,

00:37:37.588 --> 00:37:39.857
which I'll get to a bit
later on in the semester,

00:37:39.857 --> 00:37:44.829
but that's the quick kind of
right-ish, kind of wrong answer.

00:37:44.829 --> 00:37:49.400
Number two, we decide
the number of layers.

00:37:49.400 --> 00:37:50.835
We decide a whole
bunch of things.

00:37:50.835 --> 00:37:52.270
And as we'll see
in a few minutes,

00:37:52.270 --> 00:37:53.804
there is something
that's given to us

00:37:53.804 --> 00:37:55.473
and something we get
to design, and I'll

00:37:55.473 --> 00:37:59.277
make it very clear
which is which.

00:37:59.277 --> 00:38:00.978
Yeah?

00:38:00.978 --> 00:38:02.013
AUDIENCE: Yeah, so this--

00:38:02.013 --> 00:38:03.080
RAMA RAMAKRISHNAN: Did
I say your name right?

00:38:03.080 --> 00:38:04.048
AUDIENCE: Yeah.

00:38:04.048 --> 00:38:06.517
So which functions
have to be linear?

00:38:06.517 --> 00:38:09.320
And also, why does
it have to be linear?

00:38:09.320 --> 00:38:11.789
RAMA RAMAKRISHNAN: Yeah,
so these functions--

00:38:11.789 --> 00:38:13.190
the f of x here--

00:38:13.190 --> 00:38:15.159
they have to be nonlinear.

00:38:15.159 --> 00:38:16.994
As to why they have
to be nonlinear,

00:38:16.994 --> 00:38:20.131
we'll get to that
in a few minutes.

00:38:20.131 --> 00:38:23.467
So these are called "neurons."

00:38:23.467 --> 00:38:25.236
These things where
you-- basically, there's

00:38:25.236 --> 00:38:30.474
a linear function followed by
a little nonlinear function.

00:38:30.474 --> 00:38:34.211
Each one of these things
is called a "neuron."

00:38:34.211 --> 00:38:38.616
By the way, this is
loosely inspired by the way

00:38:38.616 --> 00:38:42.987
how neurons work in human--
in mammalian brains.

00:38:42.987 --> 00:38:47.858
But the connections between
neuroscience and deep learning

00:38:47.858 --> 00:38:53.764
are very heavily argued, so
I'm going to stay away from it.

00:38:53.764 --> 00:38:56.867
Suffice it to say, I
just think for building

00:38:56.867 --> 00:38:59.203
practical deep learning
systems in industry,

00:38:59.203 --> 00:39:01.839
you're not worry about this.

00:39:01.839 --> 00:39:03.941
All right, let's move on.

00:39:03.941 --> 00:39:07.745
Terminology-- this vertical
stack of linear functions

00:39:07.745 --> 00:39:09.347
or neurons--

00:39:09.347 --> 00:39:12.049
this vertical stack
is called a "layer."

00:39:12.049 --> 00:39:14.685
This a layer, that's a layer.

00:39:14.685 --> 00:39:16.220
And these little
nonlinear functions,

00:39:16.220 --> 00:39:19.056
which we haven't got into
yet, are called "activation

00:39:19.056 --> 00:39:22.727
functions," and we'll get to
why they are called that in just

00:39:22.727 --> 00:39:25.329
a second.

00:39:25.329 --> 00:39:30.868
And the input is called
an "input layer,"

00:39:30.868 --> 00:39:34.338
and I have the word "layer" in
double quotes because it's not

00:39:34.338 --> 00:39:35.673
really doing anything.

00:39:35.673 --> 00:39:39.310
It's just the input, but
we call it the input layer.

00:39:39.310 --> 00:39:42.546
And the very final thing that
produce a output is called

00:39:42.546 --> 00:39:45.683
the "output layer," obviously,
and everything in the middle is

00:39:45.683 --> 00:39:46.684
called a "hidden layer."

00:39:50.254 --> 00:39:53.391
So the final piece of
terminology is that when you

00:39:53.391 --> 00:39:55.259
have a layer like
this in which, say,

00:39:55.259 --> 00:39:58.729
three numbers are coming out
and there's another layer,

00:39:58.729 --> 00:40:01.966
if every neuron in this layer
is connected to every neuron

00:40:01.966 --> 00:40:04.735
in this layer, it's
called a "fully connected"

00:40:04.735 --> 00:40:05.803
or "dense layer."

00:40:05.803 --> 00:40:08.773
So for instance, here,
this arrow that's--

00:40:08.773 --> 00:40:11.442
whatever number is coming out--
let's say the number 3 is coming

00:40:11.442 --> 00:40:12.610
out of this thing here--

00:40:12.610 --> 00:40:16.013
that number 3 flows on
this arrow to this thing,

00:40:16.013 --> 00:40:18.115
flows on this arrow
to this neuron,

00:40:18.115 --> 00:40:20.151
and flows on this third
arrow to this neuron.

00:40:20.151 --> 00:40:21.018
That's what I mean.

00:40:21.018 --> 00:40:22.987
So every neuron--
its output is being

00:40:22.987 --> 00:40:26.190
sent to every neuron
in the following layer.

00:40:26.190 --> 00:40:29.293
We call it fully
connected or dense.

00:40:29.293 --> 00:40:32.663
And then if you look at
logistic regression--

00:40:32.663 --> 00:40:34.932
this just logistic
regression-- you can see,

00:40:34.932 --> 00:40:37.435
basically, logistic
regression is a neural network

00:40:37.435 --> 00:40:38.402
with no hidden layers.

00:40:40.671 --> 00:40:42.239
So in some sense,
logistic regression

00:40:42.239 --> 00:40:45.242
is almost the simplest possible
network you can think of--

00:40:45.242 --> 00:40:48.512
like, barely a neural network.

00:40:48.512 --> 00:40:50.181
It's got no hidden layers.

00:40:50.181 --> 00:40:52.383
That's what makes it
logistic regression.

00:40:52.383 --> 00:40:55.019
And so as you might
have guessed by now,

00:40:55.019 --> 00:40:58.856
deep learning is just neural
networks with lots and lots of--

00:40:58.856 --> 00:40:59.390
of what?

00:40:59.390 --> 00:41:00.090
AUDIENCE: Layers.

00:41:00.090 --> 00:41:02.359
RAMA RAMAKRISHNAN: Yes, layers.

00:41:02.359 --> 00:41:04.728
So here are a few.

00:41:04.728 --> 00:41:08.466
And by the way, these are
not even considered all that

00:41:08.466 --> 00:41:14.104
impressive these days, but I put
them up because this thing here

00:41:14.104 --> 00:41:18.409
is called "ResNet," and it's
famous because the ResNet neural

00:41:18.409 --> 00:41:23.347
network was, I think, the first
network to surpass human-level

00:41:23.347 --> 00:41:27.151
performance in image
classification.

00:41:27.151 --> 00:41:28.919
It's sort of like
the Skynet [CHUCKLES]

00:41:28.919 --> 00:41:30.688
of image classification.

00:41:30.688 --> 00:41:32.456
It surpassed
human-level performance.

00:41:32.456 --> 00:41:34.792
And I'm putting it up here
because we will actually work

00:41:34.792 --> 00:41:36.927
with ResNet on next Wednesday.

00:41:36.927 --> 00:41:38.462
And we'll actually
take ResNet, we'll

00:41:38.462 --> 00:41:41.899
fine-tune it, and solve
a real problem in class.

00:41:41.899 --> 00:41:45.102
All right, so it's got
lots and lots of layers.

00:41:45.102 --> 00:41:46.971
Now let's turn to these
activation functions.

00:41:46.971 --> 00:41:49.874
We've been ignoring
these little guys so far.

00:41:49.874 --> 00:41:53.577
So the activation function
at a node is, first of all,

00:41:53.577 --> 00:41:56.347
it's a function that
receives a single number

00:41:56.347 --> 00:41:57.915
and outputs a single number.

00:41:57.915 --> 00:41:59.917
It's not very complicated.

00:41:59.917 --> 00:42:02.019
It receives, basically, this--

00:42:02.019 --> 00:42:04.522
this is a linear function, which
receives all these inputs--

00:42:04.522 --> 00:42:06.657
it could be 10
inputs, 1,000 inputs--

00:42:06.657 --> 00:42:08.959
runs it through a linear
function, outputs a number.

00:42:08.959 --> 00:42:10.995
And that single
number-- a scalar--

00:42:10.995 --> 00:42:14.732
goes in here, and it comes
out as another single number.

00:42:14.732 --> 00:42:16.534
Just remember that.

00:42:16.534 --> 00:42:18.702
And so these are some of
the most common activation

00:42:18.702 --> 00:42:19.203
functions.

00:42:19.203 --> 00:42:21.472
In fact, the sigmoid we
saw, which is actually

00:42:21.472 --> 00:42:23.140
be used for the
output, is actually

00:42:23.140 --> 00:42:26.710
a kind of activation function
where a single number comes in

00:42:26.710 --> 00:42:29.813
and it gets mapped into this
curve because of this thing.

00:42:29.813 --> 00:42:32.249
So the single number
that comes in is a,

00:42:32.249 --> 00:42:33.817
and it gets transformed
as 1 divided

00:42:33.817 --> 00:42:35.319
by 1 plus e raised to minus a.

00:42:35.319 --> 00:42:38.389
And you get a shape like this,
and it's called the "sigmoid

00:42:38.389 --> 00:42:39.990
activation function."

00:42:39.990 --> 00:42:43.627
And as you can see here,
for very small values,

00:42:43.627 --> 00:42:47.798
for very negative values, it's
going to be pretty close to 0,

00:42:47.798 --> 00:42:50.234
meaning it won't get activated.

00:42:50.234 --> 00:42:52.570
And for very, very
large values, it's

00:42:52.570 --> 00:42:55.306
going to be pretty close to 1.

00:42:55.306 --> 00:42:58.108
All the action
happens in the middle.

00:42:58.108 --> 00:43:00.010
When your values are
somewhere in this range,

00:43:00.010 --> 00:43:03.881
there's a dramatic
increases in what comes out,

00:43:03.881 --> 00:43:05.349
so that little
thing in the middle

00:43:05.349 --> 00:43:08.619
is the sweet spot
for these functions.

00:43:08.619 --> 00:43:11.722
And this-- I'm almost
embarrassed to call it

00:43:11.722 --> 00:43:13.757
an activation function
because it's literally not

00:43:13.757 --> 00:43:14.658
doing anything.

00:43:14.658 --> 00:43:17.461
It's sort of gettinig
a nice label for free.

00:43:17.461 --> 00:43:19.163
Basically, it says
you just get a number,

00:43:19.163 --> 00:43:20.798
you just pass it straight along.

00:43:20.798 --> 00:43:22.333
It's a linear
activation function,

00:43:22.333 --> 00:43:25.569
but just for completeness,
I want to put it here.

00:43:25.569 --> 00:43:29.440
And then we come to the
hero of deep learning, which

00:43:29.440 --> 00:43:32.443
is the rectified linear unit.

00:43:32.443 --> 00:43:36.113
Rectified linear unit--
it's called ReLU.

00:43:36.113 --> 00:43:38.515
And ReLU is going to become
part of your vocabulary very,

00:43:38.515 --> 00:43:39.650
very quickly.

00:43:39.650 --> 00:43:42.453
And so ReLU is actually a
very interesting function.

00:43:42.453 --> 00:43:44.755
So you write it as
maximum of whatever number

00:43:44.755 --> 00:43:48.092
and 0, which is
another way of saying

00:43:48.092 --> 00:43:51.629
if the number is positive,
just send it along unchanged.

00:43:51.629 --> 00:43:55.065
If the number is negative,
send a 0 instead--

00:43:55.065 --> 00:43:56.600
squash it to 0.

00:43:56.600 --> 00:43:59.436
So which means if the number
is negative, nothing happens.

00:43:59.436 --> 00:44:03.207
If the number is
positive, it breaks up.

00:44:03.207 --> 00:44:04.875
So what happens is
that you could have

00:44:04.875 --> 00:44:07.578
a very complicated linear
function with millions

00:44:07.578 --> 00:44:09.680
of variables, and then
it puts a single number,

00:44:09.680 --> 00:44:12.016
and that number, unfortunately,
happens to be negative--

00:44:12.016 --> 00:44:13.117
the ReLU is not impressed.

00:44:13.117 --> 00:44:15.552
It's going to send a 0 out.

00:44:15.552 --> 00:44:17.488
It's a very simple function.

00:44:17.488 --> 00:44:20.557
And many, many folks who
have been in deep learning

00:44:20.557 --> 00:44:24.695
for a long, long time believe
that the use of the ReLUs

00:44:24.695 --> 00:44:27.297
is one of the key
factors that led

00:44:27.297 --> 00:44:29.566
to the amazing success
of deep learning

00:44:29.566 --> 00:44:32.469
because it's got some very
interesting properties, which

00:44:32.469 --> 00:44:36.140
we'll get to,
hopefully, on Wednesday.

00:44:36.140 --> 00:44:41.512
So the shorthand here is that
whenever you see this thing,

00:44:41.512 --> 00:44:44.148
it's just a linear activation--
linear function followed by just

00:44:44.148 --> 00:44:46.917
sending it straight out.

00:44:46.917 --> 00:44:48.318
If I put a ReLU
in here, I'm going

00:44:48.318 --> 00:44:51.789
to denote it like that,
which mimics the graph--

00:44:51.789 --> 00:44:52.556
how it looks.

00:44:52.556 --> 00:44:56.860
And if I put a sigmoid, I'm just
going to use this thing here.

00:44:56.860 --> 00:45:00.230
Just a visual shorthand.

00:45:00.230 --> 00:45:03.801
There are many other activation
functions, by the way.

00:45:03.801 --> 00:45:07.271
There's something called the
tanh function, the leaky ReLU,

00:45:07.271 --> 00:45:08.639
the GELU, the swish.

00:45:08.639 --> 00:45:11.875
I mean, it's like a menagerie
of activation functions

00:45:11.875 --> 00:45:14.278
because very often researchers
will be like, well, I don't

00:45:14.278 --> 00:45:15.579
like this activation function.

00:45:15.579 --> 00:45:17.147
Here's a little modified
version of the function,

00:45:17.147 --> 00:45:19.116
which is going to be
better for certain things.

00:45:19.116 --> 00:45:22.352
So people's research
creativity is, at this point,

00:45:22.352 --> 00:45:25.155
has gone unhinged,
so lots of options.

00:45:25.155 --> 00:45:29.059
But if you just stick to the
ReLU for your hidden layers,

00:45:29.059 --> 00:45:31.361
you can basically get
anything done, practically.

00:45:31.361 --> 00:45:33.197
You don't have to worry
about anything else.

00:45:33.197 --> 00:45:37.301
So we'll only focus on ReLUs
for all the intermediate stuff.

00:45:37.301 --> 00:45:38.435
Yeah?

00:45:38.435 --> 00:45:40.971
AUDIENCE: Yeah, how do you gauge
which activation function is

00:45:40.971 --> 00:45:42.406
more suited for your use case?

00:45:42.406 --> 00:45:44.308
RAMA RAMAKRISHNAN: Yeah,
so the rule of thumb

00:45:44.308 --> 00:45:46.810
here is that for
your hidden layers,

00:45:46.810 --> 00:45:49.646
use ReLUs because
empirically, we have seen

00:45:49.646 --> 00:45:51.849
that they do an amazing job.

00:45:51.849 --> 00:45:53.550
For your output layer--

00:45:53.550 --> 00:45:54.818
your very final thing--

00:45:54.818 --> 00:45:57.521
you actually don't have a choice
because what you have to use

00:45:57.521 --> 00:46:00.023
depends on what kind of
output you have to work with.

00:46:00.023 --> 00:46:01.692
If it's an output
which is a probability

00:46:01.692 --> 00:46:04.695
number between 0 and 1,
you have to use a sigmoid.

00:46:04.695 --> 00:46:07.531
If it is, say, 10 numbers,
all of which have to be

00:46:07.531 --> 00:46:09.633
probabilities and they
have to add up to 1,

00:46:09.633 --> 00:46:11.568
you got to use something
called the "softmax,"

00:46:11.568 --> 00:46:12.970
which we'll get to on Wednesday.

00:46:12.970 --> 00:46:14.438
So it really depends
on the output,

00:46:14.438 --> 00:46:16.173
and the nature of the
output dictates what

00:46:16.173 --> 00:46:17.341
you use in the output layer.

00:46:19.977 --> 00:46:21.845
So coming back to
this-- so if you

00:46:21.845 --> 00:46:27.317
want to design a deep neural
network, the input is the input,

00:46:27.317 --> 00:46:29.553
the output is the
output, and so you

00:46:29.553 --> 00:46:30.788
get to choose everything else.

00:46:30.788 --> 00:46:33.290
You get to choose the number
of hidden layers, the number

00:46:33.290 --> 00:46:36.226
of neurons in each layer,
the activation functions

00:46:36.226 --> 00:46:38.195
you're going to use, and--

00:46:38.195 --> 00:46:39.696
for the hidden
layers-- and then you

00:46:39.696 --> 00:46:41.298
have to make sure
that what you choose

00:46:41.298 --> 00:46:43.267
for the output layer
matches the kind of output

00:46:43.267 --> 00:46:44.768
you want to generate.

00:46:44.768 --> 00:46:47.838
So this is all in your hands.

00:46:47.838 --> 00:46:51.742
You decide what happens,
but you will-- there's

00:46:51.742 --> 00:46:53.844
a lot of guidance for how
to do these things, which

00:46:53.844 --> 00:46:56.146
we'll cover as we go along.

00:46:56.146 --> 00:46:57.781
Did you have a question?

00:46:57.781 --> 00:47:00.918
AUDIENCE: Kind of, but
I guess I'll do it.

00:47:00.918 --> 00:47:05.789
Is there also exploration
in dynamic setting

00:47:05.789 --> 00:47:08.725
of layers so that our
user is determining

00:47:08.725 --> 00:47:11.762
the layers that the system
itself [INAUDIBLE] layers.

00:47:11.762 --> 00:47:13.130
RAMA RAMAKRISHNAN:
Yeah, so there

00:47:13.130 --> 00:47:15.666
is a whole field called
neural architecture search--

00:47:15.666 --> 00:47:19.002
NAS-- where we can actually
try a whole bunch of different

00:47:19.002 --> 00:47:22.673
architectures and then use some
optimization-- and, in fact,

00:47:22.673 --> 00:47:25.676
reinforcement learning, which
we won't get to in this class--

00:47:25.676 --> 00:47:27.778
as a way to figure out
really good architectures

00:47:27.778 --> 00:47:29.746
for any particular problem.

00:47:29.746 --> 00:47:34.718
But the question of, OK, when
I'm training a model with

00:47:34.718 --> 00:47:37.321
a particular kind of data, the
first pass through the training

00:47:37.321 --> 00:47:39.489
data, I'm going to use two
layers-- the second pass,

00:47:39.489 --> 00:47:41.058
I'm going to do 7 layers--

00:47:41.058 --> 00:47:42.626
that is not done.

00:47:42.626 --> 00:47:44.461
And the reason it's
not done is because

00:47:44.461 --> 00:47:46.096
of certain other
constraints we have

00:47:46.096 --> 00:47:48.031
and how we can do
the optimization

00:47:48.031 --> 00:47:49.967
and the gradient descent
and stuff like that.

00:47:49.967 --> 00:47:52.936
But what you can do-- and we
will look at this thing called

00:47:52.936 --> 00:47:54.271
"dropout"--

00:47:54.271 --> 00:47:56.874
for certain layers, you
can actually, for each time

00:47:56.874 --> 00:48:00.477
you run it through the network,
you can decide on this layer,

00:48:00.477 --> 00:48:02.079
I'm not going to
use all the nodes.

00:48:02.079 --> 00:48:04.648
I'm going to drop out a
few other nodes randomly.

00:48:04.648 --> 00:48:06.216
And it's a very
effective technique

00:48:06.216 --> 00:48:07.584
to prevent
overfitting, and we'll

00:48:07.584 --> 00:48:10.020
come to that a little later on.

00:48:10.020 --> 00:48:11.488
Yeah?

00:48:11.488 --> 00:48:14.958
AUDIENCE: One question
regarding neural networks

00:48:14.958 --> 00:48:16.260
is about the coefficients.

00:48:16.260 --> 00:48:18.996
If it's a small size,
is it fine on its own?

00:48:18.996 --> 00:48:22.099
Or do we have as the
coefficients [INAUDIBLE].

00:48:22.099 --> 00:48:23.967
RAMA RAMAKRISHNAN: No,
the whole trick here--

00:48:23.967 --> 00:48:27.838
the whole name of the game--
is we use the training data

00:48:27.838 --> 00:48:30.107
and something called
a "loss function,"

00:48:30.107 --> 00:48:32.843
which I'll get to on Wednesday,
along with an optimization

00:48:32.843 --> 00:48:36.780
algorithm so that the network
figures out by itself what

00:48:36.780 --> 00:48:39.149
the weights need to be, what
the coefficients need to be

00:48:39.149 --> 00:48:42.019
so as to minimize
prediction error,

00:48:42.019 --> 00:48:43.453
and that's the whole thing.

00:48:43.453 --> 00:48:46.056
The magic here is that we
don't have to do anything.

00:48:46.056 --> 00:48:48.125
We only have to set
it up, sit back--

00:48:48.125 --> 00:48:49.493
often for many hours--

00:48:49.493 --> 00:48:51.728
and watch it do its thing.

00:48:51.728 --> 00:48:52.829
Yep.

00:48:52.829 --> 00:48:54.197
AUDIENCE: Just
one last question.

00:48:54.197 --> 00:48:56.099
You mentioned nodes
just now when you were

00:48:56.099 --> 00:48:57.901
answering Rolando's question.

00:48:57.901 --> 00:49:00.804
Can you just confirm
exactly what a node is?

00:49:00.804 --> 00:49:03.307
I have an idea that it's
basically any circle,

00:49:03.307 --> 00:49:05.442
but you just added
a lot more detail.

00:49:05.442 --> 00:49:05.642
RAMA RAMAKRISHNAN: Sure.

00:49:05.642 --> 00:49:07.177
No, when I'm
referring to a node,

00:49:07.177 --> 00:49:08.679
I'm literally
referring to something

00:49:08.679 --> 00:49:12.382
like this, which think of it
as a linear function followed

00:49:12.382 --> 00:49:14.618
by a nonlinear activation.

00:49:14.618 --> 00:49:16.320
So it receives a
bunch of inputs,

00:49:16.320 --> 00:49:17.888
runs it through a
linear function,

00:49:17.888 --> 00:49:20.624
and pass it through, like, a
ReLU or a sigmoid or something,

00:49:20.624 --> 00:49:22.125
and out pops a number.

00:49:22.125 --> 00:49:25.562
So in general, a node will have
many numbers potentially coming

00:49:25.562 --> 00:49:28.432
in but only one
number going out.

00:49:28.432 --> 00:49:30.000
Now, that one number
may get copied

00:49:30.000 --> 00:49:33.070
to every node in the
next layer, but what

00:49:33.070 --> 00:49:36.540
comes out of that particular
node is just a single number.

00:49:36.540 --> 00:49:42.746
All right, so let's use a DNN
for our interview example.

00:49:42.746 --> 00:49:44.848
So in this problem,
we had two inputs--

00:49:44.848 --> 00:49:46.283
GP and experience.

00:49:46.283 --> 00:49:48.285
The output variable has
to be between 0 and 1

00:49:48.285 --> 00:49:49.953
because you have to predict the
probability that someone will

00:49:49.953 --> 00:49:51.388
get called for an interview.

00:49:51.388 --> 00:49:54.024
So the output size is fixed--

00:49:54.024 --> 00:49:57.861
sorry-- the input size is
fixed, the output is fixed.

00:49:57.861 --> 00:50:00.364
So since it's really only
the very first network

00:50:00.364 --> 00:50:04.267
we're actually playing with,
let's just start simple.

00:50:04.267 --> 00:50:06.003
We'll just have
one hidden layer,

00:50:06.003 --> 00:50:08.772
and we'll have three neurons.

00:50:08.772 --> 00:50:12.109
And as I mentioned to
Tomaso's question from before,

00:50:12.109 --> 00:50:15.612
if you are choosing activation
functions in the hidden layers,

00:50:15.612 --> 00:50:17.581
just go with the
ReLU as a default.

00:50:17.581 --> 00:50:19.750
It usually works really
well out of the box,

00:50:19.750 --> 00:50:20.917
so we'll just use a ReLU.

00:50:20.917 --> 00:50:23.353
And since the output has
to be between 0 and 1,

00:50:23.353 --> 00:50:24.588
we don't have a choice.

00:50:24.588 --> 00:50:27.491
We have to use a sigmoid
for the output layer.

00:50:27.491 --> 00:50:28.025
That's it.

00:50:28.025 --> 00:50:30.594
So those are our design choices.

00:50:30.594 --> 00:50:33.196
And when we do that, this
how it's looked like.

00:50:33.196 --> 00:50:35.932
We have two inputs-- x1
and x2, GP and experience--

00:50:35.932 --> 00:50:39.136
and then it goes through
these three ReLUs,

00:50:39.136 --> 00:50:41.338
and then out comes
these three numbers,

00:50:41.338 --> 00:50:43.807
and they pass through a sigmoid,
and we get a probability y

00:50:43.807 --> 00:50:44.875
at the end.

00:50:44.875 --> 00:50:47.411
All right, quick
question-- concept check--

00:50:47.411 --> 00:50:50.881
how many weights-- how many
parameters, both weights

00:50:50.881 --> 00:50:52.783
and biases, does
the network have?

00:50:52.783 --> 00:50:53.984
Let's take a moment to count.

00:51:11.168 --> 00:51:12.302
All right, any guesses?

00:51:15.572 --> 00:51:16.973
Yeah?

00:51:16.973 --> 00:51:18.108
AUDIENCE: 12.

00:51:18.108 --> 00:51:20.077
RAMA RAMAKRISHNAN: I
think you're almost there.

00:51:23.647 --> 00:51:26.149
Are folks going to be doing a
non-binary search on this now?

00:51:26.149 --> 00:51:27.818
OK.

00:51:27.818 --> 00:51:29.553
AUDIENCE: Is it 9?

00:51:29.553 --> 00:51:31.021
RAMA RAMAKRISHNAN: No.

00:51:31.021 --> 00:51:31.521
Yes?

00:51:31.521 --> 00:51:32.055
AUDIENCE: 13.

00:51:32.055 --> 00:51:33.990
RAMA RAMAKRISHNAN:
Yes, very good.

00:51:33.990 --> 00:51:35.592
So that's 13.

00:51:35.592 --> 00:51:37.994
And my guess is that the
reason you came up with 12--

00:51:37.994 --> 00:51:40.163
and I made the same mistake--
that's why I know it--

00:51:40.163 --> 00:51:42.632
is so you probably forgot
this green thing here.

00:51:45.969 --> 00:51:49.473
So what folks often
forget is the bias.

00:51:49.473 --> 00:51:51.108
We all count the things.

00:51:51.108 --> 00:51:54.244
And the easy way to do it
is say, OK, two things here,

00:51:54.244 --> 00:51:59.416
three things here, 2 times 3 is
6, 3 times 1 is 3, another 9.

00:51:59.416 --> 00:52:04.020
And then you have to add up all
the intercepts, so you get 13.

00:52:04.020 --> 00:52:07.023
And so when we get to very
complicated networks--

00:52:07.023 --> 00:52:08.592
the first two or
three times you work

00:52:08.592 --> 00:52:10.660
with very complex networks--

00:52:10.660 --> 00:52:12.696
and we'll do it
starting very soon--

00:52:12.696 --> 00:52:14.598
just get into the habit
of hand-calculating

00:52:14.598 --> 00:52:17.267
the number of parameters just to
make sure you understand what's

00:52:17.267 --> 00:52:17.801
going on.

00:52:17.801 --> 00:52:19.469
Once you get it right
a couple of times,

00:52:19.469 --> 00:52:20.871
you don't have to do it anymore.

00:52:20.871 --> 00:52:22.572
The first couple of
times, hand-calculate

00:52:22.572 --> 00:52:24.608
to make sure you get it.

00:52:24.608 --> 00:52:25.342
So, yeah.

00:52:25.342 --> 00:52:29.679
So let's say that we have
trained this network using

00:52:29.679 --> 00:52:31.481
techniques which we'll
cover on Wednesday,

00:52:31.481 --> 00:52:34.417
and it comes back to you after
training and says, OK, these are

00:52:34.417 --> 00:52:35.485
the optimal--

00:52:35.485 --> 00:52:38.488
the best values for the
weights and the biases

00:52:38.488 --> 00:52:39.689
that I have found.

00:52:39.689 --> 00:52:42.159
So now your network
is ready for action.

00:52:42.159 --> 00:52:43.827
It's ready to be used.

00:52:43.827 --> 00:52:45.729
And so what you
can do is-- let's

00:52:45.729 --> 00:52:49.533
say that you want to
predict with this network.

00:52:49.533 --> 00:52:53.069
If you have x1 and x2,
what comes out of--

00:52:53.069 --> 00:52:54.971
so what comes out
of this top neuron?

00:52:54.971 --> 00:52:56.339
Let's call it a1.

00:52:56.339 --> 00:52:59.075
It's basically this.

00:52:59.075 --> 00:53:00.710
That's what's coming
out of this thing.

00:53:00.710 --> 00:53:03.547
For any x1 and x2, this is
what's coming out, similarly

00:53:03.547 --> 00:53:06.416
for a2 and a3.

00:53:06.416 --> 00:53:08.518
And then what comes
out at the very end

00:53:08.518 --> 00:53:12.189
is basically a1 times
that, plus a2 times that,

00:53:12.189 --> 00:53:14.758
plus a3 times that, plus 0.05.

00:53:14.758 --> 00:53:16.793
And the whole thing gets
run through the sigmoid,

00:53:16.793 --> 00:53:18.929
and that's what you get.

00:53:18.929 --> 00:53:20.864
So this slide and
the one before, just

00:53:20.864 --> 00:53:22.332
make sure you look
at it afterwards

00:53:22.332 --> 00:53:25.068
and-- to make sure you totally
understand the mechanics of it

00:53:25.068 --> 00:53:27.204
because this is
really important.

00:53:27.204 --> 00:53:29.439
And if you don't fully
understand, like, internalize

00:53:29.439 --> 00:53:31.708
the mechanics, when we get
to things like transformers,

00:53:31.708 --> 00:53:32.876
it's going to get hard.

00:53:32.876 --> 00:53:35.178
So just make sure it's, like,
automatic at this point.

00:53:35.178 --> 00:53:36.279
It should be reflexive.

00:53:38.949 --> 00:53:41.184
So, yeah, and so when you
want to predict anything,

00:53:41.184 --> 00:53:42.552
just run some
numbers through it.

00:53:42.552 --> 00:53:45.188
You get all these things,
and boom, you calculate it.

00:53:45.188 --> 00:53:46.523
It turns out to be 22.6.

00:53:46.523 --> 00:53:48.391
That's the answer.

00:53:48.391 --> 00:53:52.195
All right, so I just want to say
that-- let's say that you built

00:53:52.195 --> 00:53:56.499
this network and now you're
like, hey, given any x1 and x2,

00:53:56.499 --> 00:53:59.936
I can come up with a y, but
I'm feeling a little mathy.

00:53:59.936 --> 00:54:01.671
Can you actually write
down the function?

00:54:01.671 --> 00:54:03.240
Yeah, you can write
down the function.

00:54:03.240 --> 00:54:04.374
This is what it looks like.

00:54:07.310 --> 00:54:10.413
Super interpretable, right?

00:54:10.413 --> 00:54:14.184
So this goes to the comment
that, you made earlier on,

00:54:14.184 --> 00:54:17.587
where the act of
depicting something

00:54:17.587 --> 00:54:21.091
using this graphical layout
makes it so much easier

00:54:21.091 --> 00:54:23.627
to reason with
and to think about

00:54:23.627 --> 00:54:27.197
compared to trying to figure
out what this function is doing.

00:54:27.197 --> 00:54:30.967
The other point I want to
make is that this contrast--

00:54:30.967 --> 00:54:33.370
what we just saw with the
logistic regression thing we saw

00:54:33.370 --> 00:54:35.505
earlier, which was
this little function--

00:54:35.505 --> 00:54:40.076
and so here, even
this simple network

00:54:40.076 --> 00:54:41.778
with just three hidden layers--

00:54:41.778 --> 00:54:44.414
sorry, three nodes in
that single hidden layer--

00:54:44.414 --> 00:54:47.550
it's so much more complicated
than the logistic regression

00:54:47.550 --> 00:54:48.251
model--

00:54:48.251 --> 00:54:50.186
so much more complicated.

00:54:50.186 --> 00:54:53.657
And it is from this
complexity springs

00:54:53.657 --> 00:54:56.660
the ability of these networks to
do, basically, magical things.

00:54:56.660 --> 00:54:58.428
And that's where the
complexity comes from.

00:54:58.428 --> 00:55:00.730
That's where the
magic comes from.

00:55:00.730 --> 00:55:02.732
And here, in this case,
the number of variables

00:55:02.732 --> 00:55:03.566
hasn't even changed.

00:55:03.566 --> 00:55:05.869
It's still only two.

00:55:05.869 --> 00:55:07.504
But we can go from
the two inputs

00:55:07.504 --> 00:55:10.140
to the one output in
very complicated ways

00:55:10.140 --> 00:55:12.842
as long as we know how to train
these networks the right way.

00:55:12.842 --> 00:55:16.913
That's the secret sauce, which
we'll spend a lot of time on.

00:55:16.913 --> 00:55:19.282
So, yeah, to summarize,
this is what we have--

00:55:19.282 --> 00:55:20.517
it's a deep neural network.

00:55:20.517 --> 00:55:23.219
By the way, this kind of network
where things just flow from left

00:55:23.219 --> 00:55:27.290
to right is called a
"feedforward neural network,"

00:55:27.290 --> 00:55:30.060
in contrast to some other kinds
of networks called "recurrent

00:55:30.060 --> 00:55:33.396
networks," which we won't
get to in this class

00:55:33.396 --> 00:55:37.367
because transformers have
actually proven to be much more

00:55:37.367 --> 00:55:39.202
capable than recurrent networks.

00:55:39.202 --> 00:55:41.071
And those have become
the norm, so we'll just

00:55:41.071 --> 00:55:42.872
focus on those instead.

00:55:42.872 --> 00:55:45.342
And so this arrangement
of neurons into layers

00:55:45.342 --> 00:55:47.210
and activation functions
and all that stuff--

00:55:47.210 --> 00:55:49.512
this called the "architecture"
of the neural network.

00:55:49.512 --> 00:55:51.614
And as you will see later
on, the transformer--

00:55:51.614 --> 00:55:54.117
the famous transformer
network-- is just

00:55:54.117 --> 00:55:57.153
an example of a particular
neural network architecture,

00:55:57.153 --> 00:55:59.389
much like convolutional
neural networks,

00:55:59.389 --> 00:56:02.025
which we'll get to next
week for computer vision,

00:56:02.025 --> 00:56:04.461
or another example of
a particular network

00:56:04.461 --> 00:56:05.762
of the architecture.

00:56:05.762 --> 00:56:07.230
So we will focus
on transformers.

00:56:07.230 --> 00:56:08.998
They are a particular
kind of architecture.

00:56:08.998 --> 00:56:11.901
All right, so in summary,
that's what we have.

00:56:11.901 --> 00:56:14.371
You get to choose the hidden
layers and neuron's activation

00:56:14.371 --> 00:56:15.772
functions and stuff like that.

00:56:15.772 --> 00:56:18.174
The inputs and the outputs are
what you have to work with.

00:56:18.174 --> 00:56:20.443
And so we will
actually take this idea

00:56:20.443 --> 00:56:25.882
and then use it to actually
solve a problem from start

00:56:25.882 --> 00:56:27.751
to finish on Wednesday.

00:56:27.751 --> 00:56:28.585
So I think I'm done.

00:56:28.585 --> 00:56:30.353
I give you three minutes
back of your day.

00:56:30.353 --> 00:56:30.987
Thank you.

00:56:30.987 --> 00:56:34.324
[APPLAUSE]

