WEBVTT

00:00:00.000 --> 00:00:01.968 align:middle line:90%
[RUSTLING]

00:00:01.968 --> 00:00:03.936 align:middle line:90%
[SQUEAKING]

00:00:03.936 --> 00:00:04.920 align:middle line:90%
[CLICKING]

00:00:04.920 --> 00:00:17.060 align:middle line:90%


00:00:17.060 --> 00:00:20.680 align:middle line:84%
RAMA RAMAKRISHNAN: OK, so today
we start the natural language

00:00:20.680 --> 00:00:22.200 align:middle line:90%
processing sequence.

00:00:22.200 --> 00:00:24.260 align:middle line:84%
And so just to give
you a quick idea,

00:00:24.260 --> 00:00:28.440 align:middle line:84%
we're going to start with what's
called vectorization, and then

00:00:28.440 --> 00:00:29.700 align:middle line:90%
the bag of words model.

00:00:29.700 --> 00:00:32.159 align:middle line:84%
And then we'll spend a fair
amount of time on a Colab.

00:00:32.159 --> 00:00:34.560 align:middle line:84%
And then on Wednesday, we
talk about these things

00:00:34.560 --> 00:00:37.000 align:middle line:84%
called embeddings, which you
will come to appreciate over

00:00:37.000 --> 00:00:41.160 align:middle line:84%
the next couple of weeks,
form the core atomic unit

00:00:41.160 --> 00:00:44.120 align:middle line:84%
of all modern natural
language processing,

00:00:44.120 --> 00:00:46.960 align:middle line:84%
and for that matter,
vision processing as well.

00:00:46.960 --> 00:00:49.000 align:middle line:84%
And then the
following week, we'll

00:00:49.000 --> 00:00:51.092 align:middle line:84%
do transformers, two
lectures on transformers.

00:00:51.092 --> 00:00:52.800 align:middle line:84%
We'll get into the
theory, and then we'll

00:00:52.800 --> 00:00:54.400 align:middle line:84%
get into a bunch
of applications.

00:00:54.400 --> 00:00:58.860 align:middle line:84%
And then lectures 9 and
10 will be all about LLMs.

00:00:58.860 --> 00:01:00.980 align:middle line:84%
So it's going to
be a lot of fun.

00:01:00.980 --> 00:01:03.973 align:middle line:84%
This is one of my favorite
segments of the class.

00:01:03.973 --> 00:01:06.140 align:middle line:84%
Of course, truth be told,
every segment of the class

00:01:06.140 --> 00:01:08.540 align:middle line:84%
is my favorite,
so don't judge me.

00:01:08.540 --> 00:01:10.580 align:middle line:90%
All right, so let's get going.

00:01:10.580 --> 00:01:15.100 align:middle line:84%
So why natural
language processing?

00:01:15.100 --> 00:01:18.080 align:middle line:84%
In some sense, the things I have
on the slide here are obvious,

00:01:18.080 --> 00:01:20.980 align:middle line:84%
but I think it's actually
worth reminding ourselves

00:01:20.980 --> 00:01:24.780 align:middle line:84%
of how important text
is for everything we do.

00:01:24.780 --> 00:01:27.980 align:middle line:84%
Obviously, human knowledge
is mostly encoded as text.

00:01:27.980 --> 00:01:29.800 align:middle line:90%
The internet is mostly text.

00:01:29.800 --> 00:01:33.180 align:middle line:84%
At least this was true until the
advent of TikTok and YouTube.

00:01:33.180 --> 00:01:35.500 align:middle line:84%
And human communication
is mostly text.

00:01:35.500 --> 00:01:39.820 align:middle line:84%
And cultural production--
movies, books, arts and so on,

00:01:39.820 --> 00:01:41.860 align:middle line:90%
so much of it is so text heavy.

00:01:41.860 --> 00:01:45.740 align:middle line:84%
And so, in some sense,
text forms not just

00:01:45.740 --> 00:01:48.640 align:middle line:84%
a big chunk of all the
media that's out there,

00:01:48.640 --> 00:01:50.565 align:middle line:84%
but it also happens to
be the way in which we

00:01:50.565 --> 00:01:52.440 align:middle line:84%
think and communicate
and so on and so forth.

00:01:52.440 --> 00:01:55.760 align:middle line:84%
So its primacy
is, in my opinion,

00:01:55.760 --> 00:01:59.030 align:middle line:84%
unparalleled in how we
think about the world.

00:01:59.030 --> 00:02:01.870 align:middle line:84%
And so the tantalizing
possibility is that,

00:02:01.870 --> 00:02:05.470 align:middle line:84%
imagine if we had an AI system
which could just read and quote

00:02:05.470 --> 00:02:08.630 align:middle line:84%
unquote, "understand"
all this text.

00:02:08.630 --> 00:02:12.510 align:middle line:84%
And so you can imagine such a
system reading all of PubMed,

00:02:12.510 --> 00:02:14.250 align:middle line:84%
reading all the
medical literature,

00:02:14.250 --> 00:02:18.230 align:middle line:84%
and then coming back and saying,
for this particular disease,

00:02:18.230 --> 00:02:20.190 align:middle line:84%
this particular
protein is actually

00:02:20.190 --> 00:02:21.570 align:middle line:90%
the malfunctioning protein.

00:02:21.570 --> 00:02:22.950 align:middle line:84%
And for that, that
small molecule

00:02:22.950 --> 00:02:25.750 align:middle line:84%
is going to dock into the
protein and cure the disease.

00:02:25.750 --> 00:02:28.167 align:middle line:84%
And you didn't know this-- it
came back and told you that.

00:02:28.167 --> 00:02:29.430 align:middle line:90%
Wouldn't it be unbelievable?

00:02:29.430 --> 00:02:32.670 align:middle line:84%
So my feeling is that such
things are going to happen.

00:02:32.670 --> 00:02:34.350 align:middle line:84%
It's just that it's
not going to happen

00:02:34.350 --> 00:02:36.050 align:middle line:90%
soon enough for my lifetime.

00:02:36.050 --> 00:02:37.910 align:middle line:84%
But perhaps it'll
happen in yours.

00:02:37.910 --> 00:02:39.970 align:middle line:90%
All right, so let's continue.

00:02:39.970 --> 00:02:42.390 align:middle line:84%
So NLP is in action
all around us.

00:02:42.390 --> 00:02:45.950 align:middle line:84%
According to Google,
apparently Google autocomplete,

00:02:45.950 --> 00:02:48.750 align:middle line:84%
which uses a fair
bit of NLP, saves

00:02:48.750 --> 00:02:51.970 align:middle line:84%
200 years of typing time,
apparently, every day.

00:02:51.970 --> 00:02:54.342 align:middle line:90%


00:02:54.342 --> 00:02:56.050 align:middle line:84%
I wasn't very impressed
with this number,

00:02:56.050 --> 00:02:57.930 align:middle line:84%
frankly, because
billions of searches

00:02:57.930 --> 00:03:00.370 align:middle line:84%
are being done every day,
and I'm like, only 200 years?

00:03:00.370 --> 00:03:03.370 align:middle line:84%
So anyway, but I think the
more important point is

00:03:03.370 --> 00:03:05.530 align:middle line:90%
that it made mobile possible.

00:03:05.530 --> 00:03:07.930 align:middle line:84%
If you didn't have
autocomplete, people

00:03:07.930 --> 00:03:10.830 align:middle line:84%
would not be typing and
pecking on their keyboards.

00:03:10.830 --> 00:03:12.350 align:middle line:90%
It's going to be much worse.

00:03:12.350 --> 00:03:14.990 align:middle line:84%
You would have had a hugely
dampening effect on e-commerce,

00:03:14.990 --> 00:03:15.810 align:middle line:90%
for instance.

00:03:15.810 --> 00:03:18.090 align:middle line:84%
So this humble
little autocomplete

00:03:18.090 --> 00:03:21.235 align:middle line:84%
has incredible, incredible
impact on the world economy.

00:03:21.235 --> 00:03:22.610 align:middle line:84%
And the other
thing which I heard

00:03:22.610 --> 00:03:24.350 align:middle line:84%
about, I'm not sure
if it's 100% true,

00:03:24.350 --> 00:03:27.690 align:middle line:84%
but it's an interesting example,
apparently the very first iPhone

00:03:27.690 --> 00:03:31.890 align:middle line:84%
keyboard that came out, the soft
keyboard, not the hard keyboard,

00:03:31.890 --> 00:03:36.090 align:middle line:84%
they had some very basic
word continuation prediction

00:03:36.090 --> 00:03:36.910 align:middle line:90%
going on.

00:03:36.910 --> 00:03:40.850 align:middle line:84%
And so when you
start typing t and h,

00:03:40.850 --> 00:03:44.430 align:middle line:84%
obviously it's going to guess
the e is going to come next.

00:03:44.430 --> 00:03:46.430 align:middle line:90%
So that part is old, old news.

00:03:46.430 --> 00:03:47.230 align:middle line:90%
Nothing new there.

00:03:47.230 --> 00:03:50.210 align:middle line:84%
But apparently, the e
letter in the keyboard

00:03:50.210 --> 00:03:52.570 align:middle line:90%
will become slightly bigger.

00:03:52.570 --> 00:03:54.070 align:middle line:84%
So when your finger
goes towards it,

00:03:54.070 --> 00:03:56.260 align:middle line:84%
it has a better shot of
actually connecting with it.

00:03:56.260 --> 00:03:58.940 align:middle line:84%
So these kinds of
things are used

00:03:58.940 --> 00:04:01.760 align:middle line:84%
to change the UI in real time in
a whole bunch of applications,

00:04:01.760 --> 00:04:04.540 align:middle line:84%
and you just don't
even realize it.

00:04:04.540 --> 00:04:08.800 align:middle line:84%
And of course, we all know
about LLMs at this point.

00:04:08.800 --> 00:04:10.300 align:middle line:84%
So I asked you to
write a limerick

00:04:10.300 --> 00:04:13.027 align:middle line:84%
about the beauty and power
of deep learning yesterday.

00:04:13.027 --> 00:04:15.360 align:middle line:84%
And it says, "In a world where
data flows like a stream,

00:04:15.360 --> 00:04:16.839 align:middle line:84%
deep learning is
more than a dream.

00:04:16.839 --> 00:04:18.881 align:middle line:84%
It sifts through the noise
with an elegant poise,

00:04:18.881 --> 00:04:22.380 align:middle line:90%
unveiling insights that gleam.

00:04:22.380 --> 00:04:23.660 align:middle line:90%
Cool, right?

00:04:23.660 --> 00:04:25.980 align:middle line:84%
All right, so let's
get back to work.

00:04:25.980 --> 00:04:27.900 align:middle line:84%
So NLP has
extraordinary potential

00:04:27.900 --> 00:04:31.540 align:middle line:84%
for making products and
services much, much smarter.

00:04:31.540 --> 00:04:33.540 align:middle line:84%
And what I want
to point out here

00:04:33.540 --> 00:04:36.660 align:middle line:84%
is that even if you
focus on this very, very

00:04:36.660 --> 00:04:40.700 align:middle line:84%
simple formalism, a bunch of
text comes in, a bunch of text

00:04:40.700 --> 00:04:41.240 align:middle line:90%
goes out.

00:04:41.240 --> 00:04:41.940 align:middle line:90%
That's it.

00:04:41.940 --> 00:04:45.300 align:middle line:84%
If you take that very simple
text in, text out formalism,

00:04:45.300 --> 00:04:49.060 align:middle line:84%
this little humble little thing
has just an enormous, enormous

00:04:49.060 --> 00:04:50.780 align:middle line:90%
range of applicability.

00:04:50.780 --> 00:04:53.920 align:middle line:84%
So, obviously, you can
send a bunch of text

00:04:53.920 --> 00:04:57.400 align:middle line:84%
in and ask it to
classify it, sentiment

00:04:57.400 --> 00:04:59.127 align:middle line:90%
routed for customer support.

00:04:59.127 --> 00:05:00.960 align:middle line:84%
You can try to figure
out the intent of what

00:05:00.960 --> 00:05:02.580 align:middle line:90%
the person is asking in search.

00:05:02.580 --> 00:05:03.330 align:middle line:90%
You can filter it.

00:05:03.330 --> 00:05:05.122 align:middle line:84%
You can content filter
to make sure there's

00:05:05.122 --> 00:05:06.540 align:middle line:84%
no toxic, abusive
stuff going on.

00:05:06.540 --> 00:05:09.200 align:middle line:84%
I mean, the possibilities
for just text classification

00:05:09.200 --> 00:05:10.740 align:middle line:90%
are numerous.

00:05:10.740 --> 00:05:14.060 align:middle line:84%
But that's a use case we're
all kind of familiar with.

00:05:14.060 --> 00:05:15.780 align:middle line:90%
So no surprise there.

00:05:15.780 --> 00:05:18.660 align:middle line:84%
Now, text extraction we may
be less familiar with here.

00:05:18.660 --> 00:05:20.360 align:middle line:84%
And the idea is that
you can actually

00:05:20.360 --> 00:05:23.400 align:middle line:84%
look at a lot of
unstructured textual data

00:05:23.400 --> 00:05:27.540 align:middle line:84%
and extract all sorts of
interesting entities from it.

00:05:27.540 --> 00:05:29.100 align:middle line:90%
Hedge funds use it very heavily.

00:05:29.100 --> 00:05:31.183 align:middle line:84%
They will extract all sorts
of company information

00:05:31.183 --> 00:05:32.360 align:middle line:90%
from news articles.

00:05:32.360 --> 00:05:34.255 align:middle line:84%
And then obviously
doctor's notes.

00:05:34.255 --> 00:05:35.880 align:middle line:84%
There are a whole
bunch of NLP startups

00:05:35.880 --> 00:05:38.900 align:middle line:84%
that will take the
doctor-patient conversation,

00:05:38.900 --> 00:05:42.040 align:middle line:84%
transcribe it, and then extract
disease codes, diagnosis

00:05:42.040 --> 00:05:44.520 align:middle line:84%
codes, medication codes,
and things like that.

00:05:44.520 --> 00:05:46.960 align:middle line:84%
So the possibilities
for this are enormous.

00:05:46.960 --> 00:05:48.640 align:middle line:84%
Of course, text
summarization, and we

00:05:48.640 --> 00:05:50.720 align:middle line:84%
all have been doing
it thanks to ChatGPT.

00:05:50.720 --> 00:05:53.330 align:middle line:84%
Take text in, and
any kind of summary

00:05:53.330 --> 00:05:55.290 align:middle line:84%
that comes out of the
text, it's just text out.

00:05:55.290 --> 00:05:58.030 align:middle line:84%
And then text
generation, of course.

00:05:58.030 --> 00:06:00.430 align:middle line:84%
We can take text and do
marketing, copy sales, emails,

00:06:00.430 --> 00:06:01.730 align:middle line:84%
market summaries,
so on and so forth,

00:06:01.730 --> 00:06:03.050 align:middle line:84%
including, troublingly
for educators,

00:06:03.050 --> 00:06:04.175 align:middle line:90%
college application essays.

00:06:04.175 --> 00:06:06.850 align:middle line:90%


00:06:06.850 --> 00:06:11.650 align:middle line:84%
Code generation is a more
subtle example of text

00:06:11.650 --> 00:06:14.810 align:middle line:90%
out because code is just text.

00:06:14.810 --> 00:06:20.090 align:middle line:84%
So text in, text out also
covers text in, code out

00:06:20.090 --> 00:06:21.430 align:middle line:90%
and question answering.

00:06:21.430 --> 00:06:25.090 align:middle line:84%
So you can take a bunch of
text, you can take a whole bunch

00:06:25.090 --> 00:06:27.437 align:middle line:84%
of documents, you can
add a bit of text to it,

00:06:27.437 --> 00:06:29.770 align:middle line:84%
which is your question-- and
this whole thing at the end

00:06:29.770 --> 00:06:31.070 align:middle line:90%
of the day is just text--

00:06:31.070 --> 00:06:33.315 align:middle line:84%
send it in, and
then you can use it

00:06:33.315 --> 00:06:34.690 align:middle line:84%
to answer questions
and therefore

00:06:34.690 --> 00:06:37.270 align:middle line:84%
create chatbots for all sorts
of interesting applications.

00:06:37.270 --> 00:06:39.810 align:middle line:90%


00:06:39.810 --> 00:06:43.370 align:middle line:84%
And if you look at this
example, call centers,

00:06:43.370 --> 00:06:46.410 align:middle line:84%
that is where a lot of money is
being spent right now to build

00:06:46.410 --> 00:06:48.050 align:middle line:84%
these call center
chatbots for text in,

00:06:48.050 --> 00:06:49.490 align:middle line:90%
text out question answering.

00:06:49.490 --> 00:06:51.910 align:middle line:84%
And so just if you
drill into this,

00:06:51.910 --> 00:06:55.190 align:middle line:84%
if you imagine taking all
the call center transcripts

00:06:55.190 --> 00:06:57.750 align:middle line:84%
and their internal product
documentation, service

00:06:57.750 --> 00:07:00.950 align:middle line:84%
documentation, FAQs,
et cetera, stick it in,

00:07:00.950 --> 00:07:03.130 align:middle line:84%
you can start to answer
these kinds of questions.

00:07:03.130 --> 00:07:05.030 align:middle line:84%
OK, yesterday, what
are the top reasons why

00:07:05.030 --> 00:07:06.870 align:middle line:90%
customers were upset with us?

00:07:06.870 --> 00:07:09.570 align:middle line:84%
What interventions made by
the agent actually worked?

00:07:09.570 --> 00:07:11.390 align:middle line:90%
What did not work?

00:07:11.390 --> 00:07:13.938 align:middle line:84%
What characterizes the
best agents from the rest?

00:07:13.938 --> 00:07:16.230 align:middle line:84%
How should we grade this
particular agent's interaction

00:07:16.230 --> 00:07:17.750 align:middle line:90%
with this particular customer?

00:07:17.750 --> 00:07:20.090 align:middle line:84%
How should we change
the call center script?

00:07:20.090 --> 00:07:22.910 align:middle line:84%
How should we coach
the agent in real time?

00:07:22.910 --> 00:07:25.590 align:middle line:84%
Every one of these applications
is amenable to this very humble

00:07:25.590 --> 00:07:29.340 align:middle line:90%
text in, text out model.

00:07:29.340 --> 00:07:32.550 align:middle line:84%
And of course, everybody
knows this potential

00:07:32.550 --> 00:07:35.590 align:middle line:84%
because of the advent of
large language models.

00:07:35.590 --> 00:07:39.550 align:middle line:84%
By the way, Google released
something called Google Gemini

00:07:39.550 --> 00:07:46.870 align:middle line:84%
1.5 Pro a couple of days
ago, and it's incredible.

00:07:46.870 --> 00:07:47.650 align:middle line:90%
It's incredible.

00:07:47.650 --> 00:07:50.260 align:middle line:84%
And anyway, we'll get
back to that later.

00:07:50.260 --> 00:07:52.480 align:middle line:84%
But the point is that the
kind of potential we have

00:07:52.480 --> 00:07:54.800 align:middle line:84%
is just amazing, even
for text in, text out.

00:07:54.800 --> 00:07:56.660 align:middle line:90%
And as you would imagine--

00:07:56.660 --> 00:08:00.472 align:middle line:90%


00:08:00.472 --> 00:08:02.680 align:middle line:84%
AUDIENCE: This is all, like,
though we are calling it

00:08:02.680 --> 00:08:05.707 align:middle line:84%
language, this is all
primarily English.

00:08:05.707 --> 00:08:08.040 align:middle line:84%
RAMA RAMAKRISHNAN: There are
lots of multilingual models

00:08:08.040 --> 00:08:08.660 align:middle line:90%
as well.

00:08:08.660 --> 00:08:10.400 align:middle line:90%
There are multilingual models.

00:08:10.400 --> 00:08:13.425 align:middle line:84%
By that, I mean models which are
specialized to other languages,

00:08:13.425 --> 00:08:14.800 align:middle line:84%
non-English
languages, and models

00:08:14.800 --> 00:08:18.180 align:middle line:84%
which are truly multilingual,
like polyglot models as well.

00:08:18.180 --> 00:08:20.660 align:middle line:84%
And both of them are
available right now.

00:08:20.660 --> 00:08:24.720 align:middle line:84%
And many, many modern LLMs are
actually trained from the get-go

00:08:24.720 --> 00:08:27.240 align:middle line:84%
to be multilingual
in a bunch of what

00:08:27.240 --> 00:08:29.440 align:middle line:84%
are called high-resource
languages, languages which

00:08:29.440 --> 00:08:30.909 align:middle line:90%
are spoken by lots of people.

00:08:30.909 --> 00:08:33.159 align:middle line:84%
But actually, it's funny you
should ask that question,

00:08:33.159 --> 00:08:35.240 align:middle line:84%
because this Google
Gemini model that I just

00:08:35.240 --> 00:08:37.440 align:middle line:90%
described, they actually--

00:08:37.440 --> 00:08:39.960 align:middle line:84%
so there is a language
called Kalimpong,

00:08:39.960 --> 00:08:43.080 align:middle line:84%
which is spoken by 200
people in the world.

00:08:43.080 --> 00:08:47.220 align:middle line:84%
And so a researcher
had created one book,

00:08:47.220 --> 00:08:49.860 align:middle line:84%
which is sort of like a
grammar manual for Kalimpong

00:08:49.860 --> 00:08:53.142 align:middle line:84%
because there are no other
written works in that language.

00:08:53.142 --> 00:08:55.100 align:middle line:84%
And so what they did is
they took a whole bunch

00:08:55.100 --> 00:08:58.540 align:middle line:84%
of English dialogue
and this book,

00:08:58.540 --> 00:09:03.420 align:middle line:84%
fed it into a Google
Gemini Pro 1.5,

00:09:03.420 --> 00:09:05.300 align:middle line:84%
and it translated
it into Kalimpong

00:09:05.300 --> 00:09:07.660 align:middle line:90%
at human level proficiency.

00:09:07.660 --> 00:09:09.820 align:middle line:90%
It had never seen it before.

00:09:09.820 --> 00:09:15.620 align:middle line:90%
So that's an example of this.

00:09:15.620 --> 00:09:16.540 align:middle line:90%
Yes.

00:09:16.540 --> 00:09:19.420 align:middle line:84%
So the question text
here is all the things

00:09:19.420 --> 00:09:21.460 align:middle line:84%
you want to translate
from English to Kalimpong.

00:09:21.460 --> 00:09:24.020 align:middle line:84%
The documents here is just
one document, singular,

00:09:24.020 --> 00:09:25.700 align:middle line:90%
the grammar book, the manual.

00:09:25.700 --> 00:09:28.340 align:middle line:84%
And then what comes
out is a translation.

00:09:28.340 --> 00:09:30.380 align:middle line:84%
So these models,
even when they're not

00:09:30.380 --> 00:09:32.940 align:middle line:84%
explicitly trained on a
different language, if you give

00:09:32.940 --> 00:09:35.920 align:middle line:84%
them enough of grammar
manuals and stuff like that,

00:09:35.920 --> 00:09:38.220 align:middle line:84%
they may do a pretty
decent job from the get-go

00:09:38.220 --> 00:09:40.460 align:middle line:90%
with no training.

00:09:40.460 --> 00:09:41.820 align:middle line:90%
It's kind of a shocker.

00:09:41.820 --> 00:09:44.820 align:middle line:84%
Two years ago, people would
be like, that's impossible.

00:09:44.820 --> 00:09:50.570 align:middle line:90%
All right, so back to this.

00:09:50.570 --> 00:09:52.230 align:middle line:84%
And as you folks
may already know,

00:09:52.230 --> 00:09:54.647 align:middle line:84%
and maybe you're, in fact,
participating in this gold rush

00:09:54.647 --> 00:09:57.710 align:middle line:84%
already, lots of
people are creating

00:09:57.710 --> 00:10:00.272 align:middle line:84%
lots of really cool companies
to take some of these ideas

00:10:00.272 --> 00:10:02.230 align:middle line:84%
and actually create really
interesting products

00:10:02.230 --> 00:10:04.110 align:middle line:90%
and services out of them.

00:10:04.110 --> 00:10:06.190 align:middle line:84%
So if you're not doing
it, and if you've

00:10:06.190 --> 00:10:08.510 align:middle line:84%
been thinking about
entrepreneurial stuff,

00:10:08.510 --> 00:10:09.553 align:middle line:90%
here's a word of advice--

00:10:09.553 --> 00:10:10.220 align:middle line:90%
take the plunge.

00:10:10.220 --> 00:10:15.110 align:middle line:90%


00:10:15.110 --> 00:10:15.630 align:middle line:90%
Dismissed.

00:10:15.630 --> 00:10:17.490 align:middle line:90%
I'm just kidding.

00:10:17.490 --> 00:10:20.390 align:middle line:84%
And as you can imagine,
enterprise vendors

00:10:20.390 --> 00:10:23.130 align:middle line:84%
are rushing to add NLP
to all their products.

00:10:23.130 --> 00:10:25.750 align:middle line:84%
Salesforce-- Einstein
now has Einstein GPT,

00:10:25.750 --> 00:10:27.730 align:middle line:90%
Microsoft has Copilot.

00:10:27.730 --> 00:10:29.250 align:middle line:90%
I mean, the list goes on.

00:10:29.250 --> 00:10:31.150 align:middle line:84%
Everybody's
scrambling and really

00:10:31.150 --> 00:10:33.110 align:middle line:84%
trying hard to
infuse some GPT magic

00:10:33.110 --> 00:10:34.510 align:middle line:90%
into whatever they're doing.

00:10:34.510 --> 00:10:37.590 align:middle line:84%
Some of it is real,
a lot of it is not.

00:10:37.590 --> 00:10:40.890 align:middle line:84%
OK, so let's go to the
arc of NLP progress.

00:10:40.890 --> 00:10:44.810 align:middle line:84%
How did we get to this kind of
crazy times that we live in?

00:10:44.810 --> 00:10:48.000 align:middle line:84%
So if you look at natural
language processing, basically

00:10:48.000 --> 00:10:51.570 align:middle line:84%
the efforts to take language and
try to analyze language and do

00:10:51.570 --> 00:10:56.210 align:middle line:84%
predictions with language
and so on and so forth,

00:10:56.210 --> 00:10:58.890 align:middle line:84%
the first phase of it was
just hand-crafted rules

00:10:58.890 --> 00:11:00.210 align:middle line:90%
based on linguistics.

00:11:00.210 --> 00:11:02.130 align:middle line:84%
So these were all
linguists who would really

00:11:02.130 --> 00:11:03.792 align:middle line:84%
understand the
grammar of a language,

00:11:03.792 --> 00:11:06.250 align:middle line:84%
and then they would use their
deep knowledge of linguistics

00:11:06.250 --> 00:11:08.330 align:middle line:84%
to figure out all these
rules by which you

00:11:08.330 --> 00:11:11.610 align:middle line:84%
can process and analyze
natural language text.

00:11:11.610 --> 00:11:13.717 align:middle line:84%
And then this other
thing came along,

00:11:13.717 --> 00:11:16.050 align:middle line:84%
which was a statistical machine
learning approach, which

00:11:16.050 --> 00:11:19.250 align:middle line:84%
basically said, never mind
all that complicated knowledge

00:11:19.250 --> 00:11:20.750 align:middle line:90%
of linguistics and grammar.

00:11:20.750 --> 00:11:23.217 align:middle line:84%
Why don't we simply
count things?

00:11:23.217 --> 00:11:25.550 align:middle line:84%
Let's count the number of
times these two will co-occur.

00:11:25.550 --> 00:11:27.350 align:middle line:84%
Now, let's count that,
let's count this.

00:11:27.350 --> 00:11:28.730 align:middle line:90%
Basically, just count a lot.

00:11:28.730 --> 00:11:32.490 align:middle line:84%
And let's see if it does, if
it does for predicting things,

00:11:32.490 --> 00:11:34.450 align:middle line:84%
say for classifying
text and so on.

00:11:34.450 --> 00:11:38.137 align:middle line:84%
And shockingly, those methods
ended up being really good.

00:11:38.137 --> 00:11:39.470 align:middle line:90%
They ended up being really good.

00:11:39.470 --> 00:11:43.150 align:middle line:84%
And in fact, they actually
were better than the lovingly

00:11:43.150 --> 00:11:46.710 align:middle line:84%
hand-curated,
linguistically-driven rules,

00:11:46.710 --> 00:11:50.290 align:middle line:84%
so much so this is a
famous quote which says,

00:11:50.290 --> 00:11:52.790 align:middle line:84%
"Every time I fire a linguist,
the performance of the speech

00:11:52.790 --> 00:11:55.510 align:middle line:90%
recognizer goes up."

00:11:55.510 --> 00:11:57.190 align:middle line:84%
Obviously made in
jest, but there

00:11:57.190 --> 00:11:59.710 align:middle line:90%
is a kernel of truth to it.

00:11:59.710 --> 00:12:02.410 align:middle line:90%
So that's where we were.

00:12:02.410 --> 00:12:07.052 align:middle line:84%
And then deep learning
happened in 2012, roughly.

00:12:07.052 --> 00:12:08.510 align:middle line:84%
And then we had
these things called

00:12:08.510 --> 00:12:10.010 align:middle line:84%
recurrent neural
networks, which are

00:12:10.010 --> 00:12:13.190 align:middle line:84%
based on deep learning, which
actually moved the ball forward.

00:12:13.190 --> 00:12:16.830 align:middle line:84%
And then in 2017, something
called the transformer

00:12:16.830 --> 00:12:20.150 align:middle line:90%
was invented, 2017.

00:12:20.150 --> 00:12:23.390 align:middle line:84%
And the transformer
replaced everything else

00:12:23.390 --> 00:12:24.750 align:middle line:90%
across the board.

00:12:24.750 --> 00:12:27.310 align:middle line:84%
So we're just going
to leapfrog directly

00:12:27.310 --> 00:12:28.477 align:middle line:90%
to transformers in HODL.

00:12:28.477 --> 00:12:30.810 align:middle line:84%
We will not spend any time
on recurrent neural networks.

00:12:30.810 --> 00:12:34.510 align:middle line:84%
And that is not to
say that they're dead.

00:12:34.510 --> 00:12:36.430 align:middle line:84%
There's a very interesting
work which actually

00:12:36.430 --> 00:12:38.590 align:middle line:84%
is trying to now revive
recurrent neural networks

00:12:38.590 --> 00:12:42.180 align:middle line:84%
to make it work for these kinds
of modern LLM kinds of tasks,

00:12:42.180 --> 00:12:43.960 align:middle line:90%
but it's still very early days.

00:12:43.960 --> 00:12:46.560 align:middle line:84%
So for now, we'll just
focus on transformers.

00:12:46.560 --> 00:12:50.080 align:middle line:84%
So the very high-level
view of the problem

00:12:50.080 --> 00:12:53.020 align:middle line:84%
here is that, like most
things in deep learning,

00:12:53.020 --> 00:12:55.560 align:middle line:90%
it's basically fancy regression.

00:12:55.560 --> 00:12:57.660 align:middle line:84%
There is some variable,
x, that comes in.

00:12:57.660 --> 00:13:01.320 align:middle line:84%
It goes through this
very complicated function

00:13:01.320 --> 00:13:03.640 align:middle line:84%
along with this w,
which is the weights,

00:13:03.640 --> 00:13:05.300 align:middle line:90%
and then out pops an output.

00:13:05.300 --> 00:13:07.600 align:middle line:84%
That's just the view
that you've always had.

00:13:07.600 --> 00:13:10.940 align:middle line:84%
And so in this case,
x happens to be text.

00:13:10.940 --> 00:13:11.868 align:middle line:90%
y-hat can be text.

00:13:11.868 --> 00:13:12.660 align:middle line:90%
It could be labels.

00:13:12.660 --> 00:13:13.260 align:middle line:90%
It could be numbers.

00:13:13.260 --> 00:13:14.220 align:middle line:90%
It could be anything else.

00:13:14.220 --> 00:13:15.803 align:middle line:84%
The w is the weights,
and the function

00:13:15.803 --> 00:13:18.348 align:middle line:90%
is a deep neural network.

00:13:18.348 --> 00:13:20.140 align:middle line:84%
At this point, when
you look at this slide,

00:13:20.140 --> 00:13:23.360 align:middle line:90%
it should be blindingly obvious.

00:13:23.360 --> 00:13:26.240 align:middle line:84%
So now, the key question
here is, how do you actually

00:13:26.240 --> 00:13:27.920 align:middle line:90%
represent x?

00:13:27.920 --> 00:13:29.520 align:middle line:90%
That's the key question.

00:13:29.520 --> 00:13:32.400 align:middle line:84%
For pictures, for images,
we saw that we just

00:13:32.400 --> 00:13:35.280 align:middle line:84%
took the pixel values,
which were light intensity

00:13:35.280 --> 00:13:37.200 align:middle line:84%
numbers between 0 and
255, and you could just

00:13:37.200 --> 00:13:38.340 align:middle line:90%
use that directly.

00:13:38.340 --> 00:13:41.680 align:middle line:84%
But when a sentence comes in,
like, "I love deep learning",

00:13:41.680 --> 00:13:42.785 align:middle line:90%
what do you do?

00:13:42.785 --> 00:13:44.160 align:middle line:84%
How do you actually
represent it?

00:13:44.160 --> 00:13:46.202 align:middle line:84%
Because remember, we have
to numerical everything

00:13:46.202 --> 00:13:47.700 align:middle line:90%
that's coming in.

00:13:47.700 --> 00:13:48.620 align:middle line:90%
So that's the key.

00:13:48.620 --> 00:13:51.420 align:middle line:84%
And this actually is a very
subtle question, very important

00:13:51.420 --> 00:13:54.260 align:middle line:84%
question, and we'll
focus on that today.

00:13:54.260 --> 00:13:57.980 align:middle line:84%
And then next week, when
we look at transformers,

00:13:57.980 --> 00:14:00.580 align:middle line:84%
we will look at what neural
network architecture is best

00:14:00.580 --> 00:14:04.000 align:middle line:84%
suited to process these text
inputs that are coming in.

00:14:04.000 --> 00:14:07.260 align:middle line:84%
Those are the two big questions
we're going to look at.

00:14:07.260 --> 00:14:11.048 align:middle line:90%
All right, so processing basics.

00:14:11.048 --> 00:14:13.090 align:middle line:84%
We're going to follow this
very standard process.

00:14:13.090 --> 00:14:15.860 align:middle line:90%


00:14:15.860 --> 00:14:20.380 align:middle line:84%
This is the process by which
we take any text that comes in,

00:14:20.380 --> 00:14:22.680 align:middle line:84%
and we do run it through
these four steps.

00:14:22.680 --> 00:14:25.107 align:middle line:84%
And this process is
called text vectorization.

00:14:25.107 --> 00:14:26.940 align:middle line:84%
And as the name suggests,
we are essentially

00:14:26.940 --> 00:14:29.820 align:middle line:84%
taking text and creating
vectors of numbers out of them.

00:14:29.820 --> 00:14:31.420 align:middle line:90%
Text vectorization.

00:14:31.420 --> 00:14:33.420 align:middle line:84%
And we'll go through each
of these processes one

00:14:33.420 --> 00:14:34.500 align:middle line:90%
after the other.

00:14:34.500 --> 00:14:38.510 align:middle line:84%
So I just find it very useful to
just have this acronym S-T-I-E

00:14:38.510 --> 00:14:41.210 align:middle line:84%
in my head, like "stie",
S-T-I-E. Just keep that in mind.

00:14:41.210 --> 00:14:42.830 align:middle line:90%
It may be helpful.

00:14:42.830 --> 00:14:46.110 align:middle line:84%
All right, so what
we do is, the setup

00:14:46.110 --> 00:14:49.650 align:middle line:84%
here is that we have a
whole bunch of documents.

00:14:49.650 --> 00:14:51.290 align:middle line:90%
We call it the training corpus.

00:14:51.290 --> 00:14:54.132 align:middle line:84%
We have a whole bunch of
text documents, text data.

00:14:54.132 --> 00:14:55.590 align:middle line:84%
And as far as we
are concerned, you

00:14:55.590 --> 00:14:59.770 align:middle line:84%
can just imagine it as just
lists of long passages.

00:14:59.770 --> 00:15:00.970 align:middle line:90%
What is a novel?

00:15:00.970 --> 00:15:03.090 align:middle line:84%
It's just a long
passage of text.

00:15:03.090 --> 00:15:06.810 align:middle line:84%
So whether it's a novel or a
sentence doesn't really matter.

00:15:06.810 --> 00:15:08.270 align:middle line:84%
We just think of
them as a big list

00:15:08.270 --> 00:15:10.250 align:middle line:90%
of strings, a big list of text.

00:15:10.250 --> 00:15:12.110 align:middle line:90%
That's our training corpus.

00:15:12.110 --> 00:15:14.590 align:middle line:84%
And what we do is we
take this training corpus

00:15:14.590 --> 00:15:16.630 align:middle line:84%
and we run it
through, and we apply

00:15:16.630 --> 00:15:18.330 align:middle line:84%
standardization
and tokenization,

00:15:18.330 --> 00:15:22.390 align:middle line:84%
which I will describe, to this
entire training corpus upfront.

00:15:22.390 --> 00:15:25.630 align:middle line:90%
So we first do this.

00:15:25.630 --> 00:15:29.750 align:middle line:84%
And standardization
is basically--

00:15:29.750 --> 00:15:32.190 align:middle line:84%
the default for
most applications

00:15:32.190 --> 00:15:35.230 align:middle line:84%
tends to be this, which is
we first strip capitalization

00:15:35.230 --> 00:15:38.610 align:middle line:84%
and make everything
lowercase, and then

00:15:38.610 --> 00:15:41.750 align:middle line:84%
we remove punctuation and
accents and so on and so forth.

00:15:41.750 --> 00:15:44.050 align:middle line:90%
That's the first thing we do.

00:15:44.050 --> 00:15:46.470 align:middle line:84%
I'll talk about why we
do it in just a moment.

00:15:46.470 --> 00:15:49.410 align:middle line:84%
But the mechanics of it
are, we do this first,

00:15:49.410 --> 00:15:53.270 align:middle line:84%
then we look at words like a,
the, it, and so on and so forth,

00:15:53.270 --> 00:15:57.570 align:middle line:84%
basically filler words, which
we need to actually make

00:15:57.570 --> 00:15:59.570 align:middle line:84%
complete sentences,
but they may not have

00:15:59.570 --> 00:16:01.850 align:middle line:90%
any value in predicting things.

00:16:01.850 --> 00:16:04.350 align:middle line:84%
So we remove them, and
they are called stop words.

00:16:04.350 --> 00:16:07.510 align:middle line:84%
And then finally, we take
words which are very similar,

00:16:07.510 --> 00:16:10.070 align:middle line:84%
which have sort of the
same kind of stem or root,

00:16:10.070 --> 00:16:13.410 align:middle line:84%
and then we just map it to
a common representation,

00:16:13.410 --> 00:16:16.210 align:middle line:84%
like ate, eaten, eating, eaten,
all these things will just

00:16:16.210 --> 00:16:18.210 align:middle line:90%
become, let's say, eats.

00:16:18.210 --> 00:16:19.510 align:middle line:90%
And we do that sometimes.

00:16:19.510 --> 00:16:21.210 align:middle line:90%
So this, we almost always do.

00:16:21.210 --> 00:16:22.230 align:middle line:90%
This, we often do.

00:16:22.230 --> 00:16:23.510 align:middle line:90%
And this, we do it sometimes.

00:16:23.510 --> 00:16:25.950 align:middle line:84%
Now, why do we do
any of these things?

00:16:25.950 --> 00:16:34.490 align:middle line:90%


00:16:34.490 --> 00:16:37.780 align:middle line:84%
AUDIENCE: I think we want to try
to recognize the essential thing

00:16:37.780 --> 00:16:40.240 align:middle line:84%
with the word, whether
it's eaten or eat,

00:16:40.240 --> 00:16:42.580 align:middle line:84%
but the essential
thing is the eat part.

00:16:42.580 --> 00:16:46.220 align:middle line:84%
So we want to try to
abstract from it the

00:16:46.220 --> 00:16:47.167 align:middle line:90%
more essential thing.

00:16:47.167 --> 00:16:49.500 align:middle line:84%
RAMA RAMAKRISHNAN: Right, so
why do we need to abstract?

00:16:49.500 --> 00:16:50.600 align:middle line:90%
You're absolutely correct.

00:16:50.600 --> 00:16:51.700 align:middle line:90%
We're trying to abstract.

00:16:51.700 --> 00:16:54.200 align:middle line:84%
Why is there a benefit to
doing this abstraction?

00:16:54.200 --> 00:16:57.723 align:middle line:90%


00:16:57.723 --> 00:16:59.640 align:middle line:84%
How about somebody from
this side of the room?

00:16:59.640 --> 00:17:02.020 align:middle line:90%
Oh, yes.

00:17:02.020 --> 00:17:06.593 align:middle line:90%
AUDIENCE: [INAUDIBLE]

00:17:06.593 --> 00:17:08.260 align:middle line:84%
RAMA RAMAKRISHNAN:
Why is it a good idea

00:17:08.260 --> 00:17:10.700 align:middle line:84%
to reduce the size
of the library?

00:17:10.700 --> 00:17:12.220 align:middle line:84%
AUDIENCE: To reduce
the computation.

00:17:12.220 --> 00:17:13.887 align:middle line:84%
RAMA RAMAKRISHNAN:
Because of the amount

00:17:13.887 --> 00:17:15.500 align:middle line:90%
of computation needed.

00:17:15.500 --> 00:17:17.460 align:middle line:90%
So, that is part of the answer.

00:17:17.460 --> 00:17:21.099 align:middle line:84%
There's another part to
the answer, which is?

00:17:21.099 --> 00:17:22.599 align:middle line:84%
All right, let's
swing to the right.

00:17:22.599 --> 00:17:26.708 align:middle line:90%


00:17:26.708 --> 00:17:28.500 align:middle line:84%
AUDIENCE: Because it
facilitates comparison

00:17:28.500 --> 00:17:31.840 align:middle line:84%
between different sets if you
are able to standardize it.

00:17:31.840 --> 00:17:34.440 align:middle line:90%


00:17:34.440 --> 00:17:36.680 align:middle line:84%
RAMA RAMAKRISHNAN: OK,
so I will go with that.

00:17:36.680 --> 00:17:40.560 align:middle line:84%
But I think the key
thing to realize here

00:17:40.560 --> 00:17:43.100 align:middle line:90%
is that you want the model--

00:17:43.100 --> 00:17:45.820 align:middle line:84%
much like when we talk about
computer vision, we said,

00:17:45.820 --> 00:17:48.240 align:middle line:84%
look, if it's a
vertical line, I want

00:17:48.240 --> 00:17:50.720 align:middle line:84%
to be able to detect
it wherever it happens.

00:17:50.720 --> 00:17:53.080 align:middle line:84%
I don't want the model to
think that the vertical line

00:17:53.080 --> 00:17:54.800 align:middle line:84%
on the left side is different
from the vertical line

00:17:54.800 --> 00:17:56.800 align:middle line:84%
on the right side and
then later realize

00:17:56.800 --> 00:17:59.200 align:middle line:84%
they are the same thing,
because you would have wasted

00:17:59.200 --> 00:18:01.800 align:middle line:84%
valuable capacity learning
things which actually happen

00:18:01.800 --> 00:18:04.600 align:middle line:84%
to be the same because you
didn't know it was the same.

00:18:04.600 --> 00:18:08.000 align:middle line:84%
So here, if you, for example,
take a word and lowercase

00:18:08.000 --> 00:18:10.440 align:middle line:84%
it, clearly the case
of it, whether it's

00:18:10.440 --> 00:18:12.283 align:middle line:84%
uppercase or lowercase,
most of the time,

00:18:12.283 --> 00:18:14.700 align:middle line:84%
it's not going to matter for
anything you want to predict.

00:18:14.700 --> 00:18:16.760 align:middle line:84%
So you're essentially
telling the model,

00:18:16.760 --> 00:18:18.780 align:middle line:84%
the lowercase version,
the uppercase version,

00:18:18.780 --> 00:18:19.738 align:middle line:90%
they are not different.

00:18:19.738 --> 00:18:20.948 align:middle line:90%
They're actually the same.

00:18:20.948 --> 00:18:23.240 align:middle line:84%
And the easiest way to tell
the model they are the same

00:18:23.240 --> 00:18:25.600 align:middle line:84%
is just make
everything lowercase.

00:18:25.600 --> 00:18:28.132 align:middle line:90%
That is the key idea.

00:18:28.132 --> 00:18:29.968 align:middle line:84%
And similarly, if you
look at stop words,

00:18:29.968 --> 00:18:31.760 align:middle line:84%
the reason is that
these stop words may not

00:18:31.760 --> 00:18:32.850 align:middle line:90%
help you predict anything.

00:18:32.850 --> 00:18:36.130 align:middle line:84%
Whether a word a, and, the
showed up in a movie review

00:18:36.130 --> 00:18:39.105 align:middle line:84%
probably does not affect
the sentiment of the review.

00:18:39.105 --> 00:18:40.230 align:middle line:90%
Therefore, let's remove it.

00:18:40.230 --> 00:18:42.170 align:middle line:84%
So that's a slightly
different reason.

00:18:42.170 --> 00:18:44.390 align:middle line:84%
Stemming is the same
reason as the first one,

00:18:44.390 --> 00:18:47.470 align:middle line:84%
which is that all these words
kind of mean the same thing.

00:18:47.470 --> 00:18:49.550 align:middle line:84%
We don't have to be
super precise about it,

00:18:49.550 --> 00:18:51.810 align:middle line:84%
and so let's just collapse
them onto the same thing.

00:18:51.810 --> 00:18:55.130 align:middle line:84%
Now, these are all the
standard things we do.

00:18:55.130 --> 00:18:58.950 align:middle line:84%
There are important exceptions
to all these things.

00:18:58.950 --> 00:19:01.550 align:middle line:84%
And we'll come back to the
exceptions a bit later.

00:19:01.550 --> 00:19:03.730 align:middle line:84%
But that is the
standard thing we do.

00:19:03.730 --> 00:19:05.170 align:middle line:90%
Make sense?

00:19:05.170 --> 00:19:05.750 align:middle line:90%
All right.

00:19:05.750 --> 00:19:08.650 align:middle line:90%


00:19:08.650 --> 00:19:13.574 align:middle line:84%
So if you look at something
like this sentence here, "Hola!

00:19:13.574 --> 00:19:16.116 align:middle line:84%
What do you picture when you
think of travel to Mexico", bah,

00:19:16.116 --> 00:19:17.810 align:middle line:84%
bah, boom, and then
you can see here,

00:19:17.810 --> 00:19:19.890 align:middle line:84%
this is the
standardized version,

00:19:19.890 --> 00:19:21.570 align:middle line:90%
everything has become lowercase.

00:19:21.570 --> 00:19:23.170 align:middle line:90%
The H has become small h.

00:19:23.170 --> 00:19:24.830 align:middle line:90%
The punctuation has disappeared.

00:19:24.830 --> 00:19:26.370 align:middle line:90%
That's part of standardization.

00:19:26.370 --> 00:19:29.170 align:middle line:90%


00:19:29.170 --> 00:19:33.430 align:middle line:84%
And you can see here that
Mexico m has become small,

00:19:33.430 --> 00:19:36.990 align:middle line:84%
sipping has become
sips, think has become

00:19:36.990 --> 00:19:38.990 align:middle line:90%
thinks, and so on and so forth.

00:19:38.990 --> 00:19:41.690 align:middle line:84%
So that's an example of
standardization at work.

00:19:41.690 --> 00:19:49.000 align:middle line:90%


00:19:49.000 --> 00:19:51.510 align:middle line:84%
The next thing we do is
something very important,

00:19:51.510 --> 00:19:53.798 align:middle line:90%
and it's called tokenization.

00:19:53.798 --> 00:19:55.590 align:middle line:84%
So what we do typically
is that, OK, now we

00:19:55.590 --> 00:19:56.798 align:middle line:90%
have standardized everything.

00:19:56.798 --> 00:19:58.810 align:middle line:90%
We have a bunch of words.

00:19:58.810 --> 00:20:01.870 align:middle line:84%
We need to now split them
into what are called tokens.

00:20:01.870 --> 00:20:07.110 align:middle line:84%
So the most common default is to
just think of a word as a token.

00:20:07.110 --> 00:20:09.290 align:middle line:90%
We just split on the whitespace.

00:20:09.290 --> 00:20:11.950 align:middle line:84%
You take each string, and
wherever there is whitespace,

00:20:11.950 --> 00:20:15.430 align:middle line:84%
meaning actual spaces or
carriage returns and things

00:20:15.430 --> 00:20:17.270 align:middle line:84%
like that, boom, you
just split on them,

00:20:17.270 --> 00:20:19.110 align:middle line:84%
and you just create
words out of it.

00:20:19.110 --> 00:20:23.770 align:middle line:84%
So, for instance, if you have
this standardized sentence here,

00:20:23.770 --> 00:20:27.890 align:middle line:84%
you just split it after every
word, and you get this thing.

00:20:27.890 --> 00:20:29.770 align:middle line:90%
So each of these is now a token.

00:20:29.770 --> 00:20:32.840 align:middle line:90%


00:20:32.840 --> 00:20:36.160 align:middle line:84%
Now, this has some
disadvantages.

00:20:36.160 --> 00:20:38.760 align:middle line:84%
What are some disadvantages
of just splitting

00:20:38.760 --> 00:20:41.080 align:middle line:90%
on the space between words?

00:20:41.080 --> 00:20:43.920 align:middle line:90%
Yeah.

00:20:43.920 --> 00:20:46.240 align:middle line:84%
AUDIENCE: I think we lose
any context because we

00:20:46.240 --> 00:20:49.000 align:middle line:90%
look at each word separately.

00:20:49.000 --> 00:20:52.318 align:middle line:84%
So we don't have any passwords
or what's happened [INAUDIBLE.]

00:20:52.318 --> 00:20:53.360 align:middle line:90%
RAMA RAMAKRISHNAN: Right.

00:20:53.360 --> 00:20:55.760 align:middle line:84%
So, for example, "the
cat sat on the mat"

00:20:55.760 --> 00:21:00.040 align:middle line:84%
and "the mat sat on the
cat" will have the same set.

00:21:00.040 --> 00:21:01.722 align:middle line:90%
Yeah, so you lose the order.

00:21:01.722 --> 00:21:03.180 align:middle line:84%
What are some other
issues with it?

00:21:03.180 --> 00:21:05.190 align:middle line:90%


00:21:05.190 --> 00:21:07.440 align:middle line:84%
AUDIENCE: For words that
should have chained together,

00:21:07.440 --> 00:21:09.400 align:middle line:84%
like Puerto Vallarta,
you lose the fact

00:21:09.400 --> 00:21:11.700 align:middle line:84%
that that's one name
because you separated it.

00:21:11.700 --> 00:21:13.200 align:middle line:84%
RAMA RAMAKRISHNAN:
Correct, exactly.

00:21:13.200 --> 00:21:16.300 align:middle line:84%
So there are compound
words, like father-in-law,

00:21:16.300 --> 00:21:17.320 align:middle line:90%
for instance.

00:21:17.320 --> 00:21:18.340 align:middle line:90%
That's one problem.

00:21:18.340 --> 00:21:21.240 align:middle line:84%
Another problem is that lots
of non-English languages,

00:21:21.240 --> 00:21:24.900 align:middle line:84%
they actually don't have this
notion of a space between words.

00:21:24.900 --> 00:21:27.740 align:middle line:84%
It actually runs
one after the other,

00:21:27.740 --> 00:21:29.860 align:middle line:84%
and the native speakers
know from context

00:21:29.860 --> 00:21:32.060 align:middle line:90%
how to chunk it and break it.

00:21:32.060 --> 00:21:34.700 align:middle line:90%
So, well, what do we do then?

00:21:34.700 --> 00:21:37.860 align:middle line:84%
Because you basically will have
one word for the whole passage,

00:21:37.860 --> 00:21:38.980 align:middle line:90%
one token.

00:21:38.980 --> 00:21:41.740 align:middle line:84%
The other problem is that
there are languages--

00:21:41.740 --> 00:21:43.940 align:middle line:84%
German is perhaps the most
notable one-- in which

00:21:43.940 --> 00:21:45.170 align:middle line:90%
you have very long words.

00:21:45.170 --> 00:21:48.060 align:middle line:90%


00:21:48.060 --> 00:21:50.940 align:middle line:84%
I saw a word, which I
think I might have it

00:21:50.940 --> 00:21:53.220 align:middle line:84%
in a slide somewhere, that's,
like, this long, which

00:21:53.220 --> 00:21:59.760 align:middle line:84%
means you realize that
something amazing is happening,

00:21:59.760 --> 00:22:02.280 align:middle line:84%
but the rest of the world
hasn't woken up to it yet.

00:22:02.280 --> 00:22:04.460 align:middle line:90%
It's that feeling.

00:22:04.460 --> 00:22:06.140 align:middle line:90%
There's a word for that.

00:22:06.140 --> 00:22:07.900 align:middle line:90%
Amazing, right?

00:22:07.900 --> 00:22:11.040 align:middle line:84%
Anyway, so yeah, some words,
in Japanese, for example,

00:22:11.040 --> 00:22:12.380 align:middle line:90%
there's a word called komorebi.

00:22:12.380 --> 00:22:16.540 align:middle line:84%
Do people know the meaning
of the word komorebi?

00:22:16.540 --> 00:22:21.140 align:middle line:84%
It means the transient
beauty of sunlight

00:22:21.140 --> 00:22:24.700 align:middle line:90%
going through fall foliage.

00:22:24.700 --> 00:22:26.060 align:middle line:90%
There's a word for that.

00:22:26.060 --> 00:22:27.560 align:middle line:90%
How cool is that?

00:22:27.560 --> 00:22:28.340 align:middle line:90%
Anyway, sorry.

00:22:28.340 --> 00:22:29.220 align:middle line:90%
I love that word.

00:22:29.220 --> 00:22:30.640 align:middle line:90%
So, back to this.

00:22:30.640 --> 00:22:32.280 align:middle line:90%
So we have this thing here.

00:22:32.280 --> 00:22:34.120 align:middle line:84%
So there are reasons
for which splitting

00:22:34.120 --> 00:22:36.220 align:middle line:84%
on the space between words
is not going to work.

00:22:36.220 --> 00:22:38.860 align:middle line:90%


00:22:38.860 --> 00:22:43.920 align:middle line:84%
So what happens is that
modern large language models--

00:22:43.920 --> 00:22:47.060 align:middle line:84%
so what we have described so
far, despite its shortcomings,

00:22:47.060 --> 00:22:50.280 align:middle line:84%
is actually really good
for lots of NLP use cases.

00:22:50.280 --> 00:22:53.360 align:middle line:84%
If you want to classify text,
it's good enough, for instance.

00:22:53.360 --> 00:22:57.000 align:middle line:84%
But if you want to generate
text, like LLMs do,

00:22:57.000 --> 00:22:58.240 align:middle line:90%
it's not going to work.

00:22:58.240 --> 00:22:59.380 align:middle line:90%
It's not going to work.

00:22:59.380 --> 00:23:01.620 align:middle line:84%
Because you know when you
ask ChatGPT a question,

00:23:01.620 --> 00:23:03.760 align:middle line:84%
it comes back with
perfect punctuation.

00:23:03.760 --> 00:23:05.780 align:middle line:84%
Clearly, punctuation
was not stripped.

00:23:05.780 --> 00:23:07.988 align:middle line:84%
It comes back with a particular
upper and lower case.

00:23:07.988 --> 00:23:09.520 align:middle line:90%
Clearly, that wasn't stripped.

00:23:09.520 --> 00:23:11.120 align:middle line:84%
You can actually
make up new words

00:23:11.120 --> 00:23:12.840 align:middle line:90%
and ask it to use the new word.

00:23:12.840 --> 00:23:13.860 align:middle line:90%
It'll use it.

00:23:13.860 --> 00:23:17.240 align:middle line:84%
Therefore, it's not like it can
only recognize a finite set.

00:23:17.240 --> 00:23:19.560 align:middle line:84%
So there's a very
clever scheme called

00:23:19.560 --> 00:23:24.500 align:middle line:84%
byte-pair encoding, which is
invented to do all those things.

00:23:24.500 --> 00:23:26.710 align:middle line:84%
And I have slides at the
end, and if we have time,

00:23:26.710 --> 00:23:28.710 align:middle line:90%
we'll talk about it.

00:23:28.710 --> 00:23:30.770 align:middle line:84%
For now, let's
continue this thing.

00:23:30.770 --> 00:23:35.010 align:middle line:84%
So when this is done for every
sentence or every passage

00:23:35.010 --> 00:23:39.813 align:middle line:84%
in a training data set, we now
have a list of distinct tokens.

00:23:39.813 --> 00:23:41.230 align:middle line:84%
We have a list of
distinct tokens.

00:23:41.230 --> 00:23:43.730 align:middle line:84%
In this simple case, it happens
to be all the distinct words

00:23:43.730 --> 00:23:45.130 align:middle line:90%
that we have seen.

00:23:45.130 --> 00:23:47.330 align:middle line:90%
That's called the vocabulary.

00:23:47.330 --> 00:23:49.810 align:middle line:90%
That's called the vocabulary.

00:23:49.810 --> 00:23:51.930 align:middle line:84%
So now, we move to the
third and fourth stages.

00:23:51.930 --> 00:23:54.830 align:middle line:84%
In these stages, the
indexing and encoding stage,

00:23:54.830 --> 00:23:58.010 align:middle line:84%
we only work with
the vocabulary.

00:23:58.010 --> 00:24:01.170 align:middle line:84%
And so what we do is, the
first thing, the indexing,

00:24:01.170 --> 00:24:04.070 align:middle line:84%
we assign a unique integer
to each distinct token

00:24:04.070 --> 00:24:05.330 align:middle line:90%
in the vocabulary.

00:24:05.330 --> 00:24:07.730 align:middle line:84%
So, for instance,
let's say that you

00:24:07.730 --> 00:24:10.770 align:middle line:84%
took a whole bunch
of English literature

00:24:10.770 --> 00:24:13.670 align:middle line:84%
as your training corpus,
and you ran it through.

00:24:13.670 --> 00:24:16.050 align:middle line:84%
Basically, you come up
with English dictionary.

00:24:16.050 --> 00:24:18.330 align:middle line:84%
So it will have maybe
starting with a all the way

00:24:18.330 --> 00:24:22.010 align:middle line:84%
to zebra, a whole
bunch of words.

00:24:22.010 --> 00:24:25.790 align:middle line:84%
And so I'm just putting 50,000
here because it turns out

00:24:25.790 --> 00:24:28.290 align:middle line:84%
the GPT family uses something
called 50,000 tokens,

00:24:28.290 --> 00:24:29.410 align:middle line:90%
so I'm just using 50,000.

00:24:29.410 --> 00:24:31.910 align:middle line:84%
It's not the actual number of
words in the English language.

00:24:31.910 --> 00:24:32.970 align:middle line:90%
It's much more than that.

00:24:32.970 --> 00:24:35.970 align:middle line:84%
So let's say that we give
a number 1 through 50,000,

00:24:35.970 --> 00:24:39.030 align:middle line:84%
and then we actually also
introduce a special token called

00:24:39.030 --> 00:24:41.590 align:middle line:90%
UNK, which stands for unknown.

00:24:41.590 --> 00:24:43.110 align:middle line:84%
And we'll come
back to this later.

00:24:43.110 --> 00:24:47.370 align:middle line:84%
And we give unknown
the integer 0.

00:24:47.370 --> 00:24:49.550 align:middle line:84%
So this is what we
mean by indexing.

00:24:49.550 --> 00:24:52.250 align:middle line:84%
Take the word, the tokens
you have identified.

00:24:52.250 --> 00:24:55.470 align:middle line:90%
Just map it to an integer.

00:24:55.470 --> 00:24:57.190 align:middle line:90%
That's the indexing step.

00:24:57.190 --> 00:25:00.070 align:middle line:84%
Then, what we do is
we assign a vector

00:25:00.070 --> 00:25:03.570 align:middle line:90%
to every one of these integers.

00:25:03.570 --> 00:25:05.410 align:middle line:90%
And that is the encoding step.

00:25:05.410 --> 00:25:08.910 align:middle line:84%
We assign a vector
to each integer.

00:25:08.910 --> 00:25:10.670 align:middle line:84%
So you have a bunch
of distinct words.

00:25:10.670 --> 00:25:12.530 align:middle line:84%
Under each word, we
put an integer on it.

00:25:12.530 --> 00:25:14.738 align:middle line:84%
And then we take that integer
and map it to a vector.

00:25:14.738 --> 00:25:16.950 align:middle line:90%
Yeah.

00:25:16.950 --> 00:25:18.550 align:middle line:84%
Can you use the
microphone, please?

00:25:18.550 --> 00:25:20.870 align:middle line:84%
AUDIENCE: Can you please
explain what unknown means?

00:25:20.870 --> 00:25:21.870 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:25:21.870 --> 00:25:23.440 align:middle line:90%
So I'll come back to that.

00:25:23.440 --> 00:25:26.100 align:middle line:84%
For now, just assume that we
have a token called unknown.

00:25:26.100 --> 00:25:27.517 align:middle line:84%
And the way we are
going to use it

00:25:27.517 --> 00:25:29.487 align:middle line:84%
will become apparent
in a few minutes.

00:25:29.487 --> 00:25:31.320 align:middle line:84%
AUDIENCE: Does it mean
there's a base to it,

00:25:31.320 --> 00:25:32.980 align:middle line:90%
there's a letter or something?

00:25:32.980 --> 00:25:35.540 align:middle line:84%
RAMA RAMAKRISHNAN: It's a
placeholder for something else,

00:25:35.540 --> 00:25:38.600 align:middle line:90%
which I'll describe shortly.

00:25:38.600 --> 00:25:42.600 align:middle line:90%
So that's what we have.

00:25:42.600 --> 00:25:45.640 align:middle line:84%
So let's say that we want to
assign a vector to each integer

00:25:45.640 --> 00:25:46.920 align:middle line:90%
in our vocabulary.

00:25:46.920 --> 00:25:50.480 align:middle line:90%
And let's assume that we have--

00:25:50.480 --> 00:25:53.440 align:middle line:84%
OK, let's say we have 50,000
possible integers because we

00:25:53.440 --> 00:25:56.340 align:middle line:84%
have 50,000 possible words,
and we want to assign a vector

00:25:56.340 --> 00:25:59.220 align:middle line:84%
so that if you take the
vector two different words,

00:25:59.220 --> 00:26:01.068 align:middle line:90%
they should look different.

00:26:01.068 --> 00:26:03.360 align:middle line:84%
Clearly, that's the whole
point of mapping from integer

00:26:03.360 --> 00:26:04.320 align:middle line:90%
to vector.

00:26:04.320 --> 00:26:05.840 align:middle line:90%
They better be different.

00:26:05.840 --> 00:26:07.320 align:middle line:84%
What is the simplest
way to come up

00:26:07.320 --> 00:26:10.270 align:middle line:84%
with a vector for
each of these tokens?

00:26:10.270 --> 00:26:20.120 align:middle line:90%


00:26:20.120 --> 00:26:21.820 align:middle line:84%
AUDIENCE: It's the
same as the index.

00:26:21.820 --> 00:26:22.170 align:middle line:90%
RAMA RAMAKRISHNAN: Sorry?

00:26:22.170 --> 00:26:23.820 align:middle line:84%
AUDIENCE: It's the
same as the index.

00:26:23.820 --> 00:26:26.900 align:middle line:84%
It's just a vector one
by one by one [INAUDIBLE]

00:26:26.900 --> 00:26:31.860 align:middle line:84%
RAMA RAMAKRISHNAN: So a
vector of zeros and ones, or?

00:26:31.860 --> 00:26:34.780 align:middle line:84%
AUDIENCE: Just a vector
with one dimension.

00:26:34.780 --> 00:26:36.220 align:middle line:90%
RAMA RAMAKRISHNAN: Oh, I see.

00:26:36.220 --> 00:26:36.760 align:middle line:90%
Got it.

00:26:36.760 --> 00:26:39.775 align:middle line:84%
Well, it's creative, but it's
a little cheating, right?

00:26:39.775 --> 00:26:41.900 align:middle line:84%
Because you're essentially
putting a square bracket

00:26:41.900 --> 00:26:44.420 align:middle line:84%
around the number and
saying it's a vector.

00:26:44.420 --> 00:26:46.492 align:middle line:90%
Good try.

00:26:46.492 --> 00:26:48.200 align:middle line:84%
AUDIENCE: You could
try one-hot encoding.

00:26:48.200 --> 00:26:49.242 align:middle line:90%
RAMA RAMAKRISHNAN: Right.

00:26:49.242 --> 00:26:51.420 align:middle line:90%
You can try one-hot encoding.

00:26:51.420 --> 00:26:54.080 align:middle line:84%
So, remember, the list of
distinct tokens you have,

00:26:54.080 --> 00:26:56.260 align:middle line:84%
you can just think of them
as the distinct levels

00:26:56.260 --> 00:26:59.740 align:middle line:90%
of a categorical variable.

00:26:59.740 --> 00:27:04.540 align:middle line:84%
And you could just use
one-hot encoding for it.

00:27:04.540 --> 00:27:09.220 align:middle line:84%
So what you can do is the simple
thing is to do one-hot encoding.

00:27:09.220 --> 00:27:13.660 align:middle line:84%
And the way it is going to work
is that if you have, let's say,

00:27:13.660 --> 00:27:16.700 align:middle line:84%
50,000 possible values, the
vector is going to be 50,000

00:27:16.700 --> 00:27:17.200 align:middle line:90%
long.

00:27:17.200 --> 00:27:19.110 align:middle line:84%
It's going to have
0's everywhere

00:27:19.110 --> 00:27:22.730 align:middle line:84%
except in the index value
of whatever the token is.

00:27:22.730 --> 00:27:25.110 align:middle line:84%
So, for instance,
since we said UNK

00:27:25.110 --> 00:27:30.390 align:middle line:84%
is going to be the first
number, 0, it has a 1 here.

00:27:30.390 --> 00:27:33.010 align:middle line:84%
The zeroth index
position has a 1.

00:27:33.010 --> 00:27:34.190 align:middle line:90%
Everything is 0.

00:27:34.190 --> 00:27:36.310 align:middle line:84%
a happens to be the
second one, so it

00:27:36.310 --> 00:27:38.430 align:middle line:84%
happens to be 1 in the
second position, zero.

00:27:38.430 --> 00:27:40.942 align:middle line:90%
You get the idea.

00:27:40.942 --> 00:27:42.150 align:middle line:90%
So this is zero-hot encoding.

00:27:42.150 --> 00:27:44.910 align:middle line:84%
We can do this zero-hot
encoding, one-hot encoding.

00:27:44.910 --> 00:27:49.230 align:middle line:84%
And so the dimension of this
encoding vector, how long it is,

00:27:49.230 --> 00:27:51.150 align:middle line:84%
it's basically the
number of distinct tokens

00:27:51.150 --> 00:27:54.830 align:middle line:84%
that you have seen in the
training corpus plus one

00:27:54.830 --> 00:27:56.570 align:middle line:84%
for this UNK thing,
which we'll get to.

00:27:56.570 --> 00:28:00.610 align:middle line:90%


00:28:00.610 --> 00:28:03.370 align:middle line:84%
So that is the
dimensional-encoding vector,

00:28:03.370 --> 00:28:05.500 align:middle line:84%
which this is called
the vocabulary size.

00:28:05.500 --> 00:28:09.162 align:middle line:90%


00:28:09.162 --> 00:28:10.620 align:middle line:84%
This is called the
vocabulary size.

00:28:10.620 --> 00:28:14.110 align:middle line:90%


00:28:14.110 --> 00:28:17.650 align:middle line:84%
So, at this point, we
have created a vocabulary

00:28:17.650 --> 00:28:19.830 align:middle line:84%
from the training
data, training corpus.

00:28:19.830 --> 00:28:21.490 align:middle line:84%
Every distinct
token vocabulary has

00:28:21.490 --> 00:28:23.530 align:middle line:84%
been assigned a one-hot
vector, and we are

00:28:23.530 --> 00:28:27.370 align:middle line:90%
done with basic preprocessing.

00:28:27.370 --> 00:28:29.250 align:middle line:84%
So all the text
that has come in,

00:28:29.250 --> 00:28:32.570 align:middle line:84%
every token has been mapped
to one potentially very

00:28:32.570 --> 00:28:35.930 align:middle line:90%
long one-hot vector.

00:28:35.930 --> 00:28:38.550 align:middle line:84%
Any questions on the mechanics
of this before we continue on?

00:28:38.550 --> 00:28:45.050 align:middle line:90%


00:28:45.050 --> 00:28:48.810 align:middle line:84%
Now, let's see if, when you
get a new input sentence in,

00:28:48.810 --> 00:28:51.130 align:middle line:84%
a new sentence freshly
arriving, and we

00:28:51.130 --> 00:28:52.955 align:middle line:84%
want to feed it into
a deep neural network,

00:28:52.955 --> 00:28:55.330 align:middle line:84%
how will this process actually
apply to that new sentence

00:28:55.330 --> 00:28:56.922 align:middle line:90%
that's coming in?

00:28:56.922 --> 00:29:01.010 align:middle line:84%
So let's assume that we
have completed our S-T-I-E

00:29:01.010 --> 00:29:04.890 align:middle line:84%
on the training corpus, and
it turns out we found only 99

00:29:04.890 --> 00:29:07.870 align:middle line:84%
distinct tokens,
99 distinct words.

00:29:07.870 --> 00:29:11.850 align:middle line:84%
And then we add this UNK
thing to it, so we got 100.

00:29:11.850 --> 00:29:13.590 align:middle line:90%
So this is our vocabulary.

00:29:13.590 --> 00:29:16.840 align:middle line:84%
It starts with UNK, a, and
then goes all the way to zebra.

00:29:16.840 --> 00:29:19.780 align:middle line:84%
But there are only
100 of them in total.

00:29:19.780 --> 00:29:22.540 align:middle line:84%
And just to be very clear, we
didn't bother to do things like

00:29:22.540 --> 00:29:25.000 align:middle line:84%
stemming and stop-word
removal and stuff like that,

00:29:25.000 --> 00:29:28.170 align:middle line:84%
which is why you have words like
"the" showing up in this list.

00:29:28.170 --> 00:29:32.420 align:middle line:90%


00:29:32.420 --> 00:29:35.720 align:middle line:84%
So let's say this
input string arrives,

00:29:35.720 --> 00:29:39.180 align:middle line:84%
"the cat sat on the mat", and
then we run it through S-T-I-E.

00:29:39.180 --> 00:29:42.040 align:middle line:84%
So "the cat sat on the mat" goes
through this thing, boom, boom,

00:29:42.040 --> 00:29:43.500 align:middle line:90%
boom, boom, boom.

00:29:43.500 --> 00:29:48.620 align:middle line:84%
Then, the output is going to
be a table with a bunch of rows

00:29:48.620 --> 00:29:50.460 align:middle line:90%
and a bunch of columns.

00:29:50.460 --> 00:29:54.050 align:middle line:84%
Any guesses how many rows
and how many columns?

00:29:54.050 --> 00:30:01.878 align:middle line:90%


00:30:01.878 --> 00:30:02.920 align:middle line:90%
Please, raise your hands.

00:30:02.920 --> 00:30:03.640 align:middle line:90%
I'll call on you.

00:30:03.640 --> 00:30:13.107 align:middle line:90%


00:30:13.107 --> 00:30:14.440 align:middle line:90%
Yeah, please use the microphone.

00:30:14.440 --> 00:30:15.080 align:middle line:90%
Go for it.

00:30:15.080 --> 00:30:19.332 align:middle line:84%
AUDIENCE: I would guess
100 rows and 6 columns.

00:30:19.332 --> 00:30:20.540 align:middle line:90%
RAMA RAMAKRISHNAN: All right.

00:30:20.540 --> 00:30:23.080 align:middle line:90%
We'll take a look.

00:30:23.080 --> 00:30:25.640 align:middle line:84%
100 and 6 as well as 6
and 100 are both correct.

00:30:25.640 --> 00:30:29.500 align:middle line:84%
So the way I've done
it, it is 6 and 100,

00:30:29.500 --> 00:30:31.720 align:middle line:90%
and that's exactly right.

00:30:31.720 --> 00:30:34.520 align:middle line:84%
So the idea is that
this is your vocabulary.

00:30:34.520 --> 00:30:37.420 align:middle line:84%
So the word "the
cat sat on the mat",

00:30:37.420 --> 00:30:41.600 align:middle line:84%
once you change the case of
it, it becomes like this.

00:30:41.600 --> 00:30:46.120 align:middle line:84%
So the "the" happens to be a
one-hot vector with a 1 where

00:30:46.120 --> 00:30:48.133 align:middle line:84%
there is a "the" and
0 everywhere else.

00:30:48.133 --> 00:30:49.800 align:middle line:84%
I'm not showing all
the zeros because it

00:30:49.800 --> 00:30:52.040 align:middle line:90%
will get too cluttered.

00:30:52.040 --> 00:30:57.120 align:middle line:84%
Similarly, cat has a 1 where the
cat position is and 0 everywhere

00:30:57.120 --> 00:30:59.400 align:middle line:90%
else, and so on and so forth.

00:30:59.400 --> 00:31:00.600 align:middle line:90%
Does that make sense?

00:31:00.600 --> 00:31:03.400 align:middle line:84%
So the phrase, "the cat sat
on the mat" came in as just,

00:31:03.400 --> 00:31:07.450 align:middle line:84%
whatever, six words, and then
it became this 600-entry table.

00:31:07.450 --> 00:31:12.200 align:middle line:90%


00:31:12.200 --> 00:31:16.370 align:middle line:84%
Now, what is the best
way to feed this table

00:31:16.370 --> 00:31:18.010 align:middle line:90%
to a deep neural network?

00:31:18.010 --> 00:31:18.790 align:middle line:90%
What can we do?

00:31:18.790 --> 00:31:23.610 align:middle line:90%


00:31:23.610 --> 00:31:26.088 align:middle line:90%
It's not a vector, it's a table.

00:31:26.088 --> 00:31:27.630 align:middle line:84%
If it's a vector,
we know what to do.

00:31:27.630 --> 00:31:28.422 align:middle line:90%
We just feed it in.

00:31:28.422 --> 00:31:30.810 align:middle line:84%
We'll just maybe send
it to some hidden layer

00:31:30.810 --> 00:31:34.410 align:middle line:84%
and declare victory
at that point.

00:31:34.410 --> 00:31:35.050 align:middle line:90%
Yeah.

00:31:35.050 --> 00:31:36.425 align:middle line:84%
AUDIENCE: You need
to flatten it.

00:31:36.425 --> 00:31:38.425 align:middle line:84%
RAMA RAMAKRISHNAN: You
would like to flatten it?

00:31:38.425 --> 00:31:40.150 align:middle line:90%
And how might you do it?

00:31:40.150 --> 00:31:43.330 align:middle line:90%


00:31:43.330 --> 00:31:46.720 align:middle line:84%
Flattening is a reasonable
answer, by the way.

00:31:46.720 --> 00:31:52.490 align:middle line:84%
AUDIENCE: I think you just
have to take each column,

00:31:52.490 --> 00:31:55.302 align:middle line:84%
take the first one, go into
each word and just kinda, like--

00:31:55.302 --> 00:31:57.010 align:middle line:84%
RAMA RAMAKRISHNAN:
Yeah, so basically you

00:31:57.010 --> 00:31:59.810 align:middle line:84%
can take all the first columns
and then take the second column

00:31:59.810 --> 00:32:02.630 align:middle line:84%
and attach it under the first
column and so on and so forth.

00:32:02.630 --> 00:32:04.510 align:middle line:90%
So we can certainly do that.

00:32:04.510 --> 00:32:06.830 align:middle line:84%
And that's very akin to
how we work with images.

00:32:06.830 --> 00:32:10.003 align:middle line:84%
But there is one
downside to that.

00:32:10.003 --> 00:32:10.920 align:middle line:90%
What is that downside?

00:32:10.920 --> 00:32:15.870 align:middle line:90%


00:32:15.870 --> 00:32:18.510 align:middle line:90%
Yes.

00:32:18.510 --> 00:32:19.830 align:middle line:90%
AUDIENCE: It's pretty long.

00:32:19.830 --> 00:32:24.090 align:middle line:84%
I wonder if instead you could
for the first word, it's 1,

00:32:24.090 --> 00:32:27.130 align:middle line:84%
for the second word, it's 2,
and then you maintain the order,

00:32:27.130 --> 00:32:30.470 align:middle line:84%
but you still keep
it just as one row.

00:32:30.470 --> 00:32:32.530 align:middle line:90%
RAMA RAMAKRISHNAN: One row.

00:32:32.530 --> 00:32:34.530 align:middle line:84%
So we'll come back to
what we do about this.

00:32:34.530 --> 00:32:37.410 align:middle line:84%
But what you're pointing out
is it could be very long.

00:32:37.410 --> 00:32:41.550 align:middle line:84%
Because if each word is a
50,000-long one-hot vector,

00:32:41.550 --> 00:32:45.817 align:middle line:84%
with just six words, it
becomes a 300,000-long vector.

00:32:45.817 --> 00:32:48.150 align:middle line:84%
Can you imagine taking a
300,000-long vector and sending

00:32:48.150 --> 00:32:51.270 align:middle line:90%
it into a 100-unit hidden layer?

00:32:51.270 --> 00:32:53.590 align:middle line:90%
300,000 times 100 parameters?

00:32:53.590 --> 00:32:54.130 align:middle line:90%
Too much.

00:32:54.130 --> 00:32:56.070 align:middle line:90%
Can't learn anything.

00:32:56.070 --> 00:32:57.670 align:middle line:90%
So that's one issue.

00:32:57.670 --> 00:33:00.750 align:middle line:84%
The other issue is
that different length

00:33:00.750 --> 00:33:04.470 align:middle line:84%
texts that are coming in will
have different-sized inputs.

00:33:04.470 --> 00:33:07.650 align:middle line:84%
So here, "the cats are on
the mat" has 6 times 50,000.

00:33:07.650 --> 00:33:10.970 align:middle line:84%
But maybe the cat sat on the mat
and the rat ran over to the cat.

00:33:10.970 --> 00:33:12.470 align:middle line:90%
It becomes even longer.

00:33:12.470 --> 00:33:15.410 align:middle line:84%
We can't handle
variable-sized inputs.

00:33:15.410 --> 00:33:19.610 align:middle line:84%
The inputs all have to be
mapped to the same length.

00:33:19.610 --> 00:33:22.130 align:middle line:90%
That's another problem.

00:33:22.130 --> 00:33:25.390 align:middle line:84%
AUDIENCE: So maybe you can
sum the columns, basically,

00:33:25.390 --> 00:33:27.850 align:middle line:84%
and count how many times each
word appears since you're

00:33:27.850 --> 00:33:30.090 align:middle line:90%
using the spatial relationship.

00:33:30.090 --> 00:33:31.610 align:middle line:90%
RAMA RAMAKRISHNAN: Yes.

00:33:31.610 --> 00:33:33.810 align:middle line:84%
So both you and [MUTED]
are on the same sort

00:33:33.810 --> 00:33:37.410 align:middle line:84%
of trajectory, which is that we
need to somehow take this table

00:33:37.410 --> 00:33:38.930 align:middle line:90%
and make it into a vector.

00:33:38.930 --> 00:33:41.470 align:middle line:84%
And there are many ways, like
what you folks are describing,

00:33:41.470 --> 00:33:42.770 align:middle line:90%
to make it into a vector.

00:33:42.770 --> 00:33:43.760 align:middle line:90%
And it turns out--

00:33:43.760 --> 00:33:45.695 align:middle line:90%


00:33:45.695 --> 00:33:48.070 align:middle line:84%
this is all the things that
we've been discussing so far,

00:33:48.070 --> 00:33:50.250 align:middle line:84%
the varying length
issue and so on.

00:33:50.250 --> 00:33:54.250 align:middle line:84%
So, what we can do is we can
aggregate all these things.

00:33:54.250 --> 00:33:56.585 align:middle line:90%
If you just add them up--

00:33:56.585 --> 00:33:58.210 align:middle line:84%
this is what you
described, I believe--

00:33:58.210 --> 00:34:00.690 align:middle line:90%
it's called sum encoding.

00:34:00.690 --> 00:34:02.370 align:middle line:84%
And if, instead of
adding, you just

00:34:02.370 --> 00:34:04.350 align:middle line:84%
order them, meaning if you
look at the column and say,

00:34:04.350 --> 00:34:05.600 align:middle line:90%
is there any 1 in this column?

00:34:05.600 --> 00:34:07.430 align:middle line:84%
If there's any 1,
I'll put a stick a 1.

00:34:07.430 --> 00:34:08.780 align:middle line:90%
Otherwise it's 0.

00:34:08.780 --> 00:34:11.380 align:middle line:90%
It's called multi-hot encoding.

00:34:11.380 --> 00:34:14.540 align:middle line:84%
So if you look at this thing,
if you literally just go column

00:34:14.540 --> 00:34:18.179 align:middle line:84%
by column and count everything,
OK, there's a 1 here, 1 here.

00:34:18.179 --> 00:34:20.951 align:middle line:84%
Oh, wait, there are two
2s here, so you put a 2.

00:34:20.951 --> 00:34:22.159 align:middle line:90%
That's called count encoding.

00:34:22.159 --> 00:34:26.100 align:middle line:84%
Multi-hot encoding just looks
for any 1s and it puts 1.

00:34:26.100 --> 00:34:27.367 align:middle line:90%
Makes sense?

00:34:27.367 --> 00:34:29.659 align:middle line:84%
So by the way, there are many
ways to take these tables

00:34:29.659 --> 00:34:30.840 align:middle line:90%
and make them into vectors.

00:34:30.840 --> 00:34:33.020 align:middle line:84%
These two happen to
be very commonly used,

00:34:33.020 --> 00:34:34.690 align:middle line:84%
and they kind of
make common sense.

00:34:34.690 --> 00:34:41.520 align:middle line:90%


00:34:41.520 --> 00:34:43.659 align:middle line:84%
So this aggregation approach
that we just described

00:34:43.659 --> 00:34:46.739 align:middle line:84%
is called the Bag
of Words model.

00:34:46.739 --> 00:34:48.179 align:middle line:90%
The Bag of Words model.

00:34:48.179 --> 00:34:51.420 align:middle line:84%
And the reason is that, first
of all, this bag that we

00:34:51.420 --> 00:34:52.860 align:middle line:90%
have has words.

00:34:52.860 --> 00:34:55.179 align:middle line:84%
Either it counts whether
a word exists or not,

00:34:55.179 --> 00:35:00.380 align:middle line:84%
or it counts how many times
the word has appeared,

00:35:00.380 --> 00:35:03.620 align:middle line:84%
multi-hot versus sum
encoding, count encoding.

00:35:03.620 --> 00:35:07.720 align:middle line:84%
But more importantly, and this
goes back to your observation,

00:35:07.720 --> 00:35:11.560 align:middle line:84%
is that we have lost the
order of the words now.

00:35:11.560 --> 00:35:14.800 align:middle line:84%
Whether the phrase came in
was "the cat sat on the mat"

00:35:14.800 --> 00:35:19.160 align:middle line:84%
or "the mat sat on the cat", the
count encoding and the multi-hot

00:35:19.160 --> 00:35:20.960 align:middle line:90%
encoding are exactly the same.

00:35:20.960 --> 00:35:22.960 align:middle line:84%
There's no difference
because we're just

00:35:22.960 --> 00:35:25.140 align:middle line:84%
looking for the presence
or absence of words.

00:35:25.140 --> 00:35:26.000 align:middle line:90%
That's it.

00:35:26.000 --> 00:35:29.160 align:middle line:84%
We don't care in which
order they appear.

00:35:29.160 --> 00:35:30.780 align:middle line:90%
That's a huge limitation.

00:35:30.780 --> 00:35:33.940 align:middle line:84%
But shockingly, for many
applications, it doesn't matter.

00:35:33.940 --> 00:35:35.800 align:middle line:90%
It's good enough.

00:35:35.800 --> 00:35:39.242 align:middle line:84%
So it's called the
Bag of Words model.

00:35:39.242 --> 00:35:40.950 align:middle line:84%
So this is called the
Bag of Words model.

00:35:40.950 --> 00:35:43.640 align:middle line:90%


00:35:43.640 --> 00:35:45.925 align:middle line:84%
Now, does it have
any shortcomings?

00:35:45.925 --> 00:35:47.800 align:middle line:84%
I already talked about
the first shortcoming,

00:35:47.800 --> 00:35:50.580 align:middle line:84%
which is that it loses
sequentiality, the order.

00:35:50.580 --> 00:35:53.666 align:middle line:90%
We lost this order information.

00:35:53.666 --> 00:35:56.200 align:middle line:84%
We lose the meaning inherent
in the order of the words.

00:35:56.200 --> 00:35:57.660 align:middle line:84%
What are some other
issues with it?

00:35:57.660 --> 00:36:01.111 align:middle line:90%


00:36:01.111 --> 00:36:04.080 align:middle line:90%
AUDIENCE: [INAUDIBLE].

00:36:04.080 --> 00:36:07.090 align:middle line:84%
RAMA RAMAKRISHNAN: What
do you mean by that?

00:36:07.090 --> 00:36:11.954 align:middle line:84%
AUDIENCE: For example, you have
six different words [INAUDIBLE].

00:36:11.954 --> 00:36:14.390 align:middle line:84%
RAMA RAMAKRISHNAN: Right,
so there are lots of zeros,

00:36:14.390 --> 00:36:15.430 align:middle line:90%
not that many ones.

00:36:15.430 --> 00:36:17.722 align:middle line:84%
So it's a very sparse
amount of information,

00:36:17.722 --> 00:36:19.930 align:middle line:84%
but maybe you're carrying
around a lot of information

00:36:19.930 --> 00:36:21.190 align:middle line:90%
to make it all work.

00:36:21.190 --> 00:36:23.730 align:middle line:84%
Now, there are some tricks,
CS, Computer Science,

00:36:23.730 --> 00:36:26.310 align:middle line:84%
tricks to handle sparsity
in some clever ways,

00:36:26.310 --> 00:36:28.425 align:middle line:90%
but it is certainly an issue.

00:36:28.425 --> 00:36:30.050 align:middle line:84%
Now, the other issue
is that, let's say

00:36:30.050 --> 00:36:32.570 align:middle line:90%
the vocabulary is very long.

00:36:32.570 --> 00:36:35.530 align:middle line:84%
Each input sentence, whether
it's the collected works

00:36:35.530 --> 00:36:39.130 align:middle line:84%
of William Shakespeare or
the phrase "I love you",

00:36:39.130 --> 00:36:42.690 align:middle line:90%
will have the same length input.

00:36:42.690 --> 00:36:45.490 align:middle line:84%
We'll have the
same length input.

00:36:45.490 --> 00:36:48.090 align:middle line:84%
Because ultimately,
every incoming thing

00:36:48.090 --> 00:36:51.090 align:middle line:90%
gets mapped into one vector.

00:36:51.090 --> 00:36:54.300 align:middle line:90%
That feels a little suboptimal.

00:36:54.300 --> 00:36:56.050 align:middle line:84%
Clearly, the collected
works of literature

00:36:56.050 --> 00:37:00.010 align:middle line:84%
have a lot more stuff
going on in them.

00:37:00.010 --> 00:37:02.410 align:middle line:90%
So that's the problem.

00:37:02.410 --> 00:37:04.890 align:middle line:84%
In particular, very, very
small things that come in,

00:37:04.890 --> 00:37:07.910 align:middle line:84%
you'll be spending a lot of
compute on those long vectors

00:37:07.910 --> 00:37:09.950 align:middle line:90%
and processing them.

00:37:09.950 --> 00:37:12.110 align:middle line:84%
Now, you can
mitigate some of this

00:37:12.110 --> 00:37:15.272 align:middle line:84%
by choosing only the
most frequent words.

00:37:15.272 --> 00:37:17.230 align:middle line:84%
I think the English
language, I read somewhere,

00:37:17.230 --> 00:37:20.070 align:middle line:90%
has roughly 500,000 words or so.

00:37:20.070 --> 00:37:22.873 align:middle line:84%
But turns out the top 50,000
most frequent words were

00:37:22.873 --> 00:37:25.290 align:middle line:84%
responsible for just about
everything you're going to see,

00:37:25.290 --> 00:37:26.390 align:middle line:90%
ever.

00:37:26.390 --> 00:37:29.430 align:middle line:84%
And the other 50,000 are
what's called the long-tail.

00:37:29.430 --> 00:37:31.110 align:middle line:90%
They almost never happen.

00:37:31.110 --> 00:37:32.310 align:middle line:90%
You never see them.

00:37:32.310 --> 00:37:34.010 align:middle line:84%
So you can be very
pragmatic and say,

00:37:34.010 --> 00:37:35.870 align:middle line:84%
I'm not going to take
every little word

00:37:35.870 --> 00:37:37.490 align:middle line:90%
that I see in my vocabulary.

00:37:37.490 --> 00:37:39.450 align:middle line:84%
I'm going to only take
the most frequent words.

00:37:39.450 --> 00:37:41.993 align:middle line:84%
I'm just going to
ignore the rest.

00:37:41.993 --> 00:37:43.410 align:middle line:84%
I'm just going to
ignore the rest.

00:37:43.410 --> 00:37:46.660 align:middle line:90%


00:37:46.660 --> 00:37:49.310 align:middle line:90%
But if you ignore the rest--

00:37:49.310 --> 00:37:51.750 align:middle line:90%
let's say there is one word--

00:37:51.750 --> 00:37:54.830 align:middle line:84%
let's take some
Shakespeare word, Hamlet.

00:37:54.830 --> 00:37:57.830 align:middle line:84%
Let's assume that you ignore the
word Hamlet from your training

00:37:57.830 --> 00:37:58.330 align:middle line:90%
corpus.

00:37:58.330 --> 00:37:59.905 align:middle line:84%
You just delete it
because it's not

00:37:59.905 --> 00:38:02.030 align:middle line:84%
one of the top, most frequent
things you have seen.

00:38:02.030 --> 00:38:05.560 align:middle line:84%
And then somebody sends
you a text saying,

00:38:05.560 --> 00:38:08.240 align:middle line:90%
"Hamlet was a bad prince.

00:38:08.240 --> 00:38:10.820 align:middle line:84%
Analyze the sentiment
of the sentence", well,

00:38:10.820 --> 00:38:13.832 align:middle line:84%
when you see Hamlet, what
is your system going to do?

00:38:13.832 --> 00:38:15.540 align:middle line:84%
It's going to look at
the Hamlet and say,

00:38:15.540 --> 00:38:18.003 align:middle line:84%
I can't see it in my
vocabulary anywhere.

00:38:18.003 --> 00:38:19.920 align:middle line:84%
And if it can't see it
in the vocabulary, what

00:38:19.920 --> 00:38:22.400 align:middle line:90%
is the only thing it can do?

00:38:22.400 --> 00:38:25.040 align:middle line:90%
Replace it with UNK.

00:38:25.040 --> 00:38:28.440 align:middle line:84%
So that's where UNK
comes into the picture.

00:38:28.440 --> 00:38:29.920 align:middle line:84%
So whenever it
can't see something

00:38:29.920 --> 00:38:31.660 align:middle line:84%
in the vocabulary
in a new input,

00:38:31.660 --> 00:38:33.400 align:middle line:84%
it just replaces
it with UNK, which

00:38:33.400 --> 00:38:38.880 align:middle line:84%
means that if you had ignored
Romeo and Juliet and Hamlet

00:38:38.880 --> 00:38:41.240 align:middle line:84%
in the training
corpus, all of them

00:38:41.240 --> 00:38:43.360 align:middle line:84%
are going to be replaced
by the same UNK, which

00:38:43.360 --> 00:38:46.760 align:middle line:84%
means that we can't distinguish
between them anymore.

00:38:46.760 --> 00:38:48.880 align:middle line:84%
AUDIENCE: So is this
where hallucination

00:38:48.880 --> 00:38:54.147 align:middle line:84%
comes into play here, where it
doesn't recognize [INAUDIBLE].

00:38:54.147 --> 00:38:55.980 align:middle line:84%
RAMA RAMAKRISHNAN: Ah,
interesting question.

00:38:55.980 --> 00:38:57.740 align:middle line:84%
Is this where
hallucination comes up?

00:38:57.740 --> 00:38:59.760 align:middle line:90%
Actually, as it turns out, no.

00:38:59.760 --> 00:39:02.900 align:middle line:84%
As we will see when we
talk about LLMs later,

00:39:02.900 --> 00:39:05.890 align:middle line:84%
LLMs actually will not
have this UNK problem

00:39:05.890 --> 00:39:08.140 align:middle line:84%
because they use a different
tokenization scheme which

00:39:08.140 --> 00:39:10.848 align:middle line:84%
can handle anything you throw at
it, including new stuff you just

00:39:10.848 --> 00:39:12.460 align:middle line:90%
made up.

00:39:12.460 --> 00:39:15.940 align:middle line:90%
So we'll come back to that.

00:39:15.940 --> 00:39:17.820 align:middle line:90%
So that's what we have.

00:39:17.820 --> 00:39:20.560 align:middle line:84%
And so what we're going to do
is, despite its shortcomings,

00:39:20.560 --> 00:39:23.940 align:middle line:84%
Bag of Words is actually a
really good default for many NLP

00:39:23.940 --> 00:39:25.100 align:middle line:90%
tasks.

00:39:25.100 --> 00:39:27.460 align:middle line:84%
And in the spirit of, "do
the simple stuff first and do

00:39:27.460 --> 00:39:29.400 align:middle line:84%
complicated things only if the
simple stuff doesn't work",

00:39:29.400 --> 00:39:31.192 align:middle line:84%
we'll use the Bag of
Words model right now.

00:39:31.192 --> 00:39:36.340 align:middle line:84%
So we'll switch to a Colab
and see how it's done.

00:39:36.340 --> 00:39:39.860 align:middle line:84%
So here, the application
we're going to work with

00:39:39.860 --> 00:39:42.260 align:middle line:90%
is kind of a fun application.

00:39:42.260 --> 00:39:45.980 align:middle line:84%
We're going to try to
predict the genre of songs.

00:39:45.980 --> 00:39:48.700 align:middle line:84%
It's a nice
classification use case.

00:39:48.700 --> 00:39:51.660 align:middle line:84%
So we want to take
some arbitrary song

00:39:51.660 --> 00:39:56.300 align:middle line:84%
and then classify it into
either hip hop, rock, or pop.

00:39:56.300 --> 00:40:00.510 align:middle line:84%
And so, for instance,
this is the kind

00:40:00.510 --> 00:40:02.590 align:middle line:90%
of lyrics you're going to see.

00:40:02.590 --> 00:40:04.970 align:middle line:84%
And as you will see
in this data set,

00:40:04.970 --> 00:40:07.950 align:middle line:84%
just a quick word of
caution the data set does

00:40:07.950 --> 00:40:13.350 align:middle line:84%
have lyrics which may not be
safe for work, as it were.

00:40:13.350 --> 00:40:16.550 align:middle line:84%
So I'm not going to be exploring
the lyrics in the Colab,

00:40:16.550 --> 00:40:19.650 align:middle line:84%
but I just want you
to be aware of it.

00:40:19.650 --> 00:40:22.470 align:middle line:84%
But it's just some data set that
we downloaded from somewhere.

00:40:22.470 --> 00:40:23.770 align:middle line:90%
It's got all these lyrics.

00:40:23.770 --> 00:40:26.630 align:middle line:84%
So we're going to try to
classify each verse that we

00:40:26.630 --> 00:40:29.010 align:middle line:84%
see into one of three
things, hip hop, rock or pop.

00:40:29.010 --> 00:40:32.330 align:middle line:84%
It's a multi-class
classification problem.

00:40:32.330 --> 00:40:34.990 align:middle line:84%
Actually, what is the simplest
neural network-based classifier

00:40:34.990 --> 00:40:38.460 align:middle line:90%
we can build for this problem?

00:40:38.460 --> 00:40:41.770 align:middle line:90%


00:40:41.770 --> 00:40:43.390 align:middle line:84%
So what is the
simplest neural network

00:40:43.390 --> 00:40:46.310 align:middle line:90%
we can build for this problem?

00:40:46.310 --> 00:40:47.610 align:middle line:90%
So remember, what is the input?

00:40:47.610 --> 00:40:49.930 align:middle line:84%
The input is going to be
a bunch of song lyrics.

00:40:49.930 --> 00:40:52.430 align:middle line:84%
It's going to be a really
long song for all you know.

00:40:52.430 --> 00:40:54.950 align:middle line:84%
And we're going to use
the Bag of Words model.

00:40:54.950 --> 00:40:56.510 align:middle line:84%
And let's assume
for a moment that we

00:40:56.510 --> 00:40:59.050 align:middle line:90%
will use multi-hot encoding.

00:40:59.050 --> 00:41:02.710 align:middle line:84%
We'll create a
vocabulary from the song,

00:41:02.710 --> 00:41:04.710 align:middle line:84%
we'll take all the songs,
we'll process them,

00:41:04.710 --> 00:41:07.670 align:middle line:84%
run it through S-T-I-E,
we'll do multi-hot encoding,

00:41:07.670 --> 00:41:11.550 align:middle line:84%
which means that every song
that comes in will be a vector

00:41:11.550 --> 00:41:14.170 align:middle line:90%
that's how long?

00:41:14.170 --> 00:41:17.290 align:middle line:90%
It will be as long as the?

00:41:17.290 --> 00:41:20.690 align:middle line:90%
Correct, as the vocabulary size.

00:41:20.690 --> 00:41:24.430 align:middle line:84%
So maybe what comes
in is this phrase,

00:41:24.430 --> 00:41:26.930 align:middle line:84%
since it's supposed to be songs,
I'll say something which is

00:41:26.930 --> 00:41:32.020 align:middle line:84%
probably common to 90%
of songs, "I love you".

00:41:32.020 --> 00:41:38.410 align:middle line:84%
That goes in, it goes
into our S-T-I-E process.

00:41:38.410 --> 00:41:45.106 align:middle line:84%
And then this S-T-I-E process
gives us a vector, which is x1,

00:41:45.106 --> 00:41:50.090 align:middle line:84%
x2, all the way to xv,
where v stands for the size

00:41:50.090 --> 00:41:51.290 align:middle line:90%
of the vocabulary.

00:41:51.290 --> 00:41:56.220 align:middle line:84%
So that's our input
layer all the way.

00:41:56.220 --> 00:41:59.270 align:middle line:84%
So, knowing what we know now
about deep learning, what can we

00:41:59.270 --> 00:41:59.770 align:middle line:90%
do next?

00:41:59.770 --> 00:42:03.663 align:middle line:90%


00:42:03.663 --> 00:42:05.080 align:middle line:84%
AUDIENCE: Maybe
I'm getting ahead,

00:42:05.080 --> 00:42:07.740 align:middle line:84%
but wouldn't the
classifier just be--

00:42:07.740 --> 00:42:09.740 align:middle line:84%
like, the baseline
would be classify it

00:42:09.740 --> 00:42:11.878 align:middle line:90%
as the most common genre?

00:42:11.878 --> 00:42:13.920 align:middle line:84%
RAMA RAMAKRISHNAN: That
is the baseline, correct.

00:42:13.920 --> 00:42:16.060 align:middle line:84%
And we'll come to the
baseline a bit later.

00:42:16.060 --> 00:42:19.780 align:middle line:84%
But here, I'm saying suppose you
wanted to build a neural network

00:42:19.780 --> 00:42:21.060 align:middle line:90%
model for this.

00:42:21.060 --> 00:42:23.340 align:middle line:90%
How would you set it up?

00:42:23.340 --> 00:42:25.660 align:middle line:84%
AUDIENCE: Do you think about
the layers that you want?

00:42:25.660 --> 00:42:26.020 align:middle line:90%
RAMA RAMAKRISHNAN: Right.

00:42:26.020 --> 00:42:27.460 align:middle line:84%
And what is the
simplest thing you

00:42:27.460 --> 00:42:28.860 align:middle line:90%
can do with a neural network?

00:42:28.860 --> 00:42:30.260 align:middle line:90%
How many layers?

00:42:30.260 --> 00:42:31.553 align:middle line:90%
AUDIENCE: No additional layers.

00:42:31.553 --> 00:42:33.220 align:middle line:84%
RAMA RAMAKRISHNAN:
Well, then it becomes

00:42:33.220 --> 00:42:35.340 align:middle line:84%
problematic with a
singlular neural network

00:42:35.340 --> 00:42:37.515 align:middle line:84%
because it could just be
logistical regression.

00:42:37.515 --> 00:42:38.640 align:middle line:90%
AUDIENCE: One hidden layer.

00:42:38.640 --> 00:42:40.640 align:middle line:84%
RAMA RAMAKRISHNAN:
Yes, thank you.

00:42:40.640 --> 00:42:43.140 align:middle line:84%
I'm being a little squishy about
this because there are some

00:42:43.140 --> 00:42:45.515 align:middle line:84%
people who would be like,
well, even if there's no hidden

00:42:45.515 --> 00:42:48.055 align:middle line:84%
layers, if you're using ReLUs
and this and that and sigma,

00:42:48.055 --> 00:42:50.680 align:middle line:84%
maybe it's a neural network, and
I don't want to get into that,

00:42:50.680 --> 00:42:53.560 align:middle line:84%
"how many angels in the
tip of a pin" argument.

00:42:53.560 --> 00:42:55.267 align:middle line:84%
So yeah, we need
one hidden layer.

00:42:55.267 --> 00:42:57.600 align:middle line:84%
In this course, we need at
least one hidden layer for it

00:42:57.600 --> 00:42:59.040 align:middle line:90%
to qualify as a neural network.

00:42:59.040 --> 00:43:01.060 align:middle line:84%
OK, so let's have
a hidden layer.

00:43:01.060 --> 00:43:02.810 align:middle line:84%
And we'll have a bunch
of ReLUs, as usual.

00:43:02.810 --> 00:43:05.360 align:middle line:90%


00:43:05.360 --> 00:43:06.920 align:middle line:90%
Bunch of ReLUs.

00:43:06.920 --> 00:43:08.860 align:middle line:84%
And I'll ignore all the
arrows between them.

00:43:08.860 --> 00:43:10.280 align:middle line:90%
It's kind of a pain.

00:43:10.280 --> 00:43:12.180 align:middle line:84%
And then we come to
the output layer.

00:43:12.180 --> 00:43:15.360 align:middle line:84%
And what should the
output layer be?

00:43:15.360 --> 00:43:18.400 align:middle line:84%
How many nodes do we
need in the output layer?

00:43:18.400 --> 00:43:21.560 align:middle line:90%
Three, hip hop, rock, or pop.

00:43:21.560 --> 00:43:23.220 align:middle line:84%
And then that layer
is called what?

00:43:23.220 --> 00:43:25.400 align:middle line:90%
What activation function?

00:43:25.400 --> 00:43:26.560 align:middle line:90%
Softmax, perfect.

00:43:26.560 --> 00:43:27.060 align:middle line:90%
Love it.

00:43:27.060 --> 00:43:28.520 align:middle line:90%
Love this class.

00:43:28.520 --> 00:43:35.840 align:middle line:84%
All right, three things,
rock, hip hop, and pop.

00:43:35.840 --> 00:43:39.200 align:middle line:84%
And this is a
softmax right there.

00:43:39.200 --> 00:43:43.160 align:middle line:84%
And then it's going to give us
three probabilities that add

00:43:43.160 --> 00:43:45.420 align:middle line:90%
up to 1 because of the softmax.

00:43:45.420 --> 00:43:47.460 align:middle line:90%
So that's our basic network.

00:43:47.460 --> 00:43:47.960 align:middle line:90%
Perfect.

00:43:47.960 --> 00:43:49.107 align:middle line:90%
Yeah.

00:43:49.107 --> 00:43:51.440 align:middle line:84%
AUDIENCE: Why do you need
those probabilities at the end

00:43:51.440 --> 00:43:54.510 align:middle line:84%
if you just want to identify
the most likely genre?

00:43:54.510 --> 00:43:58.147 align:middle line:84%
The softmax just gives you a
way to add them all up at once.

00:43:58.147 --> 00:43:59.230 align:middle line:90%
Why do you need a softmax?

00:43:59.230 --> 00:44:01.563 align:middle line:84%
Why don't you just take the
max value and say it's that?

00:44:01.563 --> 00:44:03.397 align:middle line:84%
RAMA RAMAKRISHNAN: Oh,
interesting question.

00:44:03.397 --> 00:44:05.170 align:middle line:84%
Why can't we just
produce three numbers

00:44:05.170 --> 00:44:06.850 align:middle line:90%
and grab the maximum number?

00:44:06.850 --> 00:44:09.890 align:middle line:84%
So, it turns out
finding the maximum

00:44:09.890 --> 00:44:14.330 align:middle line:84%
of a bunch of numbers,
that function is not very

00:44:14.330 --> 00:44:16.930 align:middle line:90%
friendly for differentiation.

00:44:16.930 --> 00:44:19.170 align:middle line:84%
And ultimately, you
want to take this output

00:44:19.170 --> 00:44:22.290 align:middle line:84%
and run it through a loss
function like cross-entropy

00:44:22.290 --> 00:44:24.930 align:middle line:84%
and then be able to
run backprop on it.

00:44:24.930 --> 00:44:27.250 align:middle line:84%
And so, fundamentally,
backpropagation

00:44:27.250 --> 00:44:30.490 align:middle line:84%
is just differentiation, and it
requires everything inside of it

00:44:30.490 --> 00:44:33.250 align:middle line:90%
to have well-behaved gradients.

00:44:33.250 --> 00:44:37.250 align:middle line:84%
And so this little max function
is actually not well-behaved,

00:44:37.250 --> 00:44:41.130 align:middle line:84%
which is why we have a soft
version of it, softmax, which

00:44:41.130 --> 00:44:43.250 align:middle line:90%
makes it easy to differentiate.

00:44:43.250 --> 00:44:45.150 align:middle line:84%
So I can tell you
more about it offline,

00:44:45.150 --> 00:44:46.400 align:middle line:90%
but that's the quick synopsis.

00:44:46.400 --> 00:44:49.003 align:middle line:90%


00:44:49.003 --> 00:44:51.670 align:middle line:84%
So a lot of tricks you will see
in the neural network literature

00:44:51.670 --> 00:44:55.670 align:middle line:84%
are ways to avoid the
problem of having certain--

00:44:55.670 --> 00:44:57.370 align:middle line:84%
like, the obvious
choice of function

00:44:57.370 --> 00:44:59.328 align:middle line:84%
will not be well-behaved
for differentiation.

00:44:59.328 --> 00:45:00.870 align:middle line:84%
That's why you need
to go through all

00:45:00.870 --> 00:45:02.910 align:middle line:90%
these other mechanisms.

00:45:02.910 --> 00:45:04.663 align:middle line:84%
Much like we couldn't
just say accuracy,

00:45:04.663 --> 00:45:06.830 align:middle line:84%
why don't we just maximize
accuracy instead of doing

00:45:06.830 --> 00:45:08.370 align:middle line:90%
this cross-entropy business?

00:45:08.370 --> 00:45:10.790 align:middle line:90%
Same reason.

00:45:10.790 --> 00:45:11.590 align:middle line:90%
All right.

00:45:11.590 --> 00:45:13.050 align:middle line:90%
So let's come back here.

00:45:13.050 --> 00:45:20.590 align:middle line:90%


00:45:20.590 --> 00:45:25.350 align:middle line:84%
So that's what we created on
the thing, "cat sat on the mat"

00:45:25.350 --> 00:45:28.050 align:middle line:90%
vocabulary thing and so on.

00:45:28.050 --> 00:45:30.930 align:middle line:84%
And I was playing
around with it earlier,

00:45:30.930 --> 00:45:34.150 align:middle line:84%
and so I found that eight
ReLU neurons were pretty

00:45:34.150 --> 00:45:35.210 align:middle line:90%
good to get the job done.

00:45:35.210 --> 00:45:38.070 align:middle line:84%
So I'm just going to go
with eight ReLU neurons

00:45:38.070 --> 00:45:39.870 align:middle line:90%
in the hidden layer.

00:45:39.870 --> 00:45:44.030 align:middle line:84%
So I think that brings
us to the Colab.

00:45:44.030 --> 00:45:46.950 align:middle line:90%
Yeah, so let's switch to Colab.

00:45:46.950 --> 00:45:47.710 align:middle line:90%
All right.

00:45:47.710 --> 00:45:49.970 align:middle line:90%
So that's what we have here.

00:45:49.970 --> 00:45:51.693 align:middle line:84%
There's a little bit
of verbiage here,

00:45:51.693 --> 00:45:53.610 align:middle line:84%
which just describes
what I just talked about.

00:45:53.610 --> 00:45:56.490 align:middle line:84%
So we'll do the usual things
and upload everything,

00:45:56.490 --> 00:46:00.010 align:middle line:84%
import everything we want,
TensorFlow and Keras,

00:46:00.010 --> 00:46:03.610 align:middle line:84%
and the Holy Trinity of
NumPy, pandas, and Matplotlib,

00:46:03.610 --> 00:46:07.010 align:middle line:84%
set the random seed,
as usual, at 42.

00:46:07.010 --> 00:46:09.250 align:middle line:84%
There's our S-T-I-E
framework here.

00:46:09.250 --> 00:46:13.290 align:middle line:84%
And the nice thing is that all
four of these things, S-T-I-E,

00:46:13.290 --> 00:46:15.730 align:middle line:84%
are beautifully implemented
in Keras as a single,

00:46:15.730 --> 00:46:19.210 align:middle line:84%
simple layer called the
text vectorization layer,

00:46:19.210 --> 00:46:20.650 align:middle line:90%
which is nice.

00:46:20.650 --> 00:46:23.610 align:middle line:84%
So we have the text
vectorization array right here.

00:46:23.610 --> 00:46:25.290 align:middle line:84%
And so in our
first example, what

00:46:25.290 --> 00:46:28.370 align:middle line:84%
we'll do is we will use a
default standardization, which

00:46:28.370 --> 00:46:30.950 align:middle line:84%
will just remove punctuation,
convert to lowercase.

00:46:30.950 --> 00:46:32.670 align:middle line:84%
We'll use the
default tokenization,

00:46:32.670 --> 00:46:35.450 align:middle line:84%
which just means split on
the space between words.

00:46:35.450 --> 00:46:38.490 align:middle line:84%
And then we will set
the output to multi-hot.

00:46:38.490 --> 00:46:40.810 align:middle line:84%
All the things we talked
about, Keras will just

00:46:40.810 --> 00:46:42.050 align:middle line:90%
do it for you automatically.

00:46:42.050 --> 00:46:44.770 align:middle line:84%
And so output mode,
multi-hot, standardize

00:46:44.770 --> 00:46:46.730 align:middle line:84%
this, split
whitespace, and boom,

00:46:46.730 --> 00:46:48.390 align:middle line:84%
you run the text
vectorization thing.

00:46:48.390 --> 00:46:51.300 align:middle line:84%
And once you do it, Keras
creates this textualization

00:46:51.300 --> 00:46:53.380 align:middle line:84%
layer with these
settings, and it's now

00:46:53.380 --> 00:46:55.380 align:middle line:90%
ready to swing into action.

00:46:55.380 --> 00:46:57.820 align:middle line:84%
So what does swing into
action actually mean?

00:46:57.820 --> 00:47:00.360 align:middle line:84%
Well, now you need to actually
feed it a training corpus so

00:47:00.360 --> 00:47:02.860 align:middle line:84%
that it can do all the things
it's supposed to do and create

00:47:02.860 --> 00:47:04.000 align:middle line:90%
the vocabulary for you.

00:47:04.000 --> 00:47:06.780 align:middle line:90%


00:47:06.780 --> 00:47:08.720 align:middle line:84%
And that thing is
called the adapt method.

00:47:08.720 --> 00:47:11.920 align:middle line:84%
So we create a tiny
training corpus for us.

00:47:11.920 --> 00:47:14.720 align:middle line:90%
This is our data set.

00:47:14.720 --> 00:47:16.860 align:middle line:84%
This is a bunch of words
from some of these lyrics.

00:47:16.860 --> 00:47:19.420 align:middle line:84%
And then what we'll do is we'll
take this layer that we just

00:47:19.420 --> 00:47:21.480 align:middle line:84%
defined here that
we have set up here,

00:47:21.480 --> 00:47:24.420 align:middle line:84%
and then we will ask this
layer to actually create

00:47:24.420 --> 00:47:28.180 align:middle line:84%
the vocabulary using
this adapt command.

00:47:28.180 --> 00:47:30.360 align:middle line:84%
It will index the
vocabulary, and it's done.

00:47:30.360 --> 00:47:32.220 align:middle line:84%
And once it does
it, you can actually

00:47:32.220 --> 00:47:34.720 align:middle line:90%
ask it for the vocabulary--

00:47:34.720 --> 00:47:37.840 align:middle line:84%
this is vocabulary-- using
the get_vocabulary command.

00:47:37.840 --> 00:47:39.840 align:middle line:84%
And so, first of all,
how long is the vocab?

00:47:39.840 --> 00:47:40.740 align:middle line:90%
17.

00:47:40.740 --> 00:47:43.140 align:middle line:90%
17 words, 17 tokens.

00:47:43.140 --> 00:47:44.980 align:middle line:90%
What are they?

00:47:44.980 --> 00:47:45.952 align:middle line:90%
You can see here.

00:47:45.952 --> 00:47:47.660 align:middle line:84%
And you can see, these
are all the words.

00:47:47.660 --> 00:47:50.760 align:middle line:84%
And you can see it has stuck
an UNK in the very beginning.

00:47:50.760 --> 00:47:52.040 align:middle line:90%
It's sort of the default.

00:47:52.040 --> 00:47:54.857 align:middle line:84%
By the way, just a
little programming tip,

00:47:54.857 --> 00:47:56.940 align:middle line:84%
if you don't have a ton
of programming experience,

00:47:56.940 --> 00:47:59.120 align:middle line:84%
if you want to print
these Python objects,

00:47:59.120 --> 00:48:02.440 align:middle line:84%
like lists and so on, in a
pretty way, one trick that

00:48:02.440 --> 00:48:05.800 align:middle line:84%
often works is just stick
it into a data frame

00:48:05.800 --> 00:48:07.265 align:middle line:90%
and then print it.

00:48:07.265 --> 00:48:09.140 align:middle line:84%
Usually, it'll print it
in a much better way.

00:48:09.140 --> 00:48:11.640 align:middle line:90%
So you can see it like that.

00:48:11.640 --> 00:48:14.120 align:middle line:84%
So you can see here, UNK,
arrays, blah, blah, blah, blah,

00:48:14.120 --> 00:48:15.000 align:middle line:90%
blah.

00:48:15.000 --> 00:48:17.600 align:middle line:84%
And you can see integer
zeros assigned the UNK token.

00:48:17.600 --> 00:48:20.640 align:middle line:84%
By the way, how come it
picked the word "arrays"

00:48:20.640 --> 00:48:22.060 align:middle line:90%
as the second entry?

00:48:22.060 --> 00:48:28.540 align:middle line:84%
Why not something like
"an" or why not "a"?

00:48:28.540 --> 00:48:31.340 align:middle line:84%
How come "a" is not
chosen as a second entry?

00:48:31.340 --> 00:48:33.340 align:middle line:84%
Why did it pick
"arrays", do you think?

00:48:33.340 --> 00:48:40.880 align:middle line:90%


00:48:40.880 --> 00:48:43.160 align:middle line:84%
AUDIENCE: Maybe
it tried the words

00:48:43.160 --> 00:48:46.490 align:middle line:84%
that are most influential on
the meaning of the sentence

00:48:46.490 --> 00:48:49.368 align:middle line:84%
to be on the top and
less influential.

00:48:49.368 --> 00:48:50.910 align:middle line:84%
RAMA RAMAKRISHNAN:
But at this point,

00:48:50.910 --> 00:48:54.170 align:middle line:84%
it doesn't know what
we're going to use it for,

00:48:54.170 --> 00:48:56.210 align:middle line:84%
so it has no way to
know what word is useful

00:48:56.210 --> 00:48:59.450 align:middle line:84%
because we haven't told it
how we're going to use it.

00:48:59.450 --> 00:49:02.110 align:middle line:84%
But you're kind of
on the right track.

00:49:02.110 --> 00:49:06.470 align:middle line:84%
So what Keras does is it
will find all these tokens,

00:49:06.470 --> 00:49:09.730 align:middle line:84%
and then it'll actually
just sort them by frequency.

00:49:09.730 --> 00:49:11.952 align:middle line:84%
So the most frequent,
as it turns out,

00:49:11.952 --> 00:49:13.410 align:middle line:84%
in those four
sentences we gave it,

00:49:13.410 --> 00:49:15.210 align:middle line:84%
happened to be
the word "arrays".

00:49:15.210 --> 00:49:17.235 align:middle line:84%
That's why "arrays"
is showing up on top.

00:49:17.235 --> 00:49:18.610 align:middle line:84%
And you can actually
confirm this

00:49:18.610 --> 00:49:20.510 align:middle line:90%
by going to the little data set.

00:49:20.510 --> 00:49:23.690 align:middle line:84%
And you can see here "arrays"
shows up here and once up here

00:49:23.690 --> 00:49:24.270 align:middle line:90%
twice.

00:49:24.270 --> 00:49:27.790 align:middle line:84%
And that's why it
came up on top.

00:49:27.790 --> 00:49:29.770 align:middle line:90%
So that's what we have.

00:49:29.770 --> 00:49:32.790 align:middle line:84%
And now that we
have populated this,

00:49:32.790 --> 00:49:34.910 align:middle line:84%
we can run any sentence
through it easily.

00:49:34.910 --> 00:49:36.370 align:middle line:90%
Yeah.

00:49:36.370 --> 00:49:38.808 align:middle line:84%
AUDIENCE: Does it matter
that it's on the top?

00:49:38.808 --> 00:49:40.350 align:middle line:84%
RAMA RAMAKRISHNAN:
It doesn't matter.

00:49:40.350 --> 00:49:41.230 align:middle line:90%
It doesn't matter.

00:49:41.230 --> 00:49:43.250 align:middle line:84%
The reason why it's
helpful later on

00:49:43.250 --> 00:49:45.790 align:middle line:84%
is because suppose you
tell Keras, hey, don't take

00:49:45.790 --> 00:49:49.390 align:middle line:84%
every word you see here, give
me only the most frequent

00:49:49.390 --> 00:49:50.277 align:middle line:90%
100 words.

00:49:50.277 --> 00:49:51.610 align:middle line:90%
I don't want any more than that.

00:49:51.610 --> 00:49:52.690 align:middle line:90%
It can easily do that.

00:49:52.690 --> 00:49:53.570 align:middle line:90%
That's the reason.

00:49:53.570 --> 00:49:54.230 align:middle line:90%
Yeah.

00:49:54.230 --> 00:49:56.470 align:middle line:84%
AUDIENCE: We could do
it after [INAUDIBLE].

00:49:56.470 --> 00:50:00.712 align:middle line:90%


00:50:00.712 --> 00:50:02.670 align:middle line:84%
RAMA RAMAKRISHNAN: This
is just the vocabulary.

00:50:02.670 --> 00:50:05.443 align:middle line:84%
So basically, you give
it all these phrases--

00:50:05.443 --> 00:50:07.610 align:middle line:84%
it happened to be just four
phrases in our example--

00:50:07.610 --> 00:50:09.318 align:middle line:84%
and then it finds all
the distinct words,

00:50:09.318 --> 00:50:10.490 align:middle line:90%
and it does all that stuff.

00:50:10.490 --> 00:50:12.230 align:middle line:84%
And then it has
created a vocabulary.

00:50:12.230 --> 00:50:17.070 align:middle line:84%
At this point, the training
corpus you fed it is forgotten.

00:50:17.070 --> 00:50:19.190 align:middle line:84%
And the only thing that
survived this processing

00:50:19.190 --> 00:50:20.370 align:middle line:90%
is just the vocabulary.

00:50:20.370 --> 00:50:21.055 align:middle line:90%
That's it.

00:50:21.055 --> 00:50:23.430 align:middle line:84%
And now, we have to start
applying it to any kind of text

00:50:23.430 --> 00:50:25.790 align:middle line:90%
we want to use it for.

00:50:25.790 --> 00:50:29.410 align:middle line:84%
So here, when you come back
here, this is what we have.

00:50:29.410 --> 00:50:31.870 align:middle line:84%
And so what you can do is
you can take any sentence

00:50:31.870 --> 00:50:33.670 align:middle line:84%
and you can just run
it through the layer

00:50:33.670 --> 00:50:36.295 align:middle line:84%
to make sure that it actually is
doing the right thing for you.

00:50:36.295 --> 00:50:37.530 align:middle line:90%
So we'll take this sentence.

00:50:37.530 --> 00:50:40.070 align:middle line:84%
We will then run it through
the text vectorization layer

00:50:40.070 --> 00:50:41.730 align:middle line:84%
by just passing that
sentence into it,

00:50:41.730 --> 00:50:43.060 align:middle line:90%
and then we can just print it.

00:50:43.060 --> 00:50:46.000 align:middle line:90%


00:50:46.000 --> 00:50:47.740 align:middle line:84%
So now, it's giving
you a tensor.

00:50:47.740 --> 00:50:50.000 align:middle line:84%
This is a multi-hot
encoded tensor

00:50:50.000 --> 00:50:52.000 align:middle line:90%
with all these ones and zeros.

00:50:52.000 --> 00:50:55.960 align:middle line:84%
So note that this
tensor is 17 units long,

00:50:55.960 --> 00:50:58.380 align:middle line:84%
which is a good check because
our vocabulary is 17 long,

00:50:58.380 --> 00:51:00.520 align:middle line:90%
so it better match that.

00:51:00.520 --> 00:51:04.100 align:middle line:84%
Now, recall that the UNK token
is at the first location.

00:51:04.100 --> 00:51:05.440 align:middle line:90%
It's at index 0.

00:51:05.440 --> 00:51:10.440 align:middle line:84%
And it says that this encoded
sentence does have an UNK word.

00:51:10.440 --> 00:51:14.640 align:middle line:90%
So why is that?

00:51:14.640 --> 00:51:15.820 align:middle line:90%
What is this UNK word?

00:51:15.820 --> 00:51:19.040 align:middle line:90%
Anyone can guess?

00:51:19.040 --> 00:51:23.080 align:middle line:84%
Well, it turns out to be
the word still, I think.

00:51:23.080 --> 00:51:24.900 align:middle line:84%
Yeah, still is not
in our vocabulary,

00:51:24.900 --> 00:51:27.640 align:middle line:84%
because if the four sentences,
which is our training corpus

00:51:27.640 --> 00:51:29.680 align:middle line:84%
used to build the
vocabulary, they

00:51:29.680 --> 00:51:31.320 align:middle line:84%
had a lot write and
rewrite, but there

00:51:31.320 --> 00:51:32.880 align:middle line:90%
was no still in it anywhere.

00:51:32.880 --> 00:51:34.960 align:middle line:84%
That's why there
is an UNK for it.

00:51:34.960 --> 00:51:38.620 align:middle line:84%
We can just double-check that by
asking Python, is it vocabulary?

00:51:38.620 --> 00:51:39.260 align:middle line:90%
No, it is not.

00:51:39.260 --> 00:51:41.680 align:middle line:84%
OK, now, in the spirit of
making small changes to the code

00:51:41.680 --> 00:51:43.180 align:middle line:84%
to understand what's
going on, which

00:51:43.180 --> 00:51:45.620 align:middle line:84%
is a very useful
tip for folks who

00:51:45.620 --> 00:51:47.860 align:middle line:84%
don't have a ton of
programming knowledge,

00:51:47.860 --> 00:51:52.608 align:middle line:84%
let's say that you send the
phrase Sloan, HODL, and DMD.

00:51:52.608 --> 00:51:54.900 align:middle line:84%
I think you will agree with
me that none of these words

00:51:54.900 --> 00:51:57.100 align:middle line:90%
is in the training corpus.

00:51:57.100 --> 00:52:01.220 align:middle line:84%
So what is the
multi-hot encoded vector

00:52:01.220 --> 00:52:03.490 align:middle line:84%
for this phrase,
Sloan, HODL, DMD?

00:52:03.490 --> 00:52:07.320 align:middle line:90%


00:52:07.320 --> 00:52:07.820 align:middle line:90%
Three?

00:52:07.820 --> 00:52:11.420 align:middle line:90%


00:52:11.420 --> 00:52:12.560 align:middle line:90%
It's not count encoding.

00:52:12.560 --> 00:52:14.780 align:middle line:90%
It's multi-hot encoding.

00:52:14.780 --> 00:52:16.900 align:middle line:90%
It's going to be 1, 0, 0.

00:52:16.900 --> 00:52:18.540 align:middle line:84%
So you can see
here, in this case,

00:52:18.540 --> 00:52:21.920 align:middle line:90%
remember the vocabulary is 17.

00:52:21.920 --> 00:52:24.100 align:middle line:84%
So each of these
words is going to be

00:52:24.100 --> 00:52:27.420 align:middle line:90%
a one followed by 16 zeros.

00:52:27.420 --> 00:52:29.140 align:middle line:84%
And then it's going
to multi-hot encode

00:52:29.140 --> 00:52:32.980 align:middle line:84%
them, which means the three ones
in the column just become a one.

00:52:32.980 --> 00:52:36.940 align:middle line:90%
So you still have only this one.

00:52:36.940 --> 00:52:37.860 align:middle line:90%
All right, good.

00:52:37.860 --> 00:52:42.310 align:middle line:84%
So now now, let's actually
get to the data set we have,

00:52:42.310 --> 00:52:45.230 align:middle line:90%
this 90,000 songs.

00:52:45.230 --> 00:52:47.390 align:middle line:84%
And it's in this
little thing here.

00:52:47.390 --> 00:52:49.367 align:middle line:84%
We have grabbed the
data and cleaned it up,

00:52:49.367 --> 00:52:50.950 align:middle line:84%
cleaned it up meaning
formatting-wise,

00:52:50.950 --> 00:52:52.790 align:middle line:90%
not content-wise.

00:52:52.790 --> 00:52:54.630 align:middle line:84%
And then we stuck it
in this data frame.

00:52:54.630 --> 00:52:57.630 align:middle line:84%
And we already have divided it
into train, test, and validation

00:52:57.630 --> 00:53:00.430 align:middle line:84%
for your benefit, so you
don't have to worry about it.

00:53:00.430 --> 00:53:04.550 align:middle line:84%
Turns out we have almost 49,000
songs in the training set,

00:53:04.550 --> 00:53:06.450 align:middle line:84%
16,000 songs in
the validation set,

00:53:06.450 --> 00:53:09.630 align:middle line:84%
and roughly 22,000
in the test set.

00:53:09.630 --> 00:53:10.590 align:middle line:90%
It's a lot of songs.

00:53:10.590 --> 00:53:11.090 align:middle line:90%
It's a lot.

00:53:11.090 --> 00:53:12.750 align:middle line:90%
It's a big data set.

00:53:12.750 --> 00:53:15.790 align:middle line:84%
So let's just look
at the first few.

00:53:15.790 --> 00:53:17.570 align:middle line:84%
So, "Oh, girl, I
can't get ready",

00:53:17.570 --> 00:53:21.990 align:middle line:84%
"we met on rainy evening",
"paralysis through analysis".

00:53:21.990 --> 00:53:24.840 align:middle line:84%
OK, that, I can relate to as a
data science person, but anyway.

00:53:24.840 --> 00:53:27.390 align:middle line:90%


00:53:27.390 --> 00:53:29.150 align:middle line:84%
By the way, these
things are very

00:53:29.150 --> 00:53:33.150 align:middle line:84%
useful for exploration of any
data frames that you might have.

00:53:33.150 --> 00:53:36.510 align:middle line:84%
It's a Colab feature,
just check it out.

00:53:36.510 --> 00:53:38.907 align:middle line:84%
So anyway, that's
the first few rows.

00:53:38.907 --> 00:53:40.240 align:middle line:90%
Let's look at the last few rows.

00:53:40.240 --> 00:53:48.810 align:middle line:90%


00:53:48.810 --> 00:53:51.050 align:middle line:90%
"You never listen to me" is pop.

00:53:51.050 --> 00:53:52.850 align:middle line:90%
"Bimmer, Benz" is hip hop.

00:53:52.850 --> 00:53:54.664 align:middle line:90%
Yeah, of course.

00:53:54.664 --> 00:54:00.930 align:middle line:84%
[INAUDIBLE] OK, now to go back
to the question of, OK, what

00:54:00.930 --> 00:54:02.230 align:middle line:90%
could be a good baseline model?

00:54:02.230 --> 00:54:04.490 align:middle line:84%
We need to understand the
proportion of these three

00:54:04.490 --> 00:54:06.050 align:middle line:90%
classes of songs.

00:54:06.050 --> 00:54:07.730 align:middle line:90%
So we will do a quick check.

00:54:07.730 --> 00:54:10.170 align:middle line:90%
It turns out rock is 55%.

00:54:10.170 --> 00:54:12.910 align:middle line:84%
So if you had to just guess
something just naively,

00:54:12.910 --> 00:54:14.702 align:middle line:84%
you would just guess
everything to be rock,

00:54:14.702 --> 00:54:16.770 align:middle line:84%
and you'd be right
55% of the time.

00:54:16.770 --> 00:54:20.090 align:middle line:84%
So now, by the way, the target
variable, which tells you

00:54:20.090 --> 00:54:21.978 align:middle line:84%
whether which of these
three genres it is,

00:54:21.978 --> 00:54:25.030 align:middle line:90%
is actually a dummy variable.

00:54:25.030 --> 00:54:28.130 align:middle line:84%
So need to, one,
hot-encode that.

00:54:28.130 --> 00:54:31.330 align:middle line:84%
So we'll just turn that
this way using the pandas

00:54:31.330 --> 00:54:32.710 align:middle line:90%
get_dummies function.

00:54:32.710 --> 00:54:34.727 align:middle line:84%
And when we do that,
this is y_train,

00:54:34.727 --> 00:54:36.310 align:middle line:84%
which contains the
dependent variable.

00:54:36.310 --> 00:54:38.820 align:middle line:84%
And you can see that
is one hot-encoded now.

00:54:38.820 --> 00:54:41.660 align:middle line:84%
010, 010, 001, and
so on and so forth.

00:54:41.660 --> 00:54:42.160 align:middle line:90%
That's it.

00:54:42.160 --> 00:54:44.620 align:middle line:84%
So I think the
first, I forget it,

00:54:44.620 --> 00:54:47.600 align:middle line:84%
rock, hip hop, pop or
whatever, it's in some order.

00:54:47.600 --> 00:54:48.680 align:middle line:90%
We'll get to that later.

00:54:48.680 --> 00:54:50.660 align:middle line:90%
So it's one hot-encoded as well.

00:54:50.660 --> 00:54:52.860 align:middle line:84%
So that is as far as
the data downloading

00:54:52.860 --> 00:54:53.920 align:middle line:90%
and setup is concerned.

00:54:53.920 --> 00:54:55.620 align:middle line:90%
Any questions?

00:54:55.620 --> 00:54:57.460 align:middle line:90%
Yeah.

00:54:57.460 --> 00:54:59.460 align:middle line:84%
AUDIENCE: This kind of
goes back to the transfer

00:54:59.460 --> 00:55:01.780 align:middle line:84%
learning concept,
but do you always

00:55:01.780 --> 00:55:04.300 align:middle line:84%
want to build your
corpus based off

00:55:04.300 --> 00:55:06.780 align:middle line:84%
of the vocabulary of
your training data?

00:55:06.780 --> 00:55:09.500 align:middle line:84%
Or could you have a
pre-compiled, like,

00:55:09.500 --> 00:55:13.020 align:middle line:84%
somebody already made like a
list of the 50,000 most common

00:55:13.020 --> 00:55:13.600 align:middle line:90%
words?

00:55:13.600 --> 00:55:15.140 align:middle line:84%
RAMA RAMAKRISHNAN: That's
a really good question.

00:55:15.140 --> 00:55:17.390 align:middle line:84%
Unfortunately, I'm going to
punt on it for the moment,

00:55:17.390 --> 00:55:21.520 align:middle line:84%
because with modern
large language models,

00:55:21.520 --> 00:55:25.620 align:middle line:84%
a number of these NLP tasks for
which you had to roll your own

00:55:25.620 --> 00:55:27.420 align:middle line:84%
and build your
own thing, can now

00:55:27.420 --> 00:55:31.300 align:middle line:84%
be very easily done using large
language models without even

00:55:31.300 --> 00:55:32.672 align:middle line:90%
any further training.

00:55:32.672 --> 00:55:34.380 align:middle line:84%
Now, the price you
pay for it is that you

00:55:34.380 --> 00:55:35.880 align:middle line:84%
have to use a large language
model, which means you

00:55:35.880 --> 00:55:38.260 align:middle line:84%
have to pay somebody an API
call and things like that.

00:55:38.260 --> 00:55:40.080 align:middle line:84%
And there are other
issues with it.

00:55:40.080 --> 00:55:45.338 align:middle line:84%
But we will talk a lot about
transfer learning for text when

00:55:45.338 --> 00:55:47.380 align:middle line:84%
we come to it a little
later in the NLP sequence.

00:55:47.380 --> 00:55:49.620 align:middle line:84%
So if I forget, please
bring it up again.

00:55:49.620 --> 00:55:53.320 align:middle line:90%


00:55:53.320 --> 00:55:55.840 align:middle line:90%
Yeah.

00:55:55.840 --> 00:55:59.000 align:middle line:84%
AUDIENCE: Quick clarification
on the encoding vector.

00:55:59.000 --> 00:56:00.860 align:middle line:84%
I noticed it was
floats, not ints.

00:56:00.860 --> 00:56:03.440 align:middle line:84%
If it gets incredibly
long, wouldn't that

00:56:03.440 --> 00:56:05.100 align:middle line:90%
eat into compute time?

00:56:05.100 --> 00:56:06.540 align:middle line:84%
Is there a reason
why it's floats?

00:56:06.540 --> 00:56:08.080 align:middle line:84%
RAMA RAMAKRISHNAN:
Yeah, so question

00:56:08.080 --> 00:56:12.080 align:middle line:84%
is that when I showed
you that tensor,

00:56:12.080 --> 00:56:15.140 align:middle line:84%
it's written as a continuous
number, a floating point number.

00:56:15.140 --> 00:56:16.820 align:middle line:84%
But we know these
are zeros and ones.

00:56:16.820 --> 00:56:19.400 align:middle line:84%
So why do we have to
waste compute capacity

00:56:19.400 --> 00:56:22.160 align:middle line:84%
by telling the computer that
it's all big continuous numbers

00:56:22.160 --> 00:56:23.435 align:middle line:90%
when it's just a 0 or 1?

00:56:23.435 --> 00:56:25.560 align:middle line:84%
There are ways to optimize
that, but these problems

00:56:25.560 --> 00:56:27.947 align:middle line:84%
are so small we just
don't worry about it.

00:56:27.947 --> 00:56:30.280 align:middle line:84%
But when we come to something
called parameter-efficient

00:56:30.280 --> 00:56:33.680 align:middle line:84%
fine-tuning, lecture
maybe 10-ish,

00:56:33.680 --> 00:56:36.180 align:middle line:84%
we will actually exploit that
particular fact to make things

00:56:36.180 --> 00:56:38.873 align:middle line:90%
faster.

00:56:38.873 --> 00:56:40.900 align:middle line:90%
So that's what we have.

00:56:40.900 --> 00:56:44.292 align:middle line:84%
So we will do the
Bag of Words model.

00:56:44.292 --> 00:56:46.250 align:middle line:84%
By the way, there's a
whole bunch of stuff here

00:56:46.250 --> 00:56:48.750 align:middle line:84%
that just repeats what I've
been telling you in the lecture.

00:56:48.750 --> 00:56:50.460 align:middle line:84%
So feel free to read
it again, but we

00:56:50.460 --> 00:56:52.420 align:middle line:90%
can ignore it for the moment.

00:56:52.420 --> 00:56:55.300 align:middle line:84%
And now, there is a new
thing we are doing here.

00:56:55.300 --> 00:56:57.760 align:middle line:84%
So we are basically
saying, look,

00:56:57.760 --> 00:57:01.860 align:middle line:84%
instead of taking every word
you see in these 49,000 songs

00:57:01.860 --> 00:57:05.360 align:middle line:84%
in the training corpus, it's
going to be too many words,

00:57:05.360 --> 00:57:09.140 align:middle line:84%
just pick the 5,000
most frequent words.

00:57:09.140 --> 00:57:13.020 align:middle line:84%
And that's what this
max_tokens stands for.

00:57:13.020 --> 00:57:16.380 align:middle line:84%
And so we tell it, all
right, do this thing,

00:57:16.380 --> 00:57:21.360 align:middle line:84%
max_tokens 5,000 and
still do multi-hot.

00:57:21.360 --> 00:57:23.620 align:middle line:84%
And we are not explicitly
saying the standardization

00:57:23.620 --> 00:57:25.260 align:middle line:84%
and all that stuff
because the defaults

00:57:25.260 --> 00:57:27.060 align:middle line:90%
are what we're going with.

00:57:27.060 --> 00:57:29.140 align:middle line:90%
Yeah.

00:57:29.140 --> 00:57:31.220 align:middle line:84%
AUDIENCE: This is for
making it more efficient?

00:57:31.220 --> 00:57:32.870 align:middle line:84%
This is, like, don't
waste your time

00:57:32.870 --> 00:57:35.630 align:middle line:90%
on these thousands of words.

00:57:35.630 --> 00:57:37.370 align:middle line:90%
Use them more.

00:57:37.370 --> 00:57:39.553 align:middle line:84%
Just focus on that to
make it more efficient.

00:57:39.553 --> 00:57:41.470 align:middle line:84%
RAMA RAMAKRISHNAN: It
makes it more efficient,

00:57:41.470 --> 00:57:43.750 align:middle line:84%
but there is a related
and important point, which

00:57:43.750 --> 00:57:46.310 align:middle line:84%
is that, fundamentally,
the number of tokens

00:57:46.310 --> 00:57:49.590 align:middle line:84%
you allow this layer
to have dictates

00:57:49.590 --> 00:57:51.190 align:middle line:90%
the size of your vocabulary.

00:57:51.190 --> 00:57:54.310 align:middle line:84%
And the size of your vocabulary
dictates the size of the vector

00:57:54.310 --> 00:57:55.167 align:middle line:90%
that you feed in.

00:57:55.167 --> 00:57:57.250 align:middle line:84%
So shorter vectors are
better than longer vectors.

00:57:57.250 --> 00:57:58.570 align:middle line:90%
That's the efficiency point.

00:57:58.570 --> 00:58:01.670 align:middle line:84%
The other point is that the
longer the input vector,

00:58:01.670 --> 00:58:03.330 align:middle line:84%
the more the number
of parameters

00:58:03.330 --> 00:58:06.110 align:middle line:84%
the network has to learn
because the first layer itself

00:58:06.110 --> 00:58:08.390 align:middle line:84%
is the size of the
input roughly times

00:58:08.390 --> 00:58:09.610 align:middle line:90%
the size of the hidden layer.

00:58:09.610 --> 00:58:11.010 align:middle line:84%
So this thing becomes
10 times as long,

00:58:11.010 --> 00:58:12.927 align:middle line:84%
you have 10 times as
many parameters to learn.

00:58:12.927 --> 00:58:15.390 align:middle line:84%
And given a finite
amount of data,

00:58:15.390 --> 00:58:17.110 align:middle line:84%
the more parameters
you have, the worse

00:58:17.110 --> 00:58:18.350 align:middle line:84%
it's going to do when
you actually start

00:58:18.350 --> 00:58:19.475 align:middle line:90%
using it in the real world.

00:58:19.475 --> 00:58:20.730 align:middle line:90%
It's going to overfit heavily.

00:58:20.730 --> 00:58:22.355 align:middle line:84%
That's why you need
to be very careful.

00:58:22.355 --> 00:58:25.590 align:middle line:90%


00:58:25.590 --> 00:58:27.433 align:middle line:90%
Yeah.

00:58:27.433 --> 00:58:29.350 align:middle line:84%
AUDIENCE: So have you
downloaded the data set?

00:58:29.350 --> 00:58:32.110 align:middle line:84%
Are you still using the
vocabulary of the 17 words,

00:58:32.110 --> 00:58:33.348 align:middle line:90%
or did you broaden it?

00:58:33.348 --> 00:58:35.390 align:middle line:84%
RAMA RAMAKRISHNAN: No,
no, that was just for fun.

00:58:35.390 --> 00:58:37.370 align:middle line:84%
I'm going to actually
build the vocabulary now.

00:58:37.370 --> 00:58:38.070 align:middle line:90%
It's coming.

00:58:38.070 --> 00:58:40.130 align:middle line:90%
Yeah, good question.

00:58:40.130 --> 00:58:41.770 align:middle line:90%
So, let's do that.

00:58:41.770 --> 00:58:44.850 align:middle line:90%
So first, I define this layer.

00:58:44.850 --> 00:58:46.510 align:middle line:90%
OK, I just defined it.

00:58:46.510 --> 00:58:49.130 align:middle line:84%
Now, we actually build the
vocabulary by essentially

00:58:49.130 --> 00:58:55.010 align:middle line:84%
telling it to adapt the layer
using essentially the full all

00:58:55.010 --> 00:58:57.890 align:middle line:84%
basically 49,000 songs
in the training data set.

00:58:57.890 --> 00:59:00.210 align:middle line:90%
That's a long list of songs.

00:59:00.210 --> 00:59:02.090 align:middle line:84%
As far as Keras is
concerned, you're just

00:59:02.090 --> 00:59:03.543 align:middle line:90%
looking for a list of strings.

00:59:03.543 --> 00:59:05.210 align:middle line:84%
So you just give it
the list of strings.

00:59:05.210 --> 00:59:07.270 align:middle line:84%
Instead of four, we're
giving it 49,000.

00:59:07.270 --> 00:59:11.450 align:middle line:84%
The same philosophy
applies, so we run it.

00:59:11.450 --> 00:59:13.890 align:middle line:84%
It's obviously going to take
a few seconds to do that

00:59:13.890 --> 00:59:16.730 align:middle line:90%
because it's 49,000 songs.

00:59:16.730 --> 00:59:18.410 align:middle line:90%
Oh, it's about five seconds.

00:59:18.410 --> 00:59:21.730 align:middle line:84%
All right, let's look
at the most common 20.

00:59:21.730 --> 00:59:24.730 align:middle line:84%
We get the vocabulary
from our layer.

00:59:24.730 --> 00:59:27.230 align:middle line:84%
See, once you adapt the layer
and it's built a vocabulary,

00:59:27.230 --> 00:59:28.730 align:middle line:84%
the layer has sort
of been populated

00:59:28.730 --> 00:59:31.060 align:middle line:84%
with all this information,
so you can query it.

00:59:31.060 --> 00:59:32.320 align:middle line:90%
So you can get the vocab.

00:59:32.320 --> 00:59:36.380 align:middle line:84%
Top 20 words-- UNK; "the", most
frequent word, no surprise;

00:59:36.380 --> 00:59:38.340 align:middle line:90%
"you", "I", blah, blah, blah.

00:59:38.340 --> 00:59:40.940 align:middle line:90%
Let's look at the last few.

00:59:40.940 --> 00:59:46.780 align:middle line:84%
"Dagger", "cheddar",
"terrified".

00:59:46.780 --> 00:59:51.340 align:middle line:84%
Moving on, and then
once we have done that,

00:59:51.340 --> 00:59:53.500 align:middle line:84%
now we actually can
vectorize all the data sets

00:59:53.500 --> 00:59:54.947 align:middle line:90%
we have using this.

00:59:54.947 --> 00:59:56.780 align:middle line:84%
And by vectorize, you
mean take every string

00:59:56.780 --> 00:59:59.660 align:middle line:84%
and create the multi-hot
encoded vector from it.

00:59:59.660 --> 01:00:00.435 align:middle line:90%
Yeah.

01:00:00.435 --> 01:00:03.060 align:middle line:84%
AUDIENCE: Are we doing S-T-I-E
because we're keeping stuff like

01:00:03.060 --> 01:00:04.103 align:middle line:90%
D-E-A, et cetera?

01:00:04.103 --> 01:00:06.520 align:middle line:84%
RAMA RAMAKRISHNAN: Yeah, we're
not strictly doing S-T-I-E,

01:00:06.520 --> 01:00:09.580 align:middle line:84%
or to put it differently,
typically S has

01:00:09.580 --> 01:00:12.740 align:middle line:84%
lowercase/uppercase, strip
punctuation, stemming,

01:00:12.740 --> 01:00:14.000 align:middle line:90%
stop word removal.

01:00:14.000 --> 01:00:17.400 align:middle line:84%
Here, the default in Keras
happens to not do stemming,

01:00:17.400 --> 01:00:18.560 align:middle line:90%
not do stop word removal.

01:00:18.560 --> 01:00:20.102 align:middle line:84%
So we're just going
with the default.

01:00:20.102 --> 01:00:21.918 align:middle line:90%
Thanks for the clarification.

01:00:21.918 --> 01:00:23.460 align:middle line:84%
And in fact, in
practice, what I find

01:00:23.460 --> 01:00:25.508 align:middle line:84%
these days is that don't
even bother to stem.

01:00:25.508 --> 01:00:27.300 align:middle line:84%
Don't even bother to
remove the stop words.

01:00:27.300 --> 01:00:28.740 align:middle line:90%
It's going to work well enough.

01:00:28.740 --> 01:00:33.360 align:middle line:90%


01:00:33.360 --> 01:00:35.420 align:middle line:84%
OK, so now, each
phrase is a vector.

01:00:35.420 --> 01:00:37.192 align:middle line:90%
How long is this vector?

01:00:37.192 --> 01:00:38.460 align:middle line:90%
Each song is now a vector.

01:00:38.460 --> 01:00:41.040 align:middle line:90%
How long is that vector?

01:00:41.040 --> 01:00:43.860 align:middle line:84%
5,000, correct, because that
is the size of the vocabulary.

01:00:43.860 --> 01:00:44.360 align:middle line:90%
Correct.

01:00:44.360 --> 01:00:47.200 align:middle line:90%


01:00:47.200 --> 01:00:49.500 align:middle line:84%
It's max tokens
long, which is 5,000.

01:00:49.500 --> 01:00:51.025 align:middle line:90%
So if you actually look at--

01:00:51.025 --> 01:00:52.400 align:middle line:84%
oh, wait, wait,
wait, wait, wait.

01:00:52.400 --> 01:00:53.780 align:middle line:90%
I haven't run this thing yet.

01:00:53.780 --> 01:00:57.960 align:middle line:90%


01:00:57.960 --> 01:00:59.260 align:middle line:90%
It's going through 49,000.

01:00:59.260 --> 01:01:01.420 align:middle line:84%
It's going through
another 23,000.

01:01:01.420 --> 01:01:02.080 align:middle line:90%
Fine.

01:01:02.080 --> 01:01:04.400 align:middle line:90%
So let's run it.

01:01:04.400 --> 01:01:07.520 align:middle line:84%
OK, now we can see x-train,
which is all the training data

01:01:07.520 --> 01:01:12.600 align:middle line:84%
you have, is a tensor, is
a table, with 48,991 rows.

01:01:12.600 --> 01:01:18.040 align:middle line:84%
And each row is a
5,000-long vector.

01:01:18.040 --> 01:01:19.660 align:middle line:90%
All right, good.

01:01:19.660 --> 01:01:21.880 align:middle line:84%
Now, we will try this
simple neural network

01:01:21.880 --> 01:01:24.800 align:middle line:90%
that we wrote up in class.

01:01:24.800 --> 01:01:29.170 align:middle line:84%
And now, at this
point, this code

01:01:29.170 --> 01:01:31.610 align:middle line:90%
should be sort of second-nature.

01:01:31.610 --> 01:01:33.130 align:middle line:90%
Isn't that cool?

01:01:33.130 --> 01:01:35.610 align:middle line:90%
It's so easy to write the thing.

01:01:35.610 --> 01:01:37.110 align:middle line:90%
The power of abstraction.

01:01:37.110 --> 01:01:39.250 align:middle line:90%
So we take Keras' write input.

01:01:39.250 --> 01:01:40.210 align:middle line:90%
As usual, input layer.

01:01:40.210 --> 01:01:42.710 align:middle line:84%
We tell it, what is the size
of each thing that's coming in?

01:01:42.710 --> 01:01:45.790 align:middle line:84%
Well, the size of each thing
is a 50 max_tokens long vector.

01:01:45.790 --> 01:01:47.730 align:middle line:84%
So we tell it the
shape is max_tokens,

01:01:47.730 --> 01:01:51.050 align:middle line:84%
and then we run it through a
dense layer with eight ReLUs.

01:01:51.050 --> 01:01:53.676 align:middle line:90%
OK, I'm hurrying.

01:01:53.676 --> 01:01:55.230 align:middle line:90%
So we get the outputs.

01:01:55.230 --> 01:01:57.817 align:middle line:84%
Then, we string the inputs
and the outputs into a model,

01:01:57.817 --> 01:01:59.150 align:middle line:90%
and then we summarize the model.

01:01:59.150 --> 01:01:59.650 align:middle line:90%
That's it.

01:01:59.650 --> 01:02:00.970 align:middle line:90%
So we go here.

01:02:00.970 --> 01:02:02.790 align:middle line:90%
And this has 40,000 parameters.

01:02:02.790 --> 01:02:05.990 align:middle line:84%
And you can see here, when
you go from the input,

01:02:05.990 --> 01:02:10.650 align:middle line:84%
the 5,000 times 8, that gives
you 40,000 plus the 8 neurons

01:02:10.650 --> 01:02:11.630 align:middle line:90%
have a bias coming in.

01:02:11.630 --> 01:02:12.505 align:middle line:90%
That's another eight.

01:02:12.505 --> 01:02:15.010 align:middle line:90%
So you get 40,008.

01:02:15.010 --> 01:02:16.430 align:middle line:90%
And we compile it as usual.

01:02:16.430 --> 01:02:17.910 align:middle line:90%
We use Adam as usual.

01:02:17.910 --> 01:02:22.530 align:middle line:84%
And because now the output y
variable, the y-train variable,

01:02:22.530 --> 01:02:27.790 align:middle line:84%
is now itself actually
one-hot encoded, 010,

01:02:27.790 --> 01:02:30.710 align:middle line:84%
001 depending on pop, rock,
and so on and so forth.

01:02:30.710 --> 01:02:33.290 align:middle line:84%
We don't use sparse
categorical cross-entropy.

01:02:33.290 --> 01:02:36.710 align:middle line:84%
We just use plain, old
categorical cross-entropy here.

01:02:36.710 --> 01:02:40.010 align:middle line:84%
And this was explained
in lecture last week,

01:02:40.010 --> 01:02:43.050 align:middle line:84%
so you can revisit it
if it's not familiar.

01:02:43.050 --> 01:02:44.330 align:middle line:90%
We again report accuracy.

01:02:44.330 --> 01:02:45.830 align:middle line:90%
So let's compile it.

01:02:45.830 --> 01:02:47.590 align:middle line:90%
And we get a model.

01:02:47.590 --> 01:02:50.590 align:middle line:84%
So we'll just run it for 10
epochs with a batch size of 32.

01:02:50.590 --> 01:02:53.065 align:middle line:84%
And because we have validation
data already supplied to us,

01:02:53.065 --> 01:02:55.190 align:middle line:84%
we don't have to tell Keras,
take the training data

01:02:55.190 --> 01:02:57.690 align:middle line:84%
and keep 20% of it
aside for validation.

01:02:57.690 --> 01:02:59.690 align:middle line:84%
We can literally tell it
what validation to use.

01:02:59.690 --> 01:03:00.898 align:middle line:90%
That's what we're doing here.

01:03:00.898 --> 01:03:03.910 align:middle line:90%


01:03:03.910 --> 01:03:05.500 align:middle line:90%
All right, so it's running.

01:03:05.500 --> 01:03:09.092 align:middle line:90%


01:03:09.092 --> 01:03:09.800 align:middle line:90%
It's pretty fast.

01:03:09.800 --> 01:03:16.310 align:middle line:90%


01:03:16.310 --> 01:03:18.150 align:middle line:90%
Any questions so far?

01:03:18.150 --> 01:03:20.070 align:middle line:90%
Yes.

01:03:20.070 --> 01:03:23.380 align:middle line:90%
Use the microphone, please.

01:03:23.380 --> 01:03:25.380 align:middle line:84%
AUDIENCE: How do we decide
the max total length?

01:03:25.380 --> 01:03:27.120 align:middle line:84%
We defined the
number of 5,000 here,

01:03:27.120 --> 01:03:29.120 align:middle line:84%
but we do not know how
many words would be there

01:03:29.120 --> 01:03:29.940 align:middle line:90%
in the entire text.

01:03:29.940 --> 01:03:31.020 align:middle line:84%
RAMA RAMAKRISHNAN: Yeah,
so it's a good question.

01:03:31.020 --> 01:03:33.160 align:middle line:84%
How do you decide on
this maximum vocabulary?

01:03:33.160 --> 01:03:34.660 align:middle line:84%
What do you typically
do in practice

01:03:34.660 --> 01:03:37.740 align:middle line:84%
is that you actually do
it without the max tokens,

01:03:37.740 --> 01:03:39.580 align:middle line:84%
and then you see how
long the vocabulary is,

01:03:39.580 --> 01:03:41.247 align:middle line:84%
and then you can
actually get statistics

01:03:41.247 --> 01:03:44.000 align:middle line:84%
on how frequently the very
infrequent words actually

01:03:44.000 --> 01:03:44.540 align:middle line:90%
show up.

01:03:44.540 --> 01:03:48.300 align:middle line:84%
And then you'll typically see a
dramatic fall off at some point.

01:03:48.300 --> 01:03:49.965 align:middle line:84%
And you pick that
fall-off point,

01:03:49.965 --> 01:03:51.590 align:middle line:84%
and then set that to
be the max tokens.

01:03:51.590 --> 01:03:57.160 align:middle line:90%


01:03:57.160 --> 01:03:57.700 align:middle line:90%
Perfect.

01:03:57.700 --> 01:03:58.960 align:middle line:90%
Let's test it.

01:03:58.960 --> 01:04:03.560 align:middle line:84%
Accuracy is pretty good,
87% on the training and 73

01:04:03.560 --> 01:04:05.000 align:middle line:90%
on the validation.

01:04:05.000 --> 01:04:07.440 align:middle line:90%
We'll do it on the test set.

01:04:07.440 --> 01:04:09.240 align:middle line:90%
All right, 72%.

01:04:09.240 --> 01:04:12.160 align:middle line:84%
So we saw earlier the largest
class of the three-way

01:04:12.160 --> 01:04:14.020 align:middle line:90%
is rock, with around 50%.

01:04:14.020 --> 01:04:16.485 align:middle line:84%
So the naive model is
going to get 50% accuracy,

01:04:16.485 --> 01:04:18.360 align:middle line:84%
and this little neural
network model gets you

01:04:18.360 --> 01:04:20.040 align:middle line:90%
72%, which is pretty nice.

01:04:20.040 --> 01:04:22.860 align:middle line:84%
So now, let's actually
kick it up a notch

01:04:22.860 --> 01:04:25.220 align:middle line:84%
and make it slightly
more capable.

01:04:25.220 --> 01:04:29.140 align:middle line:84%
So the key thing here is
that, as has been observed

01:04:29.140 --> 01:04:31.720 align:middle line:84%
in class already, when you
go with a Bag of Words model,

01:04:31.720 --> 01:04:33.980 align:middle line:90%
we lose all notion of order.

01:04:33.980 --> 01:04:36.580 align:middle line:84%
The word order clearly matters,
and we're kind of ignoring it.

01:04:36.580 --> 01:04:40.100 align:middle line:84%
So what we do to
get around it is--

01:04:40.100 --> 01:04:43.140 align:middle line:84%
this is actually really
interesting sentence here.

01:04:43.140 --> 01:04:44.660 align:middle line:84%
Let's say this is
a movie review.

01:04:44.660 --> 01:04:48.100 align:middle line:84%
"Kate Winslet's performance
as a detective trying to solve

01:04:48.100 --> 01:04:50.940 align:middle line:84%
a terrible crime in a small
Pennsylvania town is anything

01:04:50.940 --> 01:04:52.820 align:middle line:90%
but disappointing".

01:04:52.820 --> 01:04:55.153 align:middle line:84%
Tricky, tricky thing, because
if you look at the words

01:04:55.153 --> 01:04:57.320 align:middle line:84%
separately, the word
"terrible" and "disappointing",

01:04:57.320 --> 01:04:59.940 align:middle line:84%
you're going to be,
like, negative sentiment.

01:04:59.940 --> 01:05:04.900 align:middle line:84%
But then if you actually know
that the word terrible refers

01:05:04.900 --> 01:05:08.060 align:middle line:84%
to the crime, not to the movie
or anything but disappointing

01:05:08.060 --> 01:05:10.560 align:middle line:84%
changes the meaning of the word
disappointing, you will see,

01:05:10.560 --> 01:05:12.300 align:middle line:84%
obviously, it's a
positive review.

01:05:12.300 --> 01:05:15.540 align:middle line:84%
So clearly, the
words around the word

01:05:15.540 --> 01:05:19.100 align:middle line:84%
provide valuable clues as to
how to interpret that word.

01:05:19.100 --> 01:05:23.830 align:middle line:84%
And so what we do is, how can we
make our little model a bit more

01:05:23.830 --> 01:05:27.130 align:middle line:84%
capable of recognizing the
context around every word?

01:05:27.130 --> 01:05:29.590 align:middle line:84%
And the way we do it is
something called bigrams.

01:05:29.590 --> 01:05:33.110 align:middle line:84%
And for bigrams,
what we basically

01:05:33.110 --> 01:05:37.470 align:middle line:84%
do is instead of just taking
each word, we take each word

01:05:37.470 --> 01:05:42.310 align:middle line:84%
and we further take every
pair of adjacent words,

01:05:42.310 --> 01:05:43.970 align:middle line:90%
and those become our tokens.

01:05:43.970 --> 01:05:48.250 align:middle line:84%
And because we take two adjacent
words, they are called bigrams.

01:05:48.250 --> 01:05:50.650 align:middle line:84%
You can take three
adjacent words, trigrams.

01:05:50.650 --> 01:05:52.610 align:middle line:90%
You get the idea, n-grams.

01:05:52.610 --> 01:05:54.390 align:middle line:90%
So that's the idea of bigrams.

01:05:54.390 --> 01:05:59.870 align:middle line:84%
And so, for example, if had
"the cat sat on the mat",

01:05:59.870 --> 01:06:03.330 align:middle line:84%
you will have "the", "the
cat", "cat", "cat sat", "sat".

01:06:03.330 --> 01:06:05.070 align:middle line:90%
You get the idea.

01:06:05.070 --> 01:06:06.090 align:middle line:90%
That's what we have.

01:06:06.090 --> 01:06:07.562 align:middle line:90%
So let's do a little example.

01:06:07.562 --> 01:06:08.770 align:middle line:90%
And Keras makes it very easy.

01:06:08.770 --> 01:06:13.270 align:middle line:84%
You literally tell it
n-grams equals 2, bigrams.

01:06:13.270 --> 01:06:15.470 align:middle line:84%
And now, from this,
you immediately

01:06:15.470 --> 01:06:19.050 align:middle line:84%
should know that n-grams
equals 1 is the default.

01:06:19.050 --> 01:06:21.236 align:middle line:84%
That's why we didn't
have to specify it.

01:06:21.236 --> 01:06:26.170 align:middle line:84%
So, you run it, and then
you do "cat sat on the mat"

01:06:26.170 --> 01:06:27.457 align:middle line:90%
as your training corpus.

01:06:27.457 --> 01:06:28.790 align:middle line:90%
And then you get the vocabulary.

01:06:28.790 --> 01:06:32.170 align:middle line:84%
And you can see here, it has
created all these nice bigrams

01:06:32.170 --> 01:06:33.770 align:middle line:90%
for you.

01:06:33.770 --> 01:06:34.710 align:middle line:90%
And so that's it.

01:06:34.710 --> 01:06:37.210 align:middle line:84%
All right, now, what we do
is we go back to the songs,

01:06:37.210 --> 01:06:40.550 align:middle line:84%
and we actually tell Keras
to not just take each word,

01:06:40.550 --> 01:06:42.350 align:middle line:84%
but take all the
bigrams as well.

01:06:42.350 --> 01:06:45.010 align:middle line:84%
And hopefully, it'll do a
better job of figuring out

01:06:45.010 --> 01:06:46.230 align:middle line:90%
what the sentiment is.

01:06:46.230 --> 01:06:51.390 align:middle line:84%
And now, because when you say,
OK, take the top 5,000 words,

01:06:51.390 --> 01:06:54.450 align:middle line:84%
that's great for single
unigrams, as they are called,

01:06:54.450 --> 01:06:57.250 align:middle line:84%
but when you have bigrams,
you have 5,000 possibilities

01:06:57.250 --> 01:06:59.890 align:middle line:84%
of the first word, maybe
5,000 for the second word.

01:06:59.890 --> 01:07:01.672 align:middle line:84%
That's a lot of
possibilities, 25 million.

01:07:01.672 --> 01:07:04.130 align:middle line:84%
Now, most of the 25 million
possibilities are going to show

01:07:04.130 --> 01:07:06.547 align:middle line:84%
up in the data, so you don't
need to actually make it much

01:07:06.547 --> 01:07:08.970 align:middle line:84%
larger, but you should make
the vocabulary a bit more than

01:07:08.970 --> 01:07:09.910 align:middle line:90%
5,000.

01:07:09.910 --> 01:07:12.800 align:middle line:84%
So here, we go
with, say, 20,000.

01:07:12.800 --> 01:07:14.550 align:middle line:84%
Otherwise, it's the
same, still multi-hot.

01:07:14.550 --> 01:07:15.850 align:middle line:90%
So let's run it.

01:07:15.850 --> 01:07:19.133 align:middle line:84%
And now, we will run this now
that the layer has been set up

01:07:19.133 --> 01:07:20.550 align:middle line:84%
with all the right
settings, we'll

01:07:20.550 --> 01:07:23.530 align:middle line:84%
ask it to create the
vocabulary, again,

01:07:23.530 --> 01:07:25.310 align:middle line:84%
by doing exactly
what we did before.

01:07:25.310 --> 01:07:26.450 align:middle line:90%
Create the vocabulary.

01:07:26.450 --> 01:07:29.467 align:middle line:90%


01:07:29.467 --> 01:07:30.300 align:middle line:90%
[INAUDIBLE] seconds.

01:07:30.300 --> 01:07:41.870 align:middle line:90%


01:07:41.870 --> 01:07:45.510 align:middle line:84%
Bigrams, trigrams, all of
them will get much more

01:07:45.510 --> 01:07:46.410 align:middle line:90%
compute-intensive.

01:07:46.410 --> 01:07:47.710 align:middle line:90%
That's why you're seeing this.

01:07:47.710 --> 01:07:50.670 align:middle line:84%
So, all right, let's look
at the first 10 words.

01:07:50.670 --> 01:07:53.010 align:middle line:84%
The first 10 words are
all just single words,

01:07:53.010 --> 01:07:54.790 align:middle line:84%
and that's not surprising
because the single words are

01:07:54.790 --> 01:07:56.040 align:middle line:90%
going to be the more frequent.

01:07:56.040 --> 01:07:59.182 align:middle line:90%


01:07:59.182 --> 01:08:02.470 align:middle line:84%
And then let's look
at the last few.

01:08:02.470 --> 01:08:05.510 align:middle line:84%
"Your mom", "your God",
"you shot", "you hell".

01:08:05.510 --> 01:08:09.870 align:middle line:90%


01:08:09.870 --> 01:08:15.760 align:middle line:84%
All right, let's just
index all the data we have,

01:08:15.760 --> 01:08:18.300 align:middle line:84%
the training validation test
sets, using this vocabulary.

01:08:18.300 --> 01:08:23.180 align:middle line:90%


01:08:23.180 --> 01:08:23.680 align:middle line:90%
Perfect.

01:08:23.680 --> 01:08:25.055 align:middle line:84%
Now, we come to
our second model,

01:08:25.055 --> 01:08:28.300 align:middle line:84%
where we say the incoming
shape is now 20,000 long,

01:08:28.300 --> 01:08:31.160 align:middle line:84%
because we increased max
tokens from 5,000 to 20,000.

01:08:31.160 --> 01:08:33.439 align:middle line:84%
So each thing is a
20,000-long vector.

01:08:33.439 --> 01:08:34.760 align:middle line:90%
Otherwise, it's the same.

01:08:34.760 --> 01:08:37.160 align:middle line:84%
And now, we will use
this thing called dropout

01:08:37.160 --> 01:08:40.760 align:middle line:84%
for the first time, which is a
regularization thing that I have

01:08:40.760 --> 01:08:43.520 align:middle line:84%
referred to earlier that
I never really described,

01:08:43.520 --> 01:08:45.790 align:middle line:84%
and I will describe
today if we have time.

01:08:45.790 --> 01:08:47.540 align:middle line:84%
But I'll first run
through the whole demo.

01:08:47.540 --> 01:08:51.359 align:middle line:84%
So you can just think of
dropout as just another layer

01:08:51.359 --> 01:08:52.340 align:middle line:90%
you can insert.

01:08:52.340 --> 01:08:54.840 align:middle line:84%
And it's essentially a great
way to prevent overfitting.

01:08:54.840 --> 01:08:57.279 align:middle line:90%
So I just routinely will use it.

01:08:57.279 --> 01:08:58.620 align:middle line:90%
And I'll talk more about it.

01:08:58.620 --> 01:09:01.180 align:middle line:84%
So for now, we have this
dropout layer in the middle.

01:09:01.180 --> 01:09:03.157 align:middle line:84%
It receives the input
from the dense layer

01:09:03.157 --> 01:09:04.740 align:middle line:84%
and then sends it
to the output layer.

01:09:04.740 --> 01:09:05.990 align:middle line:90%
The output layer is unchanged.

01:09:05.990 --> 01:09:09.279 align:middle line:84%
It's a three-way softmax,
same model as before.

01:09:09.279 --> 01:09:11.840 align:middle line:84%
And now, all right, I'll
come back to dropout.

01:09:11.840 --> 01:09:13.680 align:middle line:84%
So we'll compile it
the same way as before.

01:09:13.680 --> 01:09:17.260 align:middle line:84%
And then I will just
fit it for three epochs.

01:09:17.260 --> 01:09:19.100 align:middle line:84%
If you're interested
after class later on,

01:09:19.100 --> 01:09:20.725 align:middle line:84%
you can actually try
it for more epochs

01:09:20.725 --> 01:09:22.024 align:middle line:90%
and see if it does better.

01:09:22.024 --> 01:09:23.399 align:middle line:84%
For now, in the
interest of time,

01:09:23.399 --> 01:09:24.524 align:middle line:90%
we'll just do it for three.

01:09:24.524 --> 01:09:36.219 align:middle line:90%


01:09:36.219 --> 01:09:43.314 align:middle line:84%
I think 72% was the single-word
unigram thing we had.

01:09:43.314 --> 01:09:44.939 align:middle line:84%
AUDIENCE: If you're
rerunning this code

01:09:44.939 --> 01:09:46.740 align:middle line:84%
with the same number
of epochs, do you ever

01:09:46.740 --> 01:09:49.357 align:middle line:90%
expect the accuracy to change?

01:09:49.357 --> 01:09:50.899 align:middle line:84%
RAMA RAMAKRISHNAN:
If you were to run

01:09:50.899 --> 01:09:53.020 align:middle line:84%
this code on your
machine, you would

01:09:53.020 --> 01:09:54.472 align:middle line:84%
expect it to be
roughly the same,

01:09:54.472 --> 01:09:56.180 align:middle line:84%
but there are some
minute differences due

01:09:56.180 --> 01:09:58.193 align:middle line:84%
to hardware and device
drivers and things like.

01:09:58.193 --> 01:10:00.360 align:middle line:84%
AUDIENCE: If you reran it
on your own machine twice,

01:10:00.360 --> 01:10:02.900 align:middle line:84%
would you ever expect
the accuracy to change?

01:10:02.900 --> 01:10:05.580 align:middle line:84%
RAMA RAMAKRISHNAN: That's
actually a very tricky question,

01:10:05.580 --> 01:10:07.340 align:middle line:84%
because it depends
on what else I have

01:10:07.340 --> 01:10:09.620 align:middle line:90%
been doing in that notebook.

01:10:09.620 --> 01:10:12.430 align:middle line:84%
If I start fresh and
do nothing but that,

01:10:12.430 --> 01:10:14.287 align:middle line:84%
typically I get the
same numbers, typically.

01:10:14.287 --> 01:10:16.370 align:middle line:84%
But for some reason, I
don't get it exactly right.

01:10:16.370 --> 01:10:19.270 align:middle line:90%


01:10:19.270 --> 01:10:21.090 align:middle line:90%
OK, so we come to this.

01:10:21.090 --> 01:10:22.860 align:middle line:90%
Let's evaluate our little model.

01:10:22.860 --> 01:10:25.830 align:middle line:90%


01:10:25.830 --> 01:10:27.330 align:middle line:90%
OK, 75%.

01:10:27.330 --> 01:10:30.270 align:middle line:84%
So it went from 72 to 75, which
is actually a meaningful jump,

01:10:30.270 --> 01:10:31.790 align:middle line:90%
just by using bigrams.

01:10:31.790 --> 01:10:33.950 align:middle line:84%
And I ran it only
for three epochs.

01:10:33.950 --> 01:10:36.870 align:middle line:84%
If you run it for 10, maybe
it's going to do even better.

01:10:36.870 --> 01:10:39.210 align:middle line:84%
So that is the
beauty of this thing.

01:10:39.210 --> 01:10:41.950 align:middle line:84%
Now, let's just actually
do a little demo.

01:10:41.950 --> 01:10:44.190 align:middle line:84%
We'll try to
predict some lyrics.

01:10:44.190 --> 01:10:46.730 align:middle line:84%
I'll try "another
one bites the dust".

01:10:46.730 --> 01:10:49.262 align:middle line:90%


01:10:49.262 --> 01:10:49.970 align:middle line:90%
It's a rock song.

01:10:49.970 --> 01:10:52.110 align:middle line:90%
I think that's correct, yes?

01:10:52.110 --> 01:10:56.110 align:middle line:90%
OK, folks, your turn now.

01:10:56.110 --> 01:10:57.770 align:middle line:84%
Somebody, tell me
your favorite song.

01:10:57.770 --> 01:11:00.550 align:middle line:90%


01:11:00.550 --> 01:11:02.598 align:middle line:84%
AUDIENCE: "Dancing
Queen" from ABBA.

01:11:02.598 --> 01:11:03.890 align:middle line:90%
RAMA RAMAKRISHNAN: I love ABBA.

01:11:03.890 --> 01:11:04.690 align:middle line:90%
That's awesome.

01:11:04.690 --> 01:11:11.420 align:middle line:84%
All right, OK,
dancing queen lyrics.

01:11:11.420 --> 01:11:17.048 align:middle line:90%


01:11:17.048 --> 01:11:18.590 align:middle line:84%
Verse one, intro--
I don't like that.

01:11:18.590 --> 01:11:21.390 align:middle line:84%
Let's just go to something
without all this metadata.

01:11:21.390 --> 01:11:27.353 align:middle line:90%


01:11:27.353 --> 01:11:28.770 align:middle line:84%
I'll just take the
first page, OK?

01:11:28.770 --> 01:11:38.050 align:middle line:90%


01:11:38.050 --> 01:11:40.490 align:middle line:90%
Here.

01:11:40.490 --> 01:11:42.450 align:middle line:90%
Are we good?

01:11:42.450 --> 01:11:46.210 align:middle line:84%
All right, don't
let me down, model.

01:11:46.210 --> 01:11:46.820 align:middle line:90%
Let's predict.

01:11:46.820 --> 01:11:50.190 align:middle line:90%


01:11:50.190 --> 01:11:50.690 align:middle line:90%
Pop.

01:11:50.690 --> 01:11:52.110 align:middle line:90%
Just about.

01:11:52.110 --> 01:11:52.610 align:middle line:90%
Yay!

01:11:52.610 --> 01:11:55.610 align:middle line:90%
[APPLAUSE]

01:11:55.610 --> 01:11:56.570 align:middle line:90%
All right.

01:11:56.570 --> 01:11:59.488 align:middle line:84%
So, yeah, that's
basically the model.

01:11:59.488 --> 01:12:00.530 align:middle line:90%
But we have five minutes.

01:12:00.530 --> 01:12:02.655 align:middle line:84%
I want to get back to--
you can play around and put

01:12:02.655 --> 01:12:03.890 align:middle line:90%
your own lyrics in.

01:12:03.890 --> 01:12:06.210 align:middle line:84%
Typically, what happens
is that the last two years

01:12:06.210 --> 01:12:08.590 align:middle line:84%
that I've been doing
this particular lecture,

01:12:08.590 --> 01:12:10.700 align:middle line:84%
I've noticed that
the songs are always

01:12:10.700 --> 01:12:12.840 align:middle line:90%
rock songs for some reason.

01:12:12.840 --> 01:12:15.340 align:middle line:84%
It's the first time getting
a pop song from a group

01:12:15.340 --> 01:12:18.500 align:middle line:84%
that I actually
like, so thank you.

01:12:18.500 --> 01:12:21.980 align:middle line:84%
All right, let's
go back to dropout.

01:12:21.980 --> 01:12:25.020 align:middle line:84%
So the idea here in
dropout is that you

01:12:25.020 --> 01:12:26.840 align:middle line:84%
have all these--
the input comes in,

01:12:26.840 --> 01:12:30.480 align:middle line:84%
it goes through a hidden
layer, and so on and so forth.

01:12:30.480 --> 01:12:32.785 align:middle line:90%
So dropout is a layer.

01:12:32.785 --> 01:12:35.160 align:middle line:84%
And you put this layer just
like you use any other layer.

01:12:35.160 --> 01:12:38.140 align:middle line:84%
And what dropout does is that
it takes all the things that

01:12:38.140 --> 01:12:40.300 align:middle line:84%
are coming into it
from the previous layer

01:12:40.300 --> 01:12:46.100 align:middle line:84%
and randomly decides to replace
that number with a zero.

01:12:46.100 --> 01:12:46.720 align:middle line:90%
That's it.

01:12:46.720 --> 01:12:49.580 align:middle line:84%
It drops that number and
replaces it with zero.

01:12:49.580 --> 01:12:51.160 align:middle line:90%
But it does it randomly.

01:12:51.160 --> 01:12:52.640 align:middle line:90%
It basically will toss a coin.

01:12:52.640 --> 01:12:54.360 align:middle line:84%
If the coin comes
up heads, zero.

01:12:54.360 --> 01:12:57.460 align:middle line:84%
If it comes up tails, let
it through, pass it through.

01:12:57.460 --> 01:13:00.300 align:middle line:84%
And the reason why
this is very effective

01:13:00.300 --> 01:13:04.380 align:middle line:84%
is because you can
imagine all the neurons

01:13:04.380 --> 01:13:05.940 align:middle line:90%
in a particular layer.

01:13:05.940 --> 01:13:09.040 align:middle line:84%
When they overfit to
a particular data set,

01:13:09.040 --> 01:13:11.360 align:middle line:84%
the overfitting happens
because the neurons essentially

01:13:11.360 --> 01:13:13.880 align:middle line:90%
collude with each other.

01:13:13.880 --> 01:13:16.680 align:middle line:84%
They collude with each other
to actually overfit and predict

01:13:16.680 --> 01:13:19.040 align:middle line:90%
things in a very accurate way.

01:13:19.040 --> 01:13:23.320 align:middle line:84%
So you want to break any sort of
collusion between the neurons.

01:13:23.320 --> 01:13:27.140 align:middle line:84%
I'm obviously using a game
theoretic way of describing it.

01:13:27.140 --> 01:13:30.440 align:middle line:84%
But the idea is that any
kind of spurious correlations

01:13:30.440 --> 01:13:33.160 align:middle line:84%
in your data, neurons
can pick it up

01:13:33.160 --> 01:13:36.080 align:middle line:90%
by being correlated themselves.

01:13:36.080 --> 01:13:39.200 align:middle line:84%
And so the way you avoid
the spurious correlation is

01:13:39.200 --> 01:13:41.203 align:middle line:90%
by dropping neurons randomly.

01:13:41.203 --> 01:13:42.620 align:middle line:84%
You just kill the
neuron randomly,

01:13:42.620 --> 01:13:45.400 align:middle line:84%
which means that no neuron can
depend on another neuron being

01:13:45.400 --> 01:13:47.720 align:middle line:90%
available.

01:13:47.720 --> 01:13:50.120 align:middle line:84%
I know, it's a bit
grim, but that's

01:13:50.120 --> 01:13:51.720 align:middle line:90%
the basic idea of dropout.

01:13:51.720 --> 01:13:54.840 align:middle line:84%
And apparently, the story
goes that the team that

01:13:54.840 --> 01:13:58.100 align:middle line:84%
invented it, Jeff Hinton, who
won the Turing for this stuff,

01:13:58.100 --> 01:14:00.745 align:middle line:84%
not for the dropout,
just for deep learning,

01:14:00.745 --> 01:14:03.120 align:middle line:84%
I don't know if it's true,
but he said that apparently he

01:14:03.120 --> 01:14:05.120 align:middle line:84%
got the idea when
he went to a bank

01:14:05.120 --> 01:14:08.890 align:middle line:84%
and realized that, very often,
the folks who are working

01:14:08.890 --> 01:14:13.130 align:middle line:84%
in that bank branch that he
used to go to kept changing.

01:14:13.130 --> 01:14:14.380 align:middle line:90%
They were never the same.

01:14:14.380 --> 01:14:16.630 align:middle line:84%
The people would be transferring
in, transferring out.

01:14:16.630 --> 01:14:18.430 align:middle line:84%
And he was like, why can't they
just leave these people alone?

01:14:18.430 --> 01:14:19.513 align:middle line:90%
Why does it keep changing?

01:14:19.513 --> 01:14:22.090 align:middle line:84%
And then he got the insight
that maybe a lot of fraud

01:14:22.090 --> 01:14:25.210 align:middle line:84%
happens because the person
working in the branch

01:14:25.210 --> 01:14:27.050 align:middle line:90%
colludes with the customer.

01:14:27.050 --> 01:14:28.950 align:middle line:84%
But by changing the
staff constantly,

01:14:28.950 --> 01:14:31.430 align:middle line:84%
you break the risk
of fraud happening.

01:14:31.430 --> 01:14:34.450 align:middle line:84%
And that apparently was
the genesis for this idea.

01:14:34.450 --> 01:14:36.310 align:middle line:84%
True, apocryphal,
I have no idea,

01:14:36.310 --> 01:14:38.050 align:middle line:90%
but it's sort of a fun story.

01:14:38.050 --> 01:14:40.530 align:middle line:90%
Yes.

01:14:40.530 --> 01:14:42.050 align:middle line:84%
AUDIENCE: Instead
of random, if we

01:14:42.050 --> 01:14:44.450 align:middle line:84%
go to the way historical
models are built,

01:14:44.450 --> 01:14:46.010 align:middle line:84%
the concepts of
multicollinearity

01:14:46.010 --> 01:14:50.150 align:middle line:84%
and all of that, would that make
it sharper as compared to this?

01:14:50.150 --> 01:14:53.210 align:middle line:84%
RAMA RAMAKRISHNAN: The
problem is that these networks

01:14:53.210 --> 01:14:54.690 align:middle line:90%
are massive.

01:14:54.690 --> 01:14:57.370 align:middle line:84%
And for you to take
each layer and look

01:14:57.370 --> 01:14:59.150 align:middle line:84%
at its correlation
with some other layer

01:14:59.150 --> 01:15:01.570 align:middle line:84%
and so on and so forth,
first of all, investigating

01:15:01.570 --> 01:15:04.170 align:middle line:90%
multicollinearity is a problem.

01:15:04.170 --> 01:15:07.030 align:middle line:84%
The second thing is, OK,
what do you do then next?

01:15:07.030 --> 01:15:09.910 align:middle line:84%
In linear regression, you can do
things like principal components

01:15:09.910 --> 01:15:11.050 align:middle line:90%
analysis to get around it.

01:15:11.050 --> 01:15:12.490 align:middle line:90%
Here, everything is non-linear.

01:15:12.490 --> 01:15:14.290 align:middle line:84%
There is no easy way
to solve the problem.

01:15:14.290 --> 01:15:16.123 align:middle line:84%
So we are like, we'll
just solve the problem

01:15:16.123 --> 01:15:17.550 align:middle line:90%
in one shot using dropout.

01:15:17.550 --> 01:15:18.790 align:middle line:90%
That's the idea.

01:15:18.790 --> 01:15:24.070 align:middle line:84%
All right, so I had some
material on something

01:15:24.070 --> 01:15:29.092 align:middle line:84%
called byte-pair encoding, which
I will do when we get to LLMs.

01:15:29.092 --> 01:15:30.550 align:middle line:84%
And I stuck it in
the end because I

01:15:30.550 --> 01:15:32.508 align:middle line:84%
knew that we probably
wouldn't have enough time

01:15:32.508 --> 01:15:33.530 align:middle line:90%
to cover this anyway.

01:15:33.530 --> 01:15:36.690 align:middle line:84%
And that is a very clever
tokenization scheme used by,

01:15:36.690 --> 01:15:38.950 align:middle line:84%
for example, the
GPT family, that

01:15:38.950 --> 01:15:41.010 align:middle line:84%
allows them to do
beautiful punctuation,

01:15:41.010 --> 01:15:44.150 align:middle line:84%
keep the case intact, and then
use words that you just made up

01:15:44.150 --> 01:15:45.050 align:middle line:90%
and things like that.

01:15:45.050 --> 01:15:47.130 align:middle line:90%
So we have one more minute.

01:15:47.130 --> 01:15:50.662 align:middle line:84%
I'm happy to answer any
questions you might have.

01:15:50.662 --> 01:15:52.870 align:middle line:84%
AUDIENCE: So initially when
we are picking the hidden

01:15:52.870 --> 01:15:55.910 align:middle line:84%
layer, the number of
neurons, and [INAUDIBLE]

01:15:55.910 --> 01:15:59.293 align:middle line:84%
to so far all the materials
this has been given to us.

01:15:59.293 --> 01:16:00.710 align:middle line:84%
But initially, how
do you pick it?

01:16:00.710 --> 01:16:03.597 align:middle line:84%
Is it more of a trial
and error type of thing?

01:16:03.597 --> 01:16:05.680 align:middle line:84%
RAMA RAMAKRISHNAN: It tends
to be trial and error.

01:16:05.680 --> 01:16:09.820 align:middle line:84%
That's, in fact, what I did
when I created the Colabs.

01:16:09.820 --> 01:16:12.800 align:middle line:84%
And you can actually make it a
bit more systematic by trying

01:16:12.800 --> 01:16:14.180 align:middle line:90%
lots of different values.

01:16:14.180 --> 01:16:18.320 align:middle line:84%
And there is a particular Python
package called KerasTuner.

01:16:18.320 --> 01:16:21.423 align:middle line:84%
So just Google KerasTuner, and
it comes with very nice Colabs.

01:16:21.423 --> 01:16:22.840 align:middle line:84%
And if I have a
chance, maybe I'll

01:16:22.840 --> 01:16:25.100 align:middle line:84%
just record a screen
walkthrough of doing that.

01:16:25.100 --> 01:16:27.540 align:middle line:84%
But that's the very efficient
way to do these things.

01:16:27.540 --> 01:16:29.082 align:middle line:84%
And it comes under
the broad category

01:16:29.082 --> 01:16:31.200 align:middle line:84%
of something called
hyperparameter optimization,

01:16:31.200 --> 01:16:34.120 align:middle line:84%
where the number of neurons, the
activation you use, the learning

01:16:34.120 --> 01:16:36.220 align:middle line:84%
rate, all those things
can all be tried.

01:16:36.220 --> 01:16:37.740 align:middle line:90%
You can try lots of variations.

01:16:37.740 --> 01:16:41.920 align:middle line:84%
And Keras is a great way to
do it in the context of Keras.

01:16:41.920 --> 01:16:42.820 align:middle line:90%
Other questions?

01:16:42.820 --> 01:16:45.420 align:middle line:90%


01:16:45.420 --> 01:16:45.920 align:middle line:90%
No?

01:16:45.920 --> 01:16:47.260 align:middle line:84%
All right, I'll give
you 30 seconds back.

01:16:47.260 --> 01:16:48.040 align:middle line:90%
Thank you.

01:16:48.040 --> 01:16:49.830 align:middle line:90%
See you tomorrow.

01:16:49.830 --> 01:17:03.000 align:middle line:90%