The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high quality educational resources for free.
To make a donation, or view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu.
JAMES DICARLO: So let me start by first--
I already alluded to this, but let's talk
about the problem of vision.
This is just one computational challenge
that our brains solve, but it's one that many of us
are very fascinated by.
As you'll hear in the rest of the course,
there are other problems that are equally fascinating.
But I'm going to talk about problems of vision.
I'm going to talk about a specific problem of vision,
and that's the problem of object recognition.
So I will try to operationalize that for you.
And one thing you'll see when I talk
is that our field, even though we
can be motivated by words like vision and object recognition,
we're going to only make progress
if we start to operationally define things
and then decide in what domain models are going to apply.
And I think that's an important lesson that I hope
will come across in my talk.
So this is the way computer vision operationally
defines part of the problem of object recognition and vision.
It's as if you take a scene like this
and you want to do things like come up
with an answer space that looks like this, where you
have noun labels, say a car.
And you have what are called bounding boxes around the cars,
similarly for people, or buildings, or trees,
or whatever nouns that you or DARPA or whoever
wants to actually label.
Right, so this is just one way of operationalizing vision.
But I think it gets at the crux of what we're after, which is,
there is what's called latent content in this image
that all of us instantly bring to our memories,
that we can say, aha, that's a car, that's a building.
There are nouns that pop into our heads.
We also know other latent information about these things,
like the pose of this car, the position of the car,
the size of the car.
The key point that I'm going to tell you today
about this problem is that that information feels to us
that it's obvious, but it's quite
latent in the image-- that's implicit
in the pixel representation.
Those of you who have worked on this problem
will understand this and those of you who haven't, I hopefully
will give you some flavor for what that problem feels like.
So I want to back up a bit.
This is more from a cognitive science perspective,
or a human brain perspective, to ask, why would we
even bother worrying about this problem of object recognition?
And maybe this is obvious that those of you-- and I
don't need to say this, but I like
to point out that we think of the representations
of the tokens of what's out there in the world
as being the substrates of what you might do,
what's called higher level cognition, things like memory,
value judgments, decisions and actions in the world.
Imagine building a robot and having
it try to act in the world and it doesn't even
really know what's out there.
So these are the sort of substrate of these kind
of cognitive processes.
Again, from an engineering perspective,
these are processes or behaviors.
This is just a short list of them
that might depend on your good abilities
to recognize and discriminate among different objects.
I think if you look through this list,
you could imagine things that would go terribly wrong if you
didn't actually do a good job at identifying
what's out there in the world.
So that's just to think about, again,
as an engineer building a robot.
This is a slide I stuck in that I
want to connect to this course, the idea
that I know many of you are from maybe these backgrounds,
or from this background.
And when I think about the brain,
I have this coin here to say, really
these are kind of two sides-- we're studying the same coin
from two directions here.
And really the question that we have
to all be excited about, I hope many of you
are excited about it is, how does the brain work?
And you could do computer science
and not care at all about this question.
I think it's a little harder to do these and not
care about this question.
But it's possible, I guess.
So these are all trying to answer this question.
And this is maybe pretty obvious,
but when you have biological brains that
are performing tasks better than current computer
systems, machines that humans have built,
then the flow tends to want to go this way.
You discover phenomena or constraints over here.
These lead to ideas that can be built into computer code that
can say, hey, can I build a better machine based
on what we discover over here?
And many of us who came into the field excited to do this
and are still excited of this kind of direction.
But an equally important direction
is that when you have systems that
are matched with our abilities, or that
can compute some of the things that we think the brain has
to compute, then the flow goes more this way,
where there's many possible ways to implement an idea
and these become falsifiable.
That is, that they can be tested against experimental data
to ask which of these many ways of implementing a computation
are the ones that are actually occurring in the brain.
And that's important if you say you
want to build brain-machine interfaces,
or fix diseases, or do something that's
on the level of interacting with the brain directly.
I hope that you guys keep this picture in mind
because I think it's sort of the spirit of the course
that both of these directions are important.
And it's not as if we work on this for 20 years and then work
on this for 20 years.
It's really the flow across them that I think
is the most exciting to us.
So just to connect to that, a little bit of history
of where was the field on this problem of visual recognition.
I don't know if many of you heard this,
but here you are at summer school,
so there was a Summer Vision Project--
it was called, at MIT.
I used to think this story was apocryphal.
In 1966, there was a project that the final goal
was object identification, which we'll actually name,
Objects by Matching the Vocabulary of Known Objects.
So this was essentially a summer project to say,
we're going to get a couple undergraduate students together
and we're going to build a recognition system in 1966.
And this was the excitement of AI,
we can build anything that we want.
And of course, those of you who know this,
this problem turned out to be much, much
harder than anticipated.
So sometimes problems that seem easy for us are actually
quite difficult. If any of you wants this,
I would be happy to share this document with you.
It's interesting, the space of objects that they describe
things like recognizing--
of course, I would say like coffee cups on your desk.
But they also say packs of cigarettes on your desk.
So this sort of dates the time of this here.
So it's a little bit like Mad Men or something.
So now, here we are today.
And I guess I just can't help but sort of get excited about,
here's this really cool machine that's just amazing
that does these computations.
The things got-- I can't tell you all this because of the 100
billion computing elements, solves problems not solveable
by any previous machine.
And the thing, it looks crazy, but it only
requires 20 watts of power.
Those of you who have seen this slide,
I'm not talking about this thing.
I'm talking about that thing right there.
So this is a scale of what we're after.
And we often talk about power, but this is something engineers
are especially interested in as they build these systems, is
how does our brains solve these problems at such a low wattage,
so to speak.
This is, again, the spirit of many of the things
that I hope that you guys are excited about in the future
of this field.
Here's another slide that I pulled out
that I often like to show is that, from an engineer's
point of view, we often try to say,
well, we want to build machines that are as
good or better than our brain.
So machines today, you guys know this, beat us
at many things, straight calculation,
they beat us at chess.
When I was a grad student, they recently won at Jeopardy.
In memory, they've always beaten us.
Machines are way better at memory than us
in the simple form of memory.
Seeing, in pattern matching, go to the grocery
store, hey, what's that bar code done?
I don't know what that was, but it just scans in
and somehow it does pattern matching, right?
So there's forms of vision that machines
are way better than us.
But some forms of vision that are more complicated that
require generalization, like object recognition,
or more broadly, scene understanding,
we like to think that we are still the winners at.
And even things that we take for granted, like walking,
this is quite a challenging problem.
So engineers really want to move this over here.
So our goal is to discover how the brain solves
object recognition.
And the reason I put this up is, from an engineering
point of view, that just doesn't mean write a bunch of papers
in a textbook that says, this part of the brain
does it, but actually help to implement a system where this
is, at least, matched with us and I assume someday,
will be better than us.
And this is also a gateway problem.
That is, even if it's just this domain,
we think that the systems we're studying
might generalize to other, for instance, sensory domains.
Gabriel told me you were going to do
an auditory, visual comparison session later in the week.
That's an engineer's point of view,
how do I just build better systems?
Let's step back and talk from a scientist's point of view.
So this is really now to introduce the talk that I'm
going to give you today.
So when you're a scientist, what's our job?
We say we want to understand.
We all write that, understand.
What does that mean?
Well, what it really means if you boil it down,
and I would love to discuss this if you like,
is that you have some measurements in some domain.
So you can think of this as a state space here.
This is like the position of the planets today.
And this is like the position of the planets tomorrow.
Or you could say, this is the DNA sequence inside a cell.
And this is some protein that's going to get made.
So you're searching for mappings that are predictive
from one domain to another.
And we can give lots of examples of what
we call successful science, where that's true.
This is the core of science is to predict, given
some measurements or observations, what's
going to happen either in the future or some other set
of measurements.
So predictive power is the core of all science
and the core of understanding.
And I think it would be fun if you want to debate that,
that you think there's another way.
But this is what I come to in thinking about this problem.
And the reason I'm bringing this up
is because the accuracy of this predictive mapping
is a measure of the strength of any scientific field.
And some fields are further along than others.
And I would say ours is still not very far along.
Our job is to bring it from a nonpredictive state
to a very predictive state.
And so that means building models that can be falsified
and that can predict things.
And you'll hear that through my talk.
As Gabriel mentioned, what we try to do
is build models that can predict either behavior or neural
activity.
And that's what we think is what progress looks like.
So now let's translate this to the problem I gave
you, which is the problem of vision or more generally
object recognition.
You could imagine, there's a domain of images.
So just to slow down here, just so everybody's
on the same page, each dot here might be
all the pixels in this image.
In this dot, all the pixels in this image.
So there's a set of possible pixel-images
that you could see.
And we imagine that they give rise to, in the brain,
some state space.
Think of this as the whole brain for now, to just fix ideas,
that you could imagine that this image, one you're looking at,
it gives rise to some pattern of activity
across your whole brain.
And this image gives rise to a different pattern of activity
across your whole brain.
And loosely, we call this the neural representation
of this thing.
But then what we do is somehow when we ask you
for behavior reports, there's a mapping
between that neural state space and what
we measure as the output.
Whether you say it or write it, you might say,
that's a face, these are both faces,
if I asked you for nouns among them.
OK, so this is another domain of measurement.
So now you can see I'm setting up the notion of predictivity.
And what we want to do is, we have
this complex thing over here of images that somehow
map internally into neural activity
and then somehow map to the thing
we call perceptual reports.
And notice I've already put things
that we call nouns that we usually
associate with objects, cars, face, dogs, cats, clocks,
and so forth.
OK, so understanding this mapping in a predictive sense
is really a summary of what our part of the field is about.
And again, accurate predictivity is the core product
of the science that underlies our ability
to build a system like this-- many of you
are interested, to fix a system like this,
or to perhaps even augment our own systems.
If we want to inject signals here and have them give rise
to percepts, we have to know how this works.
A big part of the field of vision
is spent-- a lot of the last three decades,
working on the mapping between images and neural activity.
That's usually called encoding, predictive encoding mechanisms.
And it's driven by Hubel and Wiesel's work.
The people saw this as a great way forward.
It's like, let's go study the neurons
and try to understand what in the image is driving them.
That is, what's an image computable
model in the world that would go from images
to neural responses?
The other part is that there's some linkage, we think,
between the neural activity and these reports.
And notice, this is actually why most of us
get into neuroscience because you
notice this arrow is two-way.
This is actually quite deep here.
From an engineer's point of view,
you go, well, there's got to be some mapping
between the neural activity and the button
presses on my fingers or my saying the word noun.
There's some causal linkage between this and the things
that we observe objectively in a subject.
But this is where philosophers debate about like, well, you
know in some sense these are sort
of two sides of the same coin.
We say our own perception, there's
some aspects of the internal activity
that are the thing that we call awareness or perception.
Now I'm not going to get into all that,
but I just want to point out that if you're just
building models, you can't approach that.
It's this sort of strange thing between neurons
and these reported states that many of us are fascinated by.
So this is called predictive decoding mechanisms.
For me, it's all going to be operationalized
in terms of reports from humans or animals.
And I'll not do that philosophical part,
but I thought I'd mention that for those you like
to think about those things.
So for visual object perception, I
want to point out that, again, the history of the field
has been mostly here.
This link has been neglected or dominated
by weakly predictive word models.
That doesn't mean they're not useful starting points,
but they're weakly predictive.
And so a weakly predictive word model would be--
and for temporal cortex, a part of the brain
I'm going to tell you about today, does object recognition.
That model has been around for a long time.
It is somewhat predictive because it says,
you take that out and all object recognition
will get destroyed, would be a prediction.
Turns out that doesn't actually happen.
We can discuss that.
But it doesn't tell you how it does it, how to inject signals,
which tasks are more or less affected,
so that's what I mean by weakly predictive.
It's a word model.
Face neurons do face task, that's
probably true to some extent.
But again, it doesn't tell us-- it's more tight.
It sort of says, oh, I'll take out these smaller regions
and there'll be some set of tasks that involve faces.
I don't know, I won't say anything about other tasks.
So that's a somewhat more strongly predictive model,
but still pretty weakly predictive.
And my personal favorite that comes in from reviewers a lot
is, attention solves that.
So this is just a statement that--
just to be on the lookout for word models
that don't actually have content in terms of prediction.
I don't know what that means.
I read this as, hand of God reaches in
and solves the problem.
So there's got to be an actual predictive model that
can be falsified.
OK, so I don't mean to doubt the importance of these.
Before people start giving me a hard time,
there are attentional phenomena, there are face neurons,
there is an IT, that's what we study.
I'm just trying to emphasize for you that we
need to go beyond word models into actual testable
models that make predictions, that would stand even
if the person claiming those models is no longer around,
it would make a prediction.
Let me try to define a domain.
I said we're going to try to define stuff.
It's hard to define stuff.
It's big, vision, it's a big area.
Object recognition, I sort of said it vaguely.
And when I say this, I include faces as an object,
a socially important when.
You'll hear this from Winrich I think.
But I want to say, to try to limit it even further,
that's still a big domain.
And so we tried early on to reduce the problem even further
to something that is more, again, naturalistic,
that we think can give us more traction,
this predictive sense.
So we started by saying, when you take a scene like this
and you analyze it, you may not notice it
but your ventral stream, really your retina has high acuity
in say the central 10 degrees.
There's anatomy that I'll show you later
that the ventral stream is especially
interested in processing the central 10
degrees of information.
So that's about two hands at arm's length,
for those you see in the room.
So you may have the sense that you know what's out there,
but you don't really.
You kind of stitch that together.
And lots of people have shown this,
the way you stitch this together is making rapid eye
movements around, called saccades,
followed by fixations, which are 200 to 500 milliseconds
in duration.
You don't really see during this time here.
It's not as if your brain shuts down,
it's just that the movement is too fast for your retina
to really keep up with this.
So you make these rapid eye movements,
you fixate, fixate, fixate.
And what you do is, that brings this sort
of sampled scene to the central 10 degrees
that might look something like this.
So those are 200 millisecond snapshots
across that scan path.
And I'll play it for you one more time.
Now, you should notice that there's
one or more objects in each and every image
that you probably said, oh, there's a sign.
There's a person.
There's a car.
You might have gotten two out of each one.
But you were sort of extracting, at least
intuitively to me, at least one or more foreground
or central objects when I show you those images.
And that ability to do what I just showed you there, we think
is the core of how you analyze or build up
a scene like this, at least how the ventral stream contributes.
And therefore, we call that core recognition,
which I defined as a central 10 degrees of visual field,
100 to 200 millisecond viewing duration.
And again, it's not all of object recognition,
but we think it's a good starting point.
And a way that we probably got into this is because of a rapid
serial visual presentation movies from the 70's.
Molly Potter showed this really nicely.
This is a movie that I've been showing for 15 years now.
Notice that this is just a sequence of images
where there is typically one or more foreground objects.
And you should be quickly mapping those to memory,
even though I'm not telling you what to expect.
Like Leaning Tower of Pisa, right, I'm
not going to tell you that you're going to see Star Wars
characters-- well, I just did.
But you quickly are able to map those things
to some noun or even a more precise subordinate noun.
I know this is Yoda.
So our ability to do that, we're very, very good at that.
Notice you didn't need a lot of pre-cueing,
yet you're still able to do that.
And that is really what fascinates us about vision
and object recognition in particular.
Even without featural attention or pre-cueing,
you're able to do a remarkable amount of processing.
And I think that's a great demonstration of that.
And just to quantify this for you,
because sometimes people say, well you're
showing it too short.
Your vision system doesn't do much.
Here's an eight-way categorization task
I'll show you later under range of transformation.
These are just the example images
of eight different categories of objects.
It doesn't really matter what I much do here,
you get a very similar curve.
And that is, you get most of the performance gain
in about the first 100 milliseconds.
This is accuracy, you're about 85% correct.
This is a challenging task, as I'll show you earlier.
It looks easy here, but it's quite challenging.
85% correct, if I let you look at the image longer,
up to two seconds, you can bump up to around 90's.
So there is some gain with longer viewing duration,
but you get--
chance is 50, so you get this huge ability.
And we're not the first to show this.
This is just to show you in our own kind of task
that the data I'm going to tell you about,
where we show the image for 100 or 200 milliseconds,
this is the typical primate viewing duration
that I pin this on.
We use this for reasons of efficiency.
But you see, the performance is similar across that time.
You get a lot done.
Your visual system does a lot of work in that first glimpse.
And that's core recognition that we are trying to study here.
And I know it's not all of object recognition
or all of vision, but it's now, we
think, a much more defined domain
that we can make progress on.
And that's what we've been working on.
And that's essentially what I'm going to talk about today.
So think of vision, object recognition,
within that core recognition.
This is David Marr.
David and Tommy Poggio, I studied with a long time.
And Tommy wrote the introduction to David's-- if you guys
haven't read this book, Vision--
has anybody, guys know this book?
It's really a classic book in our field.
It's the first couple chapters that
are the part you should really read.
That's the best part of the book.
And one of the things that you take from this book,
that I think David and Tommy helped
to lay out a long time ago, is that there
is this challenge of level.
I think one of the things I take from this
is, they would try to define three clean levels.
It turns out not to be this clean in practice.
But there's one level called computational theory, what's
the goal, what's appropriate, what's the logic,
and by what strategy can it be carried out.
There's another level which is, OK,
now once you decide that, how should you represent the data?
How can you implement an algorithm to do it?
And then there's this actually, how do you run it,
how do you build it in hardware?
And neuroscientists often come in, they're like,
I'm going to study neurons and it's sort of
like jumping into your iPhone and saying,
I'm going to study transistors.
They often tend to start at the hardware level.
And I think that's the biggest lesson you take from this like,
oh wait, there's something going on here,
these transistors are flying.
And you make some story about it if you
were recording from the brain or measuring
transistors in my iPhone.
But I think the important point to take from this
is it helps to start thinking about what's
the point of the system.
What might it be doing?
How might you solve that problem?
And that leads you then to algorithm.
And then you think about representations.
So it's sort of a top down approach,
rather than just digging into the brain
and hoping that the answers will emerge.
So I'm going to try to give you that top down approach
in this problem that I'm talking about.
I've already given you a bit of it by introducing you
to the problem.
I'll say a little bit more about that and step down a little bit
this way.
And so this kind of thinking, I think,
is important to making progress in how
the brain computes things.
So here's a related slide that I made a long time ago
that, again, I pulled out for you
guys, that I think helps bridge between what I just
said about the Marr levels of analysis
and whether you're a neuroscientist or cognitive
scientist, and are a computer vision or machine learning
person.
So the first is, what is the problem we're trying to solve?
So that's Marr computational level one.
So computational vision-- now operationally,
you'll hear folks in machine learning,
they might say, well, there's some benchmarks, that's good.
There's a ImageNet Challenge or whatever
challenge they want to solve.
Sometimes they'll say, well the brain solves it.
That's not good because they didn't really
define the problem.
Neuroscientists will say, well, it's
something like perception or behavior
or there's some sort of behavior that they imagined,
although characterizing that behavior
is not usually their primary goal.
But I think there is at least some progress in that regard.
Now what does a solution look like?
This is really just to talk about language.
So useful image representations for machine learning, like what
we might call features--
but neuroscientists will talk about explicit neuronal spiking
populations.
You heard this in Haim's talk.
He was using these words interchangeably.
Again, this may be obvious to you guys,
but I thought it's worth going through.
So this is like Marr level two, representation.
How do we instantiate these solutions?
So this is still level two algorithms,
or mechanisms that actually build useful feature
representations.
Neuroscientists will think about neuronal wiring and weighting
patterns that are actually executing those algorithms.
This is what we think is a bridging language there.
And then there's this deeper level
that came up in the questions, which
is, how would you construct it from the beginning?
Learning rules, initial conditions, training images,
are words that are used here.
There is a learning machine.
Here, neuroscientists talk about plasticity, architecture,
and experience.
But again, those are similar questions just
with different language.
And I'm doing this because I think the spirit of this course
is to try to build these links at all these different levels
here.
OK, so hopefully that kind of helps
orient you to how we think about it.
Let me just go and say, I want to talk about number one.
What is a problem we're trying to solve and why is it hard?
I said, object recognition is hard
and I showed you that MIT Challenge and it was difficult.
Maybe it's hard because there's lots of objects.
Who thinks that's why it's hard?
Who thinks that's not why it's hard?
You think computers can list a bunch of objects?
It's easy, right?
This is what I said about memory,
it's a big long list of stuff.
Computers are good at that.
There's going to be thousands of objects.
A list of objects is not a hard thing for a machine to do.
What's hard is that each object can produce an essentially
infinite number of images.
And so you somehow have to be able to take
some samples of certain views or poses of an object,
this is a car under different poses,
and be able to generalize or to predict what the car might
look like in another view.
This is what's called the invariance problem.
and it's due to the fact that, again, there's
identity preserving image variation.
This is why the bar code reader in your supermarket
works fine, because the code is always laid out very simply.
But when you have to be able to generalize
across a bunch of conditions, potentially things
like background clutter, even more severely occlusion, things
you heard from Gabriel, or you may even want to generalize
across the class of cars where the cars have slightly
different geometry but they're still cars,
these kind of generalizations are what make the problem hard.
So I'm lumping them all together in what
we call the invariance problem.
Many of you in the room know this is the hard problem.
And I think that hopefully it fixes ideas of, that's
what you should think about.
It's not the number of objects, but it's the fact
that it has to deal with that invariance problem.
Haim was talking about manifolds,
and this is my version of that.
So this is to introduce you to the problem of,
why that invariance problem--
what it looks like or feels like.
I'm not going to give you math on how to solve it.
It's just a geometric feel for the problem.
So if you imagine you're a camera--
or your retina, which is capturing
an image of an object, let's call this a person,
I think I called him Joe.
So when you see this image of Joe, and this is the retina,
so now this is a state space of what's going on in your retina.
So it's a million retinal ganglion cells.
Think of them as being an analog value out of each,
so this is a million dimensional state space.
So when you see this image of Joe,
he activates every retinal ganglion cell, some a lot,
some a little, but he's some point of that million
dimensional space.
OK, everybody with me?
If everybody's heard all this before and wants me to go on,
everybody wave your hand and I'll move on.
AUDIENCE: No, it's good.
JAMES DICARLO: Keep going, OK.
So the basic idea is that if Joe undergoes a transformation,
like a change in pose, what that does
is, it's only a 1 degree of freedom
I'm turning under the hood one of those latent variables.
If I had a graphics engine, I'm changing
the pose of latent variables.
It's only one knob that I'm turning, so to speak.
And that means there's one line through here
as Joe projects across these different images here.
And I'm ignoring noise and things.
This is just the deterministic mapping
onto the retinal ganglion cells.
So Joe goes--
[MOVING NOISE]
--and he goes over here.
And if I turn the other knob, he goes over here.
And so I could imagine, if I turned those two
knobs of two axis opposed always possible
and plotted this in the million dimensional state space,
there'd be this curved up sheet of points,
which you could think of Joe's identity manifold
over those two degrees of view change.
It's only two dimensions, it's hard to start
showing more than this.
But it's this curved up sheet of points.
Everybody with me so far?
You don't actually get to see all those.
You could imagine a machine actually running them all,
but you don't really get to see them.
You've got to get samples of them.
But there's some underlying manifold structure here.
Now, what's interesting and what's important to point out
is that this thing, even though I've drawn
it and it's a little curve, but it's highly complicated
in this native pixel space.
It's all curved up and bending all over the place.
And the reason that matters, and this
is what Haim introduced you to, is
that if you want to be able to separate Joe
from another object, say not Joe, another person say,
then you need a representation.
I showed you retinal ganglion cells.
This is another imaginary state space
where you can take simple tools to extract the information.
And the simple tools that we like to use
are linear classifiers.
But you can use other simple tools.
Haim used the exact same description to you
guys in his talk, that you have some linear decoder
on the state space that can say, oh, they can separate
cleanly Joe from not Joe.
So these manifolds are nicely separated
by a separating hyperplane.
That's what these tools tend to do is they like to cut planes.
This is one thing they like to do,
or they want to find locations or regions,
like compact regions in this space,
depending on what kind of tool you use.
But you don't want the tool having
to do all kinds of complicated tracing through this space.
That's basically the original problem itself.
So what you need is, you have a simple tool box,
which we think of as downstream neurons.
So a linear classifier, as an approximation,
it's like a dot product.
It's a weighted sum, which is what we think, neuroscientists,
of downstream neurons doing.
So it's a weighted sum.
And if we want an explicit representation
in some neural state space, then we
need to be able to take weighted sums of some population
representation to be able to separate Joe from not Joe,
and Sam from Jill, and everything from everything else
that we want to separate.
If we had such a space of neural population,
we'd call that a good set of features
or an explicit representation of object shape.
And for any aficionados here, it's
not just cleanly linear separation,
it's actually being able to find this
with a low number of training examples.
So that turns out to be important.
But it helps to fix ideas to think about linear separation,
ideally with a low number of training examples.
So that's a good representation.
And notice, I'm starting to mix up terms here.
I am assuming, when I talk about shape,
that that will map cleanly to identity,
or what you might call broadly, category.
That's another topic I won't talk about, if you just
think about the shape of Joe, or separating one geometry
from another.
Now, here's a simulation that my first graduate student, Dave
Cox, who's now at Harvard, did.
This is a number of years old.
This takes these two face objects, render them
under changes, and view.
And then he actually simulated the manifolds
in a 14,000 dimensional space.
And then he wanted to visualize it.
And because we wanted to try to make
the point that these manifolds of these two objects
are highly curved and highly tangled,
this is a three dimensional view.
Remember, it's sitting on a 14,000 dimensional simulation
space.
You can't view that space.
This is a three dimensional view of it.
And the point is that it's like two sheets of paper
being all crumpled up together and they're not fused.
They look fused here because it's in three dimensions.
But they're not actually fused.
But they're complicated, you can't easily
find a separating hyperplane to separate these two objects.
We call these tangled object manifolds.
And really, they're tangled due to image variation.
Remember, if I didn't change those knobs of view or position
or scale, there would just be two points in the space
and it would be easy.
That's the easy problem of listing objects.
But if they have to undergo all this transformation,
they become these complicated structures
that need to be untangled from each other.
So the problem that's being solved
is, you have this retina sampling data,
like a camera on the front end, where things look
complicated with respect to the latent variables,
in this case shape or identity, Sam or Joe.
And that they somehow are transformed, as Haim mentioned,
they're transformed by some non-linear transformation,
some other neural population state space, shown here, where
the things look more like this.
The latent variable structure is more explicit,
that you can easily take things like separating hyperplanes
to identify things like shape, which again, roughly
corresponds to identity or other latent parameters,
like position and scale.
You maybe haven't thrown away all these other latent
parameters.
And if I have time, I'll say something about that
so you don't just get identity.
But if you can untangle this, you
would have a very nice representation with regard
to those originally latent parameters.
That's the dream of what you'd like to do.
It's like reverse graphics, if you will.
So this is what we call an untangled explicit object
information.
And we think it lives somewhere in the brain,
at least to some degree.
And I'll show you the evidence for that later on.
So what you have then is you have a poor encoding basis,
the pixel space.
And somewhere in the brain is a powerful encoding basis,
a good set of features.
And as Haim mentioned, as I already said,
this must be a non-linear transformation
because the linear transformations are just
rotations of that original space.
So now let's go down to-- actually this
would be Marr level three.
Let's go to instantiation.
Let's get into the hardware here.
We're supposed to be talking about brains.
So I'm going to give you a tour of the ventral stream.
So we would love to know how this brain solves it.
This is the human brain.
This is a non-human primate.
This is not shown to scale.
This is blown up to show you it's
a similar structure, temporal lobe, frontal lobes,
occipital lobe.
There is a non-human primate.
We like this model for a number of reasons.
One reason that we like it is that they
are very visual creatures, their acuity
is very well matched to ours.
In fact, even their object recognition abilities
are actually quite similar to our own.
This may be surprising to you, but let me just
show you some data for that.
This is actually data from Rishi Rajalingham, in my lab.
It says, impressed, but this just came out.
This is the confusion matrix patterns
of humans trying to discriminate different objects
under those transformations that I showed you earlier,
where they're not just seeing images,
but they have to deal with these invariances.
And this is rhesus monkey data trying to do the same thing.
And the task goes, I'll give you a test image
and then you get choice images.
Was it a car or a dog?
I'll show you an image, what choice was it, a dog or a tree?
And you're trying to entertain many objects all at once,
and you get an image under some unpredictable view
and unpredictable background, and then
you have to make a choice.
So this is the confusion difficulty.
And when you look at this, it's intuitive
that these are sort of geometry similar.
Camel is confused with dog, and tank is confused with truck,
and that's true of both monkeys and humans.
And to some level, this shouldn't be surprising to you.
The same tasks that are difficult for humans
are difficult for monkeys because probably they
share very similar processing structures.
They don't have to bring in a bunch of knowledge
about tanks are driven by people or that, they just have to say,
was there a tank or a truck.
And under those conditions, they make very similar patterns
of confusion.
And these patterns are very different from those
that you get when you run classifiers
on pixels or low level visual simulations.
But they're very similar to each other, in fact,
are statistically indistinguishable,
monkeys and humans, on these kind of patterns of confusion.
OK, so that's one reason we like this subject, the monkey model,
is that the behavior is very well matched to the humans.
The other reason is that we know from a lot of previous work
that I alluded to, that some studies have shown that lesions
in these parts of the brain can lead to deficits in recognition
task.
So again, we think the ventral stream solves recognition.
So we know a weak word model of where to look,
we just don't know exactly what's going on there.
Just to orient you, these ventral
areas, V1, V2, V4, and infer temporal cortex, or IT cortex--
IT projects anatomically to the frontal lobe
to regions involved in decision and action,
and around the bend to the medial temporal lobe to regions
involved in formation of long-term memory.
Because these are monkeys and not humans,
and Gabriel mentioned this in his talk, we can go in
and we can record from their brains,
and we can perturb neural activity in their brains
directly.
And we can do that in a systematic way.
This is the advantage of an animal model as opposed
to a human model.
OK, as neuroscientists now, we've
taken a problem, translated it to behavior,
taken that behavior into a species we can study,
we know roughly where to look, and now
we want to try to understand what's going on.
So as engineers, we take these curled up sheets of cortex
and think of them as I've already been showing you,
as populations of neurons.
So there's millions of neurons on each of these sheets.
I'll give you numbers on a slide coming up.
There's some sort of processing that may be common here,
I put these T's in, there might be
some common cortical algorithm processing forward this way.
There's also inter-cortical processing.
And there's also some feedback processing going on in here.
So all that's schematically illustrated in this slide
that I'll keep bringing up here when
we talk about these different levels of the ventral stream.
Now I'm most going to be talking about IT cortex here
at the end.
Why do we call these different areas?
One reason is that there's a complete retina topic
map, a map of the whole visual space in each
of these different levels.
In retina, there's one.
In LGN-- in the thalamus, there's another.
In V1, there's another map.
In V2, there's another map.
In V4, there's another map.
In IT, it's less clear that it's retinotopic,
we're not even sure that IT is one area.
Maybe we'll have time, I'll say more about that detail.
So it's not that retinotopic in IT,
except the most posterior parts of IT.
But that's why neuroscientists divide these
into different areas.
So a key concept, though, for you computationally is,
think of each of these as a population representation
that's retransforming the data from that complicated space
to some nicer space.
And it's doing this probably in a stepwise, gradual manner.
So IT is believed to be that powerful encoding
basis that I alluded to earlier, where
you have these nice flattened object manifolds.
And I'll show you the evidence for that.
This is recently from a review I did that gives
more numbers on these things.
And I've sized the areas according
to their relative cortical area in the monkey.
Here's V1, V2, V4, IT.
IT is a complex of areas.
And I'm showing you these latencies.
These are the average latencies in
these different visual areas.
You can see, it's about 50 milliseconds
from when an image hits the retina
until you get activity in V1.
60 in V2, 70-- there's about a 10 millisecond step
across these different areas.
So it's about 100 millisecond lag between an image it's here,
and you start to see changes in activity at this level
up here that I'm referring to.
When I say IT, I'm referring to AIT and CIT together.
That's my usage of the word IT for the aficionados
in the room.
And that's about 10 million output neurons in IT
just to fix numbers.
In V1 here, you have like 37 million output neurons.
There's about 200 million neurons in V1, similar in V2.
And many of you probably heard about other parts
of the visual system.
Here's MT, many of you probably heard about MT.
So you can see it's tiny compared to some of these areas
that I'm talking about here.
I'm going to show you some neural dam--
I'm just going to give you a brief tour
of these different areas, so brief, it's almost cartoonish.
But at least those of you who haven't seen this
should at least be exposed.
So in the retina--
you guys know in the retina there's
a bunch of cell layers in the retina.
The retina is a complicated device.
I think of it as a beautiful camera.
So you're down in the retina.
To me, the key thing in the retina
is in the end you've got some cells that are going to project
back along the optic nerve.
So these are the retinal ganglion cells,
they actually live on the surface.
The light comes through, photo receptors are here,
there is processing in these intermediate layers,
and then there's a bunch of retinal ganglion cell types.
There's thought to be about 20 types or so.
The original physiology, there are
two functional central types where they
have on center or off center.
Let's take an on center cell, you shine light
in the middle of a spot-- now this
is a tiny little spot on the retina,
the size depends on where you are in the visual field.
But you shine a little bit of light in the center,
the response goes up.
See the spike rate going up here.
Put light in the surround, the response rate goes down.
So it has an on center, off surround profile.
And then there's a flip type here.
So that's the basic functional type.
When you think about the retina, it
is tiled with all of these point detectors that
have some nice center surround effects.
There's some nice gain control for overall illumination
conditions.
But my toy model of the retina, it's
basically a really nice pixel map coming back down
the optic track to the LGN.
OK, I'm going to skip the LGN and go straight to V1.
People have known for a long time, functionally V1
cells they have sensitivity to especially edges.
They have what's called orientation selectivity.
Hopefully this isn't new to you guys.
Here's a simple cell in V1.
If you shine a bar of a light on it
inside it's receptive field--
does everyone know what a receptive field is?
I don't want to go--
OK.
It's OK if you ask, because I want
to make sure you guys are OK.
So the receptive field, you shine a bar light in it,
turn it on in the right orientation,
gives good response out of the cell.
Move it off this position, now not much response,
there's a little bit of an off response here.
Change the orientation, nothing happens.
Full field illumination, nothing happens.
OK, so this is called selectivity.
That is, there's some portion of the image space
that it cares about.
It doesn't just respond to any light
at that spot like the pixel wise, retinal ganglion cell
would.
So now there's this complex cell that's
also in V1, which maintains this orientation
selectivity across a change in position,
as shown here, also across some changes in scale.
So it maintains it, meaning that you have this tolerance--
so that's called position tolerance, for position.
You can move the bar around it, still likes that oriented bar.
But you change its angle and it goes down,
so it still maintains the same selectivity here
but it has some tolerance.
So you get this build up of some orientation sensitivity
followed by some tolerance.
And there are models from Hubel and Wiesel
that they thought that you could build this first
and then you build these out of these,
that's the simple version.
And here they are.
These are the Hubel and Wiesel models,
how you build these and like operators to build selectivity
from pixel-wise cells with an and like operator lining
these up correctly.
You can imagine oriented tuned cells built this way.
There's evidence for this in physiology
that this is how these are constructed.
The tolerance of these complex cells
is thought to build by a combination of simple cells.
And there's some evidence for this.
And this is again, all the way from Hubel and Wiesel,
who won a Nobel Prize for this and related work in the 1960s.
And then there were a bunch of computational models
that are really inspired by this and I
think are still the core models of how the system works.
And some of the original ones that were written down
are Fukushima in the '80s, and then
Tommy Poggio and others built what's called an HMAX Model,
you guys have probably heard about,
that's off of these similar ideas, much more
refined and much more matched to the neural data.
But I'm just try to point out that these kind
of physiological observations are
what inspired this class of largely feedforward models
that you heard about much today.
So that's a brief tour of V1.
Now, what's going on in V2?
For a long time, people thought it
was hard to tell the difference from V1 and V2.
And I just thought I'd show you guys,
this is a slide I stuck in, this is from Eero
Simoncelli and Tony Movshon.
And I think you guys have Eero teaching in the course a bit
later, so he may say some of this.
But V2 cells have some sensitivity to natural image
statistics that V1 cells don't.
And maybe I'll see if I can take you through this.
So the way that they did this is you can simulate--
so this is all driven off of work that Eero and Tony have
done--
especially Eero has done on texture synthesis.
So you have these original images,
and if you run them through a bunch of V1-like filter banks,
and then you take a new image, a random seed, which
is like white noise, and you try to make sure
that it would activate populations
of V1 cells in a similar way, there's
a large set of images that would do that because you're just
doing summary statistics, but these
are some examples of them.
For this image, this is one that one might look like.
So you can see, to you, it doesn't look the same as this.
But to V1, these are metamers, they're
very similar in the summary statistics in V1.
And then you start taking cross products of these V1 summary
statistics and then you try to match those.
And what's interesting is you start
to get something that looks, texture wise, much more
like this original image.
And this is a big part of what Eero and others did
in that work.
And the reason I'm showing you this
is that Tony's lab has gone and recorded in V1 and V2
with these kinds of stimuli, and the main observation they have
is that V1 doesn't care whether you show it this or this.
To V1, these are both the same, which
says we have the summary statistics
for V1 right in terms of the average V1 response.
That's all I'm showing you here.
The paper, if you want it, is much more detailed.
But you go to V2 and there's a big difference
between this, which V2 cells respond to more, and this,
which they respond to less.
And really one inference you can take from this
is that V2 neurons apply a repeated-- another and like
operator on V1.
That's a simple inference that these kinds of data seem
to support .
And they also tell you that these and-like operators,
these conjunctions of V1 statistics
tend to be in the direction of the statistics
of the natural world, that's naturalistic statistics.
Now lots of controls haven't been done here
to narrow in exactly what kinds ands,
but that's the spirit of where the field is
in trying to understand V2.
Everybody thinks it has something
to do with corners or a more complicated structure.
But this is a way that current in the field
to try to move these image computing models forward
in V1 and V2.
And Tony likes to point out that this is one of the strongest
differences that you see between V1 and V2,
other than the receptive field sizes.
So I think that's quite some exciting work if you
don't know about it on V2.
OK, then you get up into V4 and things get much murkier.
So what's going on in V4?
Well, let me just briefly say that one of my post-docs-- this
is more recent work just because it builds on that earlier work.
This is Nicole Rust, when she was a post-doc in the lab,
compared V4.
She actually compared it to IT.
I'll skip that.
But she was using these Simoncelli scrambled images.
These are actually the texture images from--
these are the original images and these
are the texture versions.
So this should look like a textured version of that.
You can see that these algorithms don't actually
capture the object content of these images.
And what Nicole actually showed is that similar to what
you just saw there, in the earlier work like V1,
V4 doesn't care about the differences between these.
It responds similarly, as a population, to this and this,
and this and this, and this and this.
But IT cares a lot about this versus this.
So this is just repeating the same theme, the general idea
that you have and -like operators that we think
are aligned along the ventral stream that
are tuned to the kind of statistics
that you tend to encounter in the world.
And this is some of the evidence for it in V2,
and then later in V4, and IT, and Nicole's work,
if you piece that all together.
When you go to a place like V4, remember V4
is now like three levels up.
And what does V4 do?
Look, this is Jack's work in 1996.
This is from Jack Gallant when he was
working with David Van Essen.
And people had some ideas that maybe there
are these certain functions that V4 neurons like,
and they would show these--
the same thing people have done in V2,
they would show a bunch of images like this
and figure out, well, does it like these Cartesian gratings
or these curved ones.
And you know what, you get out of this is,
you could tell some story about it,
but you get a bunch of responses out of it.
The color indicates the response.
And you kind of look at it, and people would tell some stories,
but it really was just kind of like tea leaves.
Here's a bunch of data, we don't really
know what these V4 neurons were doing.
This was a science paper, so you could go back and read it.
And then Ed Connor and Anitha Pasupathy
worked together a few years after that
to try to figure out more about what V4 neurons do.
And they did things like take images
like this, which were isolated, and try
to cut them into parts, like curved parts, pointy parts,
curved, concave, convex.
And this was motivated off of some psychology literature.
And they would define these based
on the center of the object.
So this wasn't an image computable model,
it was just a basis set that they
built around these silhouette objects.
And so they made this basis set about any kind of silhouetted
object they like here.
They hypothesized that they could
fit the responses of V4 neurons in this basis set.
And this was their attempt to do it.
They could actually fit quite well.
And that's kind of what's being shown here.
Here's the response of a V4 neuron.
The color indicates the depth of the response.
You can see, this is sort of like that previous slide,
you're looking at tea leaves.
It looks complicated, but under this model
they were able to, in the shape space, explain about half
of the response variants of V4 neurons.
The upshot is, that V4 curve is about some combination
of curves.
And then later, Scott Brincat, with Ed,
went on into posterior IT and showed
that maybe some combinations of these V4 cells
could fit posterior IT responses quite well.
So if you read the literature in V4 and IT,
you'll come across these studies.
And they are important ones to look at.
Unfortunately, they don't give you
an image computable model of what these neurons are doing.
But it's some of the work that you should know about
if you want to look in V4 or early IT,
so I'm telling it to you.
So let me go on to IT, which is what I want to talk
about for the rest of today.
Again, I'm talking about AIT and CIT.
And I'll just quickly say that the anatomy, again,
suggests that the IT is the central 10 degrees.
And even though V1, V2, and V4 cover the whole visual field,
if you make injections in V4, that's
shown here, where you make injections
in the more peripheral parts of the V4 representation, which
is up here, that you don't get much projection into IT, which
is here.
You don't see much green color, whereas, you make projections
in the center part of V4, these red sites here,
you see much more coverage into IT, which is shown here.
So when I say 10 degrees, that's rough.
Everything in biology is messy.
But this is some of the evidence, beyond recordings,
there's anatomical evidence that as you go down into IT,
you are more and more focused on the central 10 degrees.
OK, let me talk about a little bit of the history of IT
recordings.
This is when people got excited about IT, in the 70s.
This is work by Charlie Gross, who's one of the first people
to record an IT cortex.
And I'll show you what they did here.
This was in an era where, remember, Hubel and Wiesel
had just done their work in the '60s.
And they recorded from the cat visual cortex.
And they had found these edge cells,
and they ended up winning the Nobel Prize for that.
So it was the heyday of like, let's record
and figure out what makes cells go.
So they were brave enough to put an electrode down an IT cortex
in 1970 and said, what makes this neuron go.
Remember, that's an encoding question,
what's the image content that will drive this neuron.
And it's fun to just look back on this
and what they were doing.
So they didn't have computer monitors.
They were actually waving around stimuli
in front of the animals.
This is an anesthetized animal on a table.
This is a monkey.
Actually, they started with a cat
and then they later went to monkey.
The use of these stimuli was begun one day when,
having failed to drive a unit with any light stimulus-- that
probably means spots of light, edges things
that Hubel and Wiesel had been using.
We waved a hand at the stimulus screen,
they waved in front of the monkey,
and elicited a very vigorous response
from the previously unresponsive neuron.
And then we spent the next 12 hours-- so the animal's
anesthetized on the table, their recording from this neuron.
It's 12 hours because nothing's moving,
so you can record for a long period of time.
So singular neuron, they're recording,
listening to the spikes.
We spent the next 12 hours testing various paper cut
outs in attempt to find the trigger feature.
You can see, that's a Hubel and Wiesel idea,
what makes this neuron go.
What's the best thing, that's become
a lot of what the field spent time doing.
Trigger feature for this unit, when the entire stimulus set
were used, were ranked according to the strength of the response
that they produced.
We could not find a simple physical dimension
that correlated with this rank order.
However, the rank order of adequate stimuli
did correlate with similarity for us,
that means psychophysical judged,
to the shadow of a monkey hand.
So these are their rank order of the stimuli.
And they say look, it looks like it's some sort of hand neuron.
That's all I know how to describe it.
I can't find some simple thing on here.
So this kind of study then launched a whole domain
where people started to go in to record these neurons
and they found interesting different types.
Bob Desimone, who worked with Charlie Gross,
later showed much more nicely under more controlled
conditions, yes, there are indeed neurons that respond.
You can see more to these hand-- this is the post stimulus time
histogram, lots of spikes, lots of spikes, lots of spikes--
respond more to these hands than to these other kind
of stimuli here.
So you could say, these neurons have
tuned to specific combinations of high selectivity.
You'll hear from Winrich that others
had shown that you could record some
of the neurons are really like faces that you could find,
and not so much hands.
So you could find neurons that seem
to have some interesting selectivity in IT cortex.
And then others later went on to show
in a number of studies-- this is from Nico Logothetis' work
of a number of years later.
It's just one example that this selectivity had some tolerance
to, say, the position of the stimulus, that's
what's shown here.
The fact that these bars are high just
means that it tolerates movement in where the--
sorry, this is size, degrees of visual angle.
This is position, moving the stimulus around.
So this was known for a number of years
that there's some tolerance to position and size
changes at least.
OK, so I'm putting these up and you say,
there's some selectivity and there's some tolerance.
And that should remind you of what we already said in V1,
there's some selectivity, simple cells.
There's some tolerance, complex cells.
So you have the same themes here,
just different kinds of types of stimuli being used.
Then people really went on, in the 80s especially, and said,
let's go after this trigger feature.
And Tanaka's group really went after this really hard.
Tanaka's group would find the best stimulus
they would find, dangle a bunch of objects
in front of a recorded neuron, find the best out
of a whole set of objects, and then they
try to do a reduction.
They'd try to figure out, how can I reduce this.
This is their attempt to reduce the stimulus to its features
without lowering the neural response.
So high response, high response, high response, high response,
high response, suddenly I do this, the response drops.
I do this, the response drops.
And they have lots of examples of this.
And they want you to try to get to the simplest thing that
could capture the response.
And when they did this, they would take stimuli like this,
and end up with stimuli that looked like that.
Now, many of you should probably start to wonder here,
there's lots of paths for stimulus space.
It's not clear that these are elemental in any way.
There's lots of ways that you can show with modeling
that you can get easily lost in this space of navigating around
here.
This is just, again, a history of the work.
This is the kind of things that people were doing.
And then from that, they presented
what we think of as the ice cube model of IT,
that I think is actually still a very reasonable approximation.
They not only showed that neurons
tended to like certain relatively reduced stimulus
features, not full objects, but that they
are gathered together.
So these are millimeter scale regions of IT
that nearby neurons, within a millimeter or so,
have similar preferences.
They're not just scattered willy-nilly
throughout the tissue.
When you go record nearby neurons, they're similar.
So there's some mapping within IT cortex.
This is schematic here.
This is optical imaging data of IT cortex also
from Tanaka's group that show you
that these different blobs of tissue
get activated by different images shown here.
And I'm just showing you the scale of this,
it's around a little less than a millimeter.
And our lab has evidence of this too.
So there's some sort of spatial organization in IT,
but we really don't really yet understand the features,
these elemental features yet, or at least, not at this time.
Then later, there's lots of beautiful work in IT.
Again, I'm probably not telling you all of it.
Some of the most exciting work recently--
and you'll hear about this from Winrich,
that people started to use fMRIs.
So Doris Tsao and Winrich Freiwald and Marge Livingstone
all together started to use fMRI data to compare
faces versus objects.
This was motivated from human work,
by work like Nancy Kanwisher lab and others.
What they found was that in monkeys,
you could find different parts that would show up,
what are called face patches, where
you have a relative preference for faces over objects.
Again, I don't want to take all of Winrich's talk here,
but you have these different patches here.
And then what's really cool is, you go in and record
from these patches and then you find a very enriched locations
for face neurons.
And these enriched locations were known
from a number of other studies.
But this is a nice correlation between functional imaging
and this enrichment of these face cells.
And that's what's shown here, that these neurons respond
mostly to faces and not so much other objects.
Although, you see they still sort of respond to these.
So this kind of says fMRI and physiology are
telling you similar things.
It also tells you there's some spatial clumping, at least
for face-like objects, at a scale of a few millimeters
or so, the size of these patches.
OK, so that's larger scale organization.
This is data from our own lab that shows the same thing.
Maybe I'll just skip through this in the interest of time--
that we can map and record the neurons very precisely,
map them spatially and compare that with fMRI.
So this is just a larger field of view maps of the same idea.
So what we have then, just to wrap up
this whirlwind tour of the ventral stream,
is that we had some untangled explicit information.
And what I want to try to convince you of now, is that--
I've told you about the ventral stream,
but I'm going to try to tell you that, in IT cortex,
this is a powerful representation for encoding
object information.
And then we'll take a break because we've already probably
been going a while.
Yeah, about 10 more minutes and then we'll take a break.
So what I've told you is, I've led you up the ventral stream,
I've given you a bit of the history,
so now let's talk about IT more precisely.
So now this is work from my own lab.
You go in and record IT.
You go record extracellularly.
You travel down into IT cortex, which is down here.
And you record from this.
And similar to what you saw, another version of what
you saw from Charlie Gross or Bob Desimone,
you show a bunch of images.
And they could be arbitrary images.
You take an IT recording site, and see these little dots,
those are action potential spikes out of a particular IT
site.
And these are repeatable.
You have some Poisson variability here.
But you see that there's more spikes here,
there's little more here, less here, less there.
These images are all randomly interleaved
when you collect the data, as I'll show you in a minute.
And you go to different sites and it likes different images.
So there is certainly some image selectivity.
This should not be surprising because I already showed
you this from previous work.
This is just data from our own lab.
You can also see now that you are looking closely
at the time lag, remember, I said around 100 milliseconds
stimulus on.
Stimulus off, the stimulus is actually off
before the spikes actually start to occur out
here in IT because, again, there's a long time
lag, 100 milliseconds.
OK, so that's what the neural responses look like.
I don't know if you guys can hear this,
maybe I should have hooked up audio.
Maybe you might be able to hear--
this is actually a recording that Chou Hung did
when he collected his data in my lab for the early studies
we did in the lab.
I don't know if you guys can hear.
[STATIC]
[BEEP]
Those high beeps are the animal getting reward
for fixating on that dot.
You're not even going to be able to parse that.
I mean, you hear the spikes clicking by, those--
[STATIC]
Those are action potentials.
And I don't expect you to look at anything like, oh, it's
a face neuron, or whatever.
I just want you to get a feel for how those data were
originally collected.
This is a pretty grainy video.
But you get the idea.
You collect data like that.
And again, you can find selectivity in those population
patterns, as I just showed you.
But then, Gabriel and Tommy and I, so the three of us,
I think all in this room, way back when
in 2005 said, well look, the population of IT
might have good, useful information
for solving this difficult object manifold
tangling problem.
It might be a good explicit representation.
So we did a, what I call, early test of this idea.
We took this simple image set from eight different categories
that we had chosen.
And there's good stories of why we chose those objects,
if you like to hear them.
But let me just say, simple objects, we moved them
across position and scale, and we collected the responses
of IT of a bunch of sites to changes
to all these different visual images.
And we showed them as I just showed you.
We just showed them for 100 milliseconds.
This is this core recognition regime,
were just showing them for 100 milliseconds.
And then we show another one, and they're just
randomly interleaved.
And from this, what you do is you
could get a population set of data
where we recorded 350 IT sites.
Here's a sample of 63 sites.
This is 78 images, the mean neural response
here is the mean response to an image.
This is 78 of the images we showed.
There's nothing for you to read into here to say, other than,
you have this rich population data.
And now our question is, well, what lives in this population
data that we've collected.
Is it explicit with regard to categories?
So we come back to what I showed you
earlier about those tangled manifolds and said,
we need simple decoding tools.
Can a simple decoding tool look at that population
and tell me what's out there?
And again, we were using linear classifiers at the time,
because we took that, as you heard
from Haim as our operational definition of what
a simple tool is.
And if it could decode information about the object
identity, then we'd say, well, that means,
by that operational definition, this
is explicit, available, accessible information, or just
generally good.
So if you imagine that the activity-- this is schematic.
Each dot, this is neuron one, neuron two,
and you could have a bunch of IT neurons.
But if you can separate any object
from all the other object, these points
represent the population response
to each image of an object.
Remember, there's many images of each object.
But if you could linearly separate that,
that would mean it was explicit.
And if you had a hard time separating it,
this would be implicit.
These are like tangled object manifold.
This is Inaccessible, or bad, information.
So we just-- we, and when I mean we,
I mean Chou Hung, who led the study.
Gabriel, Tommy, and I did this.
We took the response of an image, like this one.
It produced a population vector.
Again, we recorded a bunch of neurons.
We recorded them sequentially and then pieced together
this population vector.
So these are the spikes simulated off
a population of IT.
We could do various things.
In fact, I think Gabriel did everything possible,
as I remember at the time.
And one of the things we did was just count spikes.
One of the simple things, that turns out to work quite well,
is count the spikes over 100 milliseconds.
So this neuron counts spikes.
That gives you a number, one number here,
count spikes get one number.
So you have n neurons, you get n numbers.
So it's a point in a n dimensional state space where
n is the number of neurons.
And then we had already pre-divided the images
into different categories, as shown here.
These are the categories.
And again, we just asked how well
you could do faces versus non-faces,
toys versus non-toys, so on and so forth.
These are old slides.
But you get the idea, is that basically, you
don't need that many sites to already get
to very high levels of performance
on both categorization and identification.
The interesting thing about this was
that you could solve simple forms
of this invariance problem in this representation
quite easily.
That if you just trained on the central objects, the center
and size, the simple three degree size center position,
and test it on the same thing, just
held out repeats of this data, you did quite well.
That's a baseline.
But what's interesting is you test at different position
and scale.
And then you also do almost nearly as well.
So you naturally generalize to these other conditions
by training on these simple conditions.
So this is evidence that the population
is a good basis set for solving these kind of problems.
A few number of training examples on this population
then generalizes, well, across conditions
makes the problem hard.
So again, we published that a long time ago.
This was an early step to say, look,
the phenomenology looks right for the story that I've
been telling you so far.
You can't do this easily in earlier visual areas like V1,
or simulated V1 or V4.
And we later show that a number of ways.
This is consistent with work I was showing you
with Logothetis position tolerance, size
tolerance, the selectivity.
It's really just an explicit test of the idea population
encoding.
So the take home here is that there's this explicit object
representation in IT.
I didn't prove to you that this is
the link, this predictive model to decoding yet.
We're going to talk about that next.
But this was some of the important population
phenomenology that we did.
What I try to tell you today--
hopefully I've introduced you to the problem of visual object
recognition and the way we restricted it
to core object recognition.
We talked a lot about predictive models as being the goal,
although I haven't presented much to you yet.
Hopefully, that's the second part of the talk.
I've given you a tour of the ventral stream.
But it was a poor tour.
I'm sure everybody i work with would
say that you've neglected all this work because there's
no way I can do that all in even a whole week.
I just tried to hit some of the highlights for you.
And I told you that the IT population
seems to have solved a key problem,
this sort of invariance problem that I set up.
And one way to step back and say, over the last 40 years
or so, from those early studies of Charlie Gross
or even Hubel and Wiesel, we, the field of ventral stream
physiology, we've largely described important
phenomenology.
Even that last study is population phenomenology.
And so now we need these more advanced models.
So the next phase of the field is developing and testing
these predictive models that I've
motivated at the beginning, but I
haven't given you much of yet.
So this was hopefully a bit of history and set context
to where we are.