The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high quality educational resources for free.
To make a donation or to view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu.
PHILIPPE RIGOLLET: Doesn't to run Flash Player.
So I had to run them on Chrome.
All right, so let's move on to our second chapter.
And hopefully, in this chapter, you
will feel a little better if you felt
like it was going a bit fast in the first chapter.
And the main reason we actually went fast, especially
in terms of confidence intervals.
Some of you came and asked me what
you mean by this is a confidence interval?
What does it mean that it's not happening
in there with probability 95%, et cetera?
I just went really fast so that you
could see why I didn't want to give you a first week doing
probability only without understanding
what the statistical context for it was.
So hopefully, all these things that we've
done in terms of probability, you actually
know why we've been doing them.
And so we're basically going to go back to what we're doing,
maybe start with some statistical setup.
But the goal of this lecture is really
going to go back again to what we've seen from a purely
statistical perspective.
All right?
So the first thing we're going to do
is explain why we're doing statistical modeling, right?
So in practice, if you have data,
if you observe a bunch of points--
in here, I gave you some numbers, for example.
So here's the partial data sets with the number of siblings,
including self, that were collected from college
students a few years back.
So I was teaching a class like yours
and actually asked students to go
and fill out some Google form and tell me a bunch of things.
And one of the questions was, including yourself,
how many siblings do you have?
And so they gave me this list of numbers, right?
And there's many ways I can think of this list of numbers,
right?
I could think of it as being just a discrete distribution
on the set of numbers between 1--
I know there's not going to be an answer which is less than 1,
unless, well, someone doesn't understand the question.
But all the answers I should get are positive integers--
1, 2, 3, et cetera.
And there probably is an upper bound,
but I don't know it on the top of my head.
So maybe I should say 100.
Maybe I should say 15.
It depends, right?
And so I think the largest number I got for this was 6.
All right?
So here you can see you have pretty standard families,
you know, lots of 1s, 2s, and 3s.
What statistical modeling is doing
is to try to compress this information that I
could actually describe in a very naive way.
So let's start with the basic usual statistical set
up, right?
So I will start with many of the boards that look
like x1, xn, random variables.
And what I'm going to assume, as we said typically
is that those guys are IID.
And they have some distribution, all right?
So they all share the same distribution.
And the fact that their IID is so that I can actually
do statistics.
Statistics means looking at the global averaging thing
so that I can actually get a sense of what
the global behavior is for the population, right?
If I start assuming that those things are not identically
distributed-- they all live on their own--
that my sequence of number is your number of siblings--
the shoe size of this person--
the depth of the Charles River, and I
start measuring a bunch of stuff.
There's nothing I can actually get together.
I need to have something that's cohesive.
And so here, I collected some data that was cohesive.
And so the goal here--
the first thing is to say what is the distribution that I
actually have here, right?
So I could actually be very general.
I could just say at some distribution p.
And let's so those are random variables, not random vectors,
right?
I could collect entire vectors about students,
but let's say those are just random variables.
And so now I can start making assumptions
on this distribution p, right?
What can I say about a distribution?
Well, maybe if those numbers are continues,
for example, I could assume they have a density--
a probability density function.
That's already an assumption.
Maybe I could start to assume that they're probability
density function is smooth.
That's another assumption.
Maybe I could actually assume that it's piecewise constant.
That's even better, right?
And those things make my life simpler and simpler,
because what I do by making the successive assumptions is
reducing the degrees of freedom of the space in which I
am actually searching the distribution.
And so what we actually want is to have something
which is small enough so we can actually
have some averaging going on.
But we also want something which is big enough
that it can actually express.
It has chances of actually containing a distribution that
makes sense for us.
So let's start with the simplest possible example,
which is when the xi's belong to 0, 1.
And as I said, here, we don't have a choice.
The distribution of those guys has to be Bernoulli.
And since they are IID, they all share the same p.
So that's definitely the simplest possible thing
I could think of.
They are just Bernoulli p.
And so all I would have to figure out in this case is p.
And this is the simplest case.
And unsurprisingly, it has the simplest answer, right?
We will come back to this example
when we study maximum likelihood estimators or method
of moments estimators by method of moments.
But at the end of the day, the things
that we did-- the things that we will do
are always the naive estimator you
would come up with is what is the proportion of 1.
And this will be, in pretty much all respects,
the best estimator you can think of.
All right?
So then we're going to try to assess this performance.
And we saw how to do that in the first chapter as well.
So this problem here somehow is completely understood.
We'll come back to it.
But there's nothing fancy that is going to happen.
But now, I could have some more complicated things.
For example, in the example of the students now,
my xi's belong to the sequence of integers 1, 2, 3,
et cetera, OK, which is also denoted by n, maybe without 0
if you want to put 0 in that, right?
So the positive integers.
Or I could actually just maybe put
some prior knowledge about how humans
have time to have families.
But maybe some people thought of their college mates
as being their brothers and sisters.
And one student would actually put 465 siblings,
because we're all good friends.
Or maybe they actually think that all their Facebook
contacts are actually their siblings.
And so you never know what's going to happen.
So maybe you want to account for this,
but maybe you know that people are reasonable,
and they will actually give you something like this.
Now intuitively, maybe you would say, well,
why would you bother doing this if you're not
really sure about the 20?
But I think that probably all of you
intuitively guess that this is probably a good idea
to start putting this kind of assumption
rather than allowing for any number in the first place,
because this eventually will be injected
into the precision of our estimators.
If I allow anything, it's going to be more complicated for me
to get an accurate estimator.
If I know that the numbers are either 1 or 2, then
I'm actually going to be slightly more accurate as well.
All right?
Because I know that, for example, somebody put the 5,
I can remove it.
Then it's not going to actually corrupt with my estimator.
All right, so now, let's say we actually
agree that we have numbers.
And here I put seven numbers, OK?
So I just said, well, let's assume
that the numbers I'm going to get
are going to be 1 all the way to say this number that I denote
by larger than or equal to 7, which
is a placeholder for any numbers larger than 7, OK?
Because I know maybe I don't want
to distinguish between people that have 9 or 25 siblings.
OK, and so now, this is a distribution
on seven possible values-- the discrete distributions.
And you know from your probability class
that the way you describe this distribution
is using the probability mass function.
OK, or PMF--
So that's how we describe a discrete distribution.
And the PMF is just a list of numbers, right?
So as I wrote here, you have a list of numbers.
And here, you wrote the possible value
that your random variable can take.
And here you rate the probability
that your random variable takes this value.
So the possible values being 1, 2, 3 all the way to larger than
or equal to 7.
And then I'm trying to estimate those numbers.
Right?
If I give you those numbers, at least up to this
you know compression of all numbers that are equal to 7,
you have the full description of your distribution.
And that is the ultimate goal of statistics, right?
The ultimate goal of statistics is
to say what distribution your data came from,
because that's basically the best you're
going to be able to do.
Now admittedly, if I started looking at the fraction of 1s,
and the fraction of 2s, and the fraction of 3s, et cetera,
I would actually eventually get those numbers--
just like looking at the fraction of 1s
gave me a good estimate for p in the Bernoulli case,
it would do the same in this case, right?
It's a pretty intuitive idea.
It's just the law of large numbers.
Everybody agrees with that?
If I look at the proportion of 1s, the proportion of 2s,
the proportion of 3s, that should actually
give me something that gets closer and closer, as my sample
size increases to what I want.
But the problem is if my sample size is not huge,
here I have seven numbers to estimate.
And if I have 20 observations, the ratio
is not really in my favor--
20 observations to estimate seven parameters-- some of them
are going to be pretty off, typically
the ones with the large values.
If you have only 20 students, look at the list of numbers.
I don't know how many numbers I have, but it probably
is close to 20--
maybe 15 or something.
And so if you look at this list, nobody's
actually-- nobody has four or more siblings, right?
There's no such person.
So that means that eventually from this data set,
my estimates--
so those numbers I denote by say p1, p2, p3, et cetera--
those estimates p4 hat would be equal to what from this data?
0, right?
And p5 hat equal to 0 and p6 hat would be equal to 0.
And p larger than or equal to 7 hat would be equal to 0.
That would be my estimate from this data set.
So maybe this is not--
maybe I want to actually pull some information
from the people who have less siblings
to try to make a guess, which is probably
slightly better for the larger values, right?
It's pretty clear that in average, there is more than 0--
the proportion of the population of households
that have four children or more is definitely more than 0,
all right?
So it means that my data set is not
representative of what I'm going to try
to do is to find a model that tries to use the data they have
for the smaller values that I can observe and just push it up
to the other ones.
And so what we can do is to just reduce those parameters
into something that's understood.
And this is part of the modeling that I talked about
in the first place.
Now, how do you succinctly describe a number of something?
Well, one thing that you do is the Poisson distribution,
right?
Why do Poisson?
There's many reasons.
Again, that's part of statical modeling.
But once you know that you have number of something that
can be modeled by a Poisson, why not try a Poisson, right?
You could just fit a Poisson.
And the Poisson is something that looks like this.
And I guess you've all seen it.
But if x follows a Poisson distribution
with parameter lambda, than the probability
that x is equal to little x is equal to lambda
to the x over factorial x e to the minus lambda.
OK?
And if you did the sheet that I gave you on the first day,
you can check those numbers.
So this is, of course, for x equals 0, 1, et cetera, right?
So x is in natural integers.
And if you sum from x equals 0 to infinity, this thing you get
is e to the lambda.
And so they cancel, and you have some
which is equal to 1, which is indeed a PMF.
But what's key about this PMF is that it never takes value 0.
Like this thing is always strictly positive.
So whatever value of lambda I find from this data
will give me something that's certainly
more interesting than just putting the value 0.
But more importantly, rather than having
to estimate seven parameters and, as a consequence,
to actually have to estimate 1, 2,
3, 4 of them being equal to 0, I have only one parameter
to estimate which is lambda.
The problem with doing this is that now lambda may not
be just something as simple as computing the average number.
Right?
In this case, it will.
But in many instances, it's actually not clear
that this parametrization with lambda that I chose--
I'm going to be able to estimate lambda just by computing
the average number that I get.
It will be the case.
But if it's not, remember this example of the exponential
we did in the last lecture--
we could use the delta method and things
like that to estimate them.
All right, so here's modeling 101.
So the purpose of modeling is to restrict
the base of possible distributions
to a subspace that's actually plausible, but much simpler
for me to estimate.
So we went from all distributions
on seven parameters, which is a large space--
that's a lot of things--
to something which is just one number.
This number is positive.
Any question about the purpose of doing this?
OK, so we're going to have to do a little bit of formalism now.
And so if we want to talk--
this is a statistics classroom.
I'm not going to want to talk about the Poisson model
specifically every single time.
I'm going to want to talk about generic models.
And then you're going to build to plug in your favorite word--
Poisson, binomial, exponential, uniform--
all these words that you've seen,
you're going to be able to plug in there.
But we're just going to have some generic notations
and some generic terminology for a statistical model.
All right?
So here is the formal definition.
So I'm going to go through it with you.
OK, so the definition is that of a statistical model.
OK?
Sorry, that's a statistical experiment, I should say.
So a statistical experiment is actually just a pair--
E. And that's a set--
and a family of distributions P theta, where theta ranges
in some set capital theta.
OK?
So I hope you're up to date with your Greek letters.
So the small theta is the capital theta.
And enough of us--
I don't have the handwriting.
So if you don't see something, just ask me.
And so this thing now-- so each of this guy
is a probability distribution.
All right?
So for example, this could be a Poisson with parameter theta
or a Bernoulli with parameter theta--
OK, or an exponential with parameter--
I don't know-- 1 over theta squared if you want.
OK, but they're just indexed by theta.
But for each theta, this completely
describes the distribution.
It could be more complicated.
This theta should be a pair-- could be a pair-- a mu sigma
square.
And that could actually give you some n mu sigma squared.
OK so anything where you can actually--
rather than actually giving you a full distribution,
I can compress into a parameter.
But it could be worse.
It could be this guy here.
Right?
Theta could be p1--
p larger than or equal to 7.
And my distribution could just be something that has PMF--
p1-- p larger than 7.
That's another parameter.
This one is seven dimensional.
This one is two dimensional.
And all these guys are just one dimensional.
All these guys are parameters.
Is that clear?
What's important here is that once they give you theta,
you know exactly all the probabilities associated
with this random variable.
You know its distribution perfectly.
So this is the definition.
Is that clear?
Is there a question about this distribution--
about this definition, sorry?
All right.
So really, the key thing is the statistical model associated
to a statistical experiments.
OK?
So let's just see some examples.
It's probably just better because, again, the formalism
is never really clear.
Actually, that's the next slide.
OK, so there's two things we need to assume.
OK, so the purpose of a statistical model
is once I estimate the parameter,
I actually know exactly what distribution it has, OK?
So it means that I could potentially
have several parameters that give me
the same distribution that would still be fine, because I could
estimate one guy.
Or I could estimate the other guy.
And I would still recover the underlying distribution
of my data.
The problem is that this creates really annoying
theoretical problems, like things
don't work, the algorithms won't work,
the guarantees won't work.
And so what we typically assume is that the model
is so-called well-specified.
Sorry, that's not well specified.
I'm jumping ahead of myself.
OK, well-specified means that your data--
the distribution of your data is actually one of those guys.
OK?
So some vocabulary-- so well-specified
means that for my observations x,
there exists a theta in capital theta
such that x follows p sub theta.
I should put a double bar.
OK, so that's what well-specified means.
So that means that the distribution
of your actual data is just one of those guys.
This is a bit strong of an assumption.
It's strong in the sense that--
I don't know if you've heard of this sense, which I don't know,
I can tell you who it's attributed to,
but that probably means that this person did not
come up with it.
But I said that all models are wrong, but some of them
are useful.
All right, so all models are wrong
means that maybe it's not true that this Poisson distribution
that I assume for the number of siblings for college students--
maybe that's not perfectly correct.
Maybe there's a spike at three, right?
Maybe there's a spike at one, because you know,
maybe those are slightly more educated families.
They have less children.
Maybe this is actually not exactly perfect.
But it's probably good enough for our purposes.
And when we make this assumption,
we're actually assuming that the data really
comes from a Poisson model.
There is a lot of research that goes on
about misspecified models and that tells you
how well you're doing in the model that's
the closest to the actual distribution.
So that's pretty much it.
Yeah?
AUDIENCE: [INAUDIBLE].
PHILIPPE RIGOLLET: So my data--
so it's always the way I denote one
of the generic observations, right?
So my observations are x1, xn.
And they're IID with distribution p--
always.
So x is just one of those guys.
I don't want to write x5 or x4.
They're IID.
So they all have the same distribution.
So OK-- no, no, no.
They're all IID.
So they all have the same p data.
They'll have the same p, which means
they'll have the same p data.
So I can pick any one of them.
So I'd just remove the index just so we're clear.
OK?
So when I write x, I just mean think of x1.
Right they're an idea.
I can pick whichever I want.
I'm not going to write x1.
It's going to be weird.
OK?
Is that clear?
OK.
So this particular theta is called the true parameter.
Sometimes since we're going to want some variable theta,
we might denote it by theta star as opposed
to theta hat, which is always our estimator.
But I'll keep it to be theta for now.
And so the aim of this physical experiment
is to estimate theta so that once I actually
plug in theta in the form of my distribution, for example,
I could plug in theta here.
So theta here was actually lambda.
So once I estimate this guy, I would plug it in,
and I would know the probability that my random variable takes
any value, by just putting the lambda hat and the lambda hat
here.
OK?
So my goal is going to be to estimate
this guy so that I can actually compute those distributions.
But actually, we'll see, for example,
when we talk about regression that this parameter actually
has a meaning in many instances.
And so just knowing the parameter itself
intuitively or say more--
let's say more so than just computing probabilities,
will actually tell us something about the process.
For example, we're going to run linear regression.
And when we do linear regression,
there's going to be some coefficients
in the linear regression.
And the value of this coefficient
is actually telling me what is the sensitivity of the response
that I'm looking at to this particular input.
All right?
So just knowing if this number is larger
or if this number is small is actually
going to be useful for us to just look at this guy.
All right?
So there's going to be some instances where
it's going to be important.
Sometimes we're going to want to know if this parameter is
larger or smaller than something or if it's equal to something
or not equal to something.
And those things are also important-- for example,
if theta actually measures the true--
right?
So theta is the true unknown parameter-- true efficacy
of a drug.
OK?
Let's say I want to know what the true efficacy of a drug is.
And what I'm going to want to know is maybe it's a score.
Maybe I'm going to want to know if theta is larger than 2.
Maybe I want to know if theta is the average number of siblings.
Is this true number larger than 2 or not?
Right?
Maybe I am interested in knowing if college students come from--
so maybe from a sociological perspective,
I'm interested in knowing if college students come
from households with more than two children.
All right, so those can be the questions
that I may ask myself.
I'm going to want to know maybe theta is going
to be equal to 1/2 or not.
So maybe for a drug efficacy, is it completely
standard-- maybe for elections.
Is the proportion of the population
that is going to vote for this particular candidate
equal to 0.5?
Or is it different from 0.5?
OK, and I can think of different things.
When I'm talking about the regression,
I'm going to want to test if this coefficient is actually
0 or not, because if it's 0, it means
that the variable that's in front of it actually goes out.
And so those are things we're testing.
Actually having this very specific yes/no answer
is going to give me a huge intuition or huge understanding
of what's going on in the phenomenon that I observe.
But actually, since the questions are so precise,
it's going to be much more--
I'm going to be much better at answering them rather
than giving you an estimate for theta
with some confidence around it.
All right, it's sort of the same principle as trying to reduce.
What you're trying to do as a statistician
is to inject as much knowledge about the question and about
the problem that you can so that the data has
to do a minimal job.
And henceforth, you actually need less data.
So from now on, we will always assume--
and this is because this is an intro stats class--
you will always assume that theta--
the subset of parameters is a subset of r to the d.
That means that theta is a vector
with at most a finite number of coordinates.
Why do I say this?
Well, this is called a parametric model.
So it's called a parametric model or sometimes
parametric statistics.
Actually, we don't really talk about parametric statistics.
But we talk a lot about nonparametric statistics
or a non-parametric model.
Can somebody think of a model which is non-parametric?
For example, in the siblings example,
if I did not cap the number of siblings to 7,
but I let this list go to infinity,
I would have an infinite number of parameters to estimate.
Very likely, the last ones would be 0.
But still, I would have an infinite number of parameters
to estimate.
So this would not be a parametric model
if I just let this list of things
to be estimated to be infinite.
But there's other classes that are actually infinite
and cannot represented by vectors.
For example, function-- right?
If I tell you my model, pf, is just
the distribution of x's, the probability distributions,
that have density f, right?
So what I know is that the density is non-negative
and that it integrates to one, right?
That's all I know about densities.
Well f is not something you're going
to be able to describe with a finite number of values, right?
All possible functions is the huge set.
It's certainly not representable by 10 numbers.
And so non-parametric estimation is typically
when you actually want to parametrize this
by a large class of functions.
And so for example, histograms is the prime tool
of non-parametric estimation, because when
you fit a histogram to data, you're
trying to estimate the density of your data,
but you're not trying to represent it
as a finite number of points.
That's really-- I mean, effectively,
you have to represent it, right?
So you actually truncate somewhere and just
say those things are not going to matter.
All right?
But really the key thing is that this is non-parametric
where you have a potentially infinite number of parameters.
Whereas we're going to only talk about finites.
And actually finite in the overwhelming majority of cases
is going to be 1.
So theta is going to be a subset of r1.
OK, we're going to be interested in estimating
one parameter just like the parameter of a Poisson
or the parameter of an exponential--
the parameter of Bernoulli.
But for example, really, we're going
to be interested in estimating mu and sigma
square for the normal.
So here are some statistical models.
All right?
So I'm going to go through them with you.
So if I tell you I observe--
I'm interested in understanding--
I'm still [INAUDIBLE] I'm interested in understanding
the proportion of people who kiss by bending
their head to the right.
And for that, I collected n observations.
And I'm interested in making some inference
in the statistical model.
My question to you is, what is the statistical model?
Well, if you want to read the statistical model,
you're going to have to write this E--
oh, sorry, I never told you what E was.
OK, well actually just go to the examples,
and then you'll know what E is.
So you're going to have to write to me an E and a p theta, OK?
So let's start with the Bernoulli trials.
So this e here is called the sample space.
And in the normal people's words,
it just means the space or the set in which x and--
back to your question, x is just a generic observation lips.
OK, and hopefully, this is the smallest you can think of.
OK, so for example, for Bernoulli trials,
I'm going to observe a sequence of 0's and 1's.
So my experiment is going to be-- as written on the board,
is going to be 1, 0, 1.
And then the probability distributions
are going to be, well, it's just going
to be the Bernoulli distributions indexed
by p, right?
So rather than writing p sub p, I'm
going to write it as Bernoulli p,
because it's clear what I mean when I write that.
Is everybody happy?
Actually, I need to tell you something more.
This is a family of distributions.
So I need p.
And maybe I don't want to have to p
that's a value 0, 1, right?
It doesn't make sense.
I would probably not look at this problem
if I anticipated that everybody would kiss to the right.
And everybody would kiss to the left.
So I am going to assume that p is in 0, 1,
but does not have 0 and 1.
OK?
So that's the statistical model for a Bernoulli trial.
OK, now the next one, what do we have?
Exponential.
OK?
OK, so when I have exponential distributions,
what is the support of the exponential distribution?
What value is it going to take?
0 to infinity, right?
So what I have is that my first space
is the value that my random variables can take.
So its-- well, actually I can remove the 0 again--
0 to plus infinity.
And then the family of distributions
that I have are exponential with parameter lambda.
And again, maybe you've seen me switching
from p, to lambda, to theta, to mu, to sigma square.
Honestly you can do whatever you want.
But its just that it's customary to have this particular group
of letters.
OK?
And so the parameters of an exponential
are just positive numbers.
OK?
And that's my exponential model.
What is the third one?
Can somebody tell me?
Poisson, OK?
OK, so Poisson-- is a Poisson random verbal
discrete or continuous?
Go back to your probability.
All right, so the answer being the opposite of continuous--
good job.
All right, so it's going to be--
what value can a Poisson take?
All the natural integers, right?
So 0, 1, 2, 3, all the way to infinity.
We don't have any control of this.
So I'm going to write this as n without 0.
I think in the slides, it's n-star maybe.
Actually, no, you can take value 0.
I'm sorry.
This actually takes value 0 quite a lot.
That's typically, in many instances, actually the mode.
So it's n, and then I'm going to write it
as Poisson with parameter-- well,
here it's again lambda as a parameter.
And lambda can take any positive value.
OK?
And that's where you can actually see that the model
that we had for the siblings-- right?
So let me actually just squeeze in the siblings model here.
So that was the bad model that I had in the first place
when I actually kept this.
Let's say we just kept it at 7.
Forget about larger than or equal to 7.
We just assumed it was 7.
What was our sample space?
We said 7.
So it's 1, 2, to 7, right?
Those were the possible values that this thing would take.
And then what was my--
what's my parameter space?
So it's going to be a nightmare write.
But I'm going to write it.
OK, so I'm going to write it as something like the probability
that x is equal to k is equal to p sub k.
OK?
And that's going to be for p.
OK, so that's for all k's, right?
Or for k equal 1 to 7.
And here the index is the set of parameters p1 to pk.
And I know a little more about those guys, right?
I know there are going to be non-negative--
PJ non-negative.
And I know that they sum to 1.
OK, so maybe writing this, you start
seeing why we like those Poisson, exponential,
and short notation, because I actually don't have
to write the PMF of a Poisson.
The Poisson is really just this.
But I call it Poisson so I don't have
to rewrite this all the time.
And so here, I did not use a particular form.
So I just have this thing, and that's what it is.
The set of parameters is the set of positive numbers of--
p1 to p7, pk--
and sum to 7, right?
And so this as just a list of numbers
that are non-negative and sum up to 1.
So that's my parameter space.
OK?
So here that's my theta.
This whole thing here--
this is my capital theta.
OK?
So that's just the set of parameters
that theta-- the set of parameters
that theta is allowed to take.
OK, and finally, we're going to end with the star of all,
and that's the normal distribution.
And in the normal distribution, you still
have also some flexibility in terms of choices,
because then naturally, the normal distribution
is parametrized by--
the normal distribution is parametrized by two parameters,
right?
Mean and variance.
So what values can a Gaussian random variable take?
An entire real line, right?
And the set of parameters that it
can take it-- so this is going to be n, mu, sigma square.
And mu is going to be positive.
And stigma square is going--
sorry, m is going to be an r.
And sigma square is going to be positive.
OK, so again here, that's the way
you're supposed to write it.
If you really want to identify what theta is,
well, theta formally is the set of mu sigma square such that--
well, in r times 0 infinity, right?
That's just to be formal, but this does the job just fine.
OK?
You don't have to be super formal.
OK, that's not three.
That's like five.
Actually, I just want to write another one.
Let's call it 5-bit.
And 5-bit is just Gaussian with known variants.
And this arises a lot in labs when
you have measurement error--
when you actually receive your measurement device.
This thing has been tested by the manufacturer
so much that it actually comes in on the side of the box.
It says that the standard deviation of your measurements
is going to be 0.23.
OK, and actually why you do this is because we
can brag about accuracy, right?
That's how they sell you this particular device.
And so you actually know exactly what sigma square is.
So once you actually get your data in the lab,
you actually only have to estimate mu,
because stigma comes on the label.
So now, what is your statistical model?
Well, the numbers are collecting still in r.
But now, the models that I have is n, mu, sigma squared.
But the parameter space is not mu, and r, and sigma positive.
It's just mu and r.
And to be a little more emphatic about this,
this is enough to describe it, right?
Because if sigma is the sigma that
was specified by the manufacturer,
then this is the sigma you want.
But you can actually write sigma is equal to--
sigma square is equal to sigma square manufacturer.
Right?
You can just fix it to be this particular value.
Or maybe you don't want to write that index that's
the manufacturer.
And so you just say, well, the sigma--
when I write n squared what I mean
is the sigma square from the manufacturer.
Yeah?
AUDIENCE: [INAUDIBLE]
PHILIPPE RIGOLLET: Yeah.
For a particular measuring device?
You know, you're in a lab, and you have some measuring device.
I don't know-- something that measures
tensile strength of something.
And it's just going to measure something.
And it will naturally make errors.
But it's been tested so much by the manufacturer
and calibrated by them.
They know it's not going to be perfect.
But they knew exactly what error it
was making, because they've actually tried it
on things for which they exactly knew
what the tensile strength was.
OK?
Yeah.
AUDIENCE: [INAUDIBLE]
PHILIPPE RIGOLLET: This?
AUDIENCE: [INAUDIBLE]
PHILIPPE RIGOLLET: Oh, like that's pointing to--
5 prime?
OK?
And we can come up with other examples, right?
So for example, here's another one.
So the names don't really matter, right?
I call it the siblings model.
But you won't find the siblings model in the textbook, right?
So I wouldn't worry too much.
But for example, let's say you have something-- so
let's call it 6.
You have-- I don't know--
a truncated-- and that's the name I just came up with.
But it's actually not exactly describing what I want.
But let's say I observe y, which is the indicator of x larger
than say 5 when x follows some exponential with parameter
lambda.
OK?
This is what I get to observe.
I only observe if my waiting time
was more than five minutes, because I
see somebody coming out of the Kendall Station
being really upset.
And that's all I record is I've been waiting
for more than five minutes.
And that's all I get to record.
OK?
That happens a lot.
These are called censored data.
I should probably not call it truncated,
but this should be censored.
OK?
You see a lot of censored data when you ask people
how much they make.
They say, well, more than five figures.
And that's all they want to tell you.
OK?
And so you see a lot of censored data in survival analysis,
right?
You are trying to understand how long your patients are going
to live after some surgery, OK?
And maybe you're not going to keep people alive,
and you're not going to actually be
in touch in their family every day and ask them,
is the guy still alive?
And so what you can do is just you
ask people maybe five years after your study
and say, please, come in.
And you will just happen to have some people say, well, you
know, the person is deceased.
And you will only be able to know that the person deceased
less than five years ago.
But you only see what happens after that, OK?
And so this is this truncated and censored data.
It happens all the time just because you
don't have the ability to do better than that.
So this could happen here.
So what is my physical experiment, right?
So here, I should probably write this like this,
because I just told you that my observations are going to be x,
but there is some unknown y.
I will never get to see this y.
I only get to see the x.
What is my statistical experiment?
Please help me.
So is it the real line?
My sample space-- is it the real line?
Sorry, who does not know what this means?
I'm sorry.
OK.
So this is called an indicator.
So I read it as--
if I wrote well, that would be one with a double bar.
You can also write i if you prefer
if you don't feel like writing one in double bars.
And it's one of say--
I'm going to write it like that--
1 of a is equal to 1 if a is true and 0 if a is false.
OK?
So that means that if y is larger than 4, this thing is 1.
And if y is not larger than 5, this thing is 0.
OK.
So that's called an indicator--
indicator function.
It was very useful to just turn anything into a 0, 1.
So now that I'm here, what is my sample space?
0, 1.
Well, whatever this thing I did not tell you
was taking value with the thing you should have--
if I end up telling you that is taking value 6 or 7 that
would be your sample space, OK?
OK, so it takes values 0, 1.
And then what is the probability here?
What should I write here?
What should you write without even thinking?
Yeah.
So let's assume there's two seconds
before the end of the exam.
You're going to write Bernoulli.
And that's where you're going to start checking if I'm going
to give you extra time, OK?
So you write Bernoulli without thinking,
because it's taking value 0, 1.
So you just write Bernoulli, but you still have to tell me
what possible parameters this thing is taking, right?
So I'm going to write it p, because I don't know.
And then p take value--
OK, so sorry.
I could write it like that.
Right?
That would be perfectly valid, but actually no more.
It's not any p.
The p is the probability that an exponential lambda
is larger than 5.
And maybe I want to have lambda as a parameter.
OK, so what I need to actually compute is,
what is the probability that y is larger than 5--
when y is this exponential lambda,
which means that what I need to compute
is the integral between 5 and infinity of--
what is it?
1 over lambda.
How did I define it in this class?
Did I change it-- what?
AUDIENCE: [INAUDIBLE].
PHILIPPE RIGOLLET: Yeah, right, right, right.
Yeah.
Lambda e to the minus lambda x dx, right?
So that's what I need to compute.
What is this?
Yeah, so what is the value of this integral?
Can you take appropriate measures?
AUDIENCE: [INAUDIBLE]
PHILIPPE RIGOLLET: OK?
And again, you can cancel this, right?
So when I'm going to integrate this guy,
those guys are going to cancel.
I'm going to get 0 for infinity.
I'm going to get a 5 for this guy.
And well, I know it's going to be positive number, so I'm not
really going to bother with the signs,
because I know that's what it should be.
OK, so I get e to the minus 5 lambda.
And so that means that I can actually write this like that--
and now parametrize this thing by lambda positive.
OK?
So what I did here is I changed the parametrization from p
to lambda.
Why?
Well, because maybe if I know this is happening,
maybe I am actually interested in reporting lambda
to MBTA, for example.
Maybe I'm actually trying to estimate 1 over lambda, so
that I know it is--
well, lambda is actually the intensity
of arrival of my Poisson process, right?
I have a Poisson process.
That's how my trains are coming in.
And so I'm interested in lambda.
So I will parametrize things by lambda.
So the thing I get is lambda.
You can play with this, right?
I mean, I could parametrize this by 1 over lambda
and put 1 over lambda here if I want it.
But you know, the context of your problem
will tell you exactly how to parametrize this.
OK?
So what else did I want to tell you?
OK, let's do a final one.
By the way, are you guys OK with Poisson exponential,
Bernoulli's--
I don't know, binomial, normal--
all these things.
I'm not going to go back to it, but I'm
going to use them heavily.
So just spend five minutes on Wikipedia
if you forgot about what those things are.
Usually, you must have seen them the in your probability class.
So they should not be a crazy name.
And again, I'm not expecting you.
I don't remember what the density of an exponential is.
So it would be pretty unfair of me
to actually ask you to remember what it is.
Even for the Gaussian, I don't expect
you to remember what it is.
But I want you to remember that if I add 5 to a Gaussian, then
I have a Gaussian with me and mu plus 5 if I multiply it
by something, right?
You need to know how to operate those things.
But knowing complicated densities
is definitely not part of the game.
OK?
So let's do a final one.
I don't know what number I have now.
I'm going to just do uniform.
That's another one.
Everybody knows what uniform is?
So it's uniform, right?
So I'm going to have x, which my observations are
going to be uniform on the interval 0 theta, right?
So if I want to define a uniform distribution
for a random variable, I have to tell you which interval
or which set I want it to be uniform on.
And so here I'm telling you is the interval 0 theta.
And so what is going to be my sample space?
AUDIENCE: [INAUDIBLE]
PHILIPPE RIGOLLET: I'm sorry?
0 to theta.
And then what is my probability distribution?
My family of parameters?
So well, I can write it like this, right?
Uniform theta, right?
And theta let's say is positive.
Can somebody tell me what's wrong with what I wrote?
This makes no sense.
Tell me why.
Yeah?
Yeah, this set depends on theta, and why is that a problem?
AUDIENCE: [INAUDIBLE]
PHILIPPE RIGOLLET: There is no theta.
Right now, there's the families of theta.
Which one did you pick here?
Right, this is just something that's indexed by theta,
but I could have very well written it as, you know,
just not being Greek for a second,
I could have just written this as t rather than theta.
That would be the same thing.
And then what the hell is theta?
There's no such thing as theta.
We don't know what the parameter is.
This parameter should move with everyone.
And so that means that I actually am not
allowed to pick this theta.
I'm actually-- just for the reason that there is no
parameter to put on the left side--
there should not be, right?
So you just said, well, there's a problem because the parameter
is on the left-hand side.
But there's not even a parameter.
I'm describing the family of possible parameters.
There is no one that you can actually plug it in.
So this should really be 1.
And I'm going to go back to writing
this as theta because that's pretty standard.
Is that clear for everyone.
I cannot just pick one and put it in there and just take the--
before I run my experiments, I could potentially
get numbers that are all the way up to 1,
because I don't know what theta is going to be ahead of time.
Now, if somebody promised to me that theta
was going to be less than 0.5, that would be--
sorry, why do I put 1 here?
I could put theta between 0 and 1.
But if somebody is going to promise me, for example,
if theta is going to be less than 1,
then you expect to put 0, 1.
All right?
Is that clear?
OK, so now you know how to answer the question--
what is the statistical model?
And again, within the scope of this class,
you will not be asked to just come up with a model right that
will just tell you.
Poisson would be probably be a good idea here.
And then you would just have to trust me that indeed it
would be a good idea.
All right, so what I started talking about 20 minutes ago--
so it's definitely ahead of myself
is the notion-- so that's when I was
talking about well-specified.
Remember, well-specified says that the true distribution
is one of the distributions in this parametric families
of distribution.
The true distribution of my siblings
is actually a Poisson with some parameters.
And all I need to figure out is what this parameter is.
When I started saying that, I said, well,
but then that could be that there
are several parameters that give me
the same distribution, right?
It could be the case that Poisson 5 and Poisson 17
are exactly the same distributions when
I started putting those numbers in the formula which I erased,
OK?
So it could be the case that two different numbers would give me
exactly the same probabilities.
And in this case, we see that the model is not identifiable.
I mean, the parameter is not identifiable.
I cannot identify the parameter, even if you actually gave me
an infinite amount of data, which means that I could
actually estimate exactly the PMF.
I might not be able to go back, because there would
be several candidates, and I would not
be able to tell you which one it was in the first place.
OK?
So what we want is that this function--
theta maps to p theta is injective.
And that really can be fancy.
What I really mean is that if theta
is different from theta prime, then p of theta
is different from p of theta prime.
Or, if you prefer to think about the contrapositive of this,
this is the same as saying that if p theta gives me
the same distribution as theta prime,
then that implies that theta must
be equal to the theta prime.
The logic of those two things are equivalent, right?
So that's what this means.
So this is-- we say that the parameter is identifiable
or identified-- it doesn't really matter--
in this model.
And this is something we're going to want.
OK?
So in all the examples that I gave you,
those parameters are completely identified.
Right?
If I tell you--
I mean, if those things are in probability box,
it means that they were probably thought through, right?
So when I say exponential lambda,
I'm really talking about one specific distribution and not--
there's not another lambda going to give you
exactly the same distribution.
OK so that was the case.
And you can check that, but it's a little annoying.
So I would probably not do it.
But rather than doing this, let me just
give you some examples where it would not be the case.
Again, here's an example, if I take xi--
so now I'm back to just using this indicator function--
but now for a Gaussian.
So what I observe is x is the indicator
that y is, what did we say?
Positive.
OK?
So this is a Bernoulli random variable, right?
And it has some parameter p.
But p now is going to depend-- sorry,
and here y is n mu sigma square.
So the p, the probability that this thing is positive,
is actually--
I don't think I put the 0.
Oh, yeah, because I have mu.
OK, so this distribution-- this p the probability
that it's positive is just the probably
that some Gaussian is positive.
And it will depend on mu and sigma, right?
Because if I draw a 0, and I draw my Gaussian around mu,
then the probability of this Bernoulli being 1
is really the area under the curve here.
Right?
And this thing-- well, if mu is very large,
it's going to become very large.
If mu is very small, it's going to become very small.
And if sigma changes, it's also going to effect--
is that clear for everyone?
But we can actually compute this, right?
So the parameter p that I'm looking for here
as a function of mu and sigma is simply
the probability that some y is non-negative,
which is the probability that y minus mu divided by sigma
is larger than minus mu divided by sigma.
But when you study probability, is that some operation you
were used to making?
Removing the mean and dividing by the standard deviation?
What is the effect of doing that on the Gaussian
random variable?
Yeah, so you normalize it, right?
And you standardize it.
You make it a standard Gaussian.
You remove the mean.
The mean 0 is Gaussian.
And you remove the variance for it to become 1.
So when you have a Gaussian, remove the mean
and divide by the standard deviation,
it becomes a standard Gaussian--
which this thing has n , 0, 1 distribution,
which is the one you can read the quintiles of at the end
of the book.
Right?
And that's exactly what we did.
OK?
So now you have the probability that some standard Gaussian
exceeds negative mu over sigma, which
I can write in terms of the cumulative distribution
function, capital phi--
like we did in the first lecture.
So if I do this cumulative distribution function,
what is this probability in terms of phi?
[INAUDIBLE]?
AUDIENCE: [INAUDIBLE].
PHILIPPE RIGOLLET: Well, that's what your name tag says.
1 minus--
AUDIENCE: [INAUDIBLE].
PHILLIPPE RIGOLLET: 1 minus mu of sigma.
What happens with phi in our--
do you think I defined this for fun?
1 minus phi of mu over sigma, right?
Right?
Because this is 1 minus the probability
that it's less than this.
And this is exactly the definition
of the cumulative distribution function.
So in particular, this thing only depends on mu over sigma.
Agreed?
So in particular, if I had 2 mu over 2 sigma,
p would remain unchanged.
If I have 12 mu over 12 sigma, this thing
would remain unchanged, which means
that p does not change if I scale mu
and sigma by the same factor.
So there's no way just by observing x, even
an infinite times, so that I can actually get exactly what p is.
I'm never going to be able to get mu and sigma separately.
All I'm going to be able to get is mu over sigma.
So here, we say that mu sigma--
the parameter mu sigma--
or actually each of them individually-- those guys--
they're not identifiable.
But the parameter mu over sigma is identifiable.
So if I wanted to write a statistical model in which
the parameter is identifiable--
I would write 0, 1 Bernoulli.
And then I would write 1 minus phi over mu over sigma.
And then I would take two parameters,
which are mu and r and sigma squared positive.
So let's write sigma positive.
Right?
No, this is not identifiable.
I cannot write those two guys as being two things different.
Instead, what I want to write is 0, 1, Bernoulli 1 minus--
and now my parameter--
I forgot this-- my parameter is mu over sigma.
Can somebody tell me where mu over sigma lives?
What values can this thing take?
Any real value, right?
OK, so now I've done this definitely out of convenience,
right?
Because that was the only thing I was able to identify--
the ratio of mu over sigma.
But it's still something that has some meaning.
It's the normalized mean.
It really tells me what the mean is compared
to the standard deviation.
So in some models, in reality, in some real applications,
this actually might have a good meaning.
It's just telling me how big the mean
is compared to the standard fluctuations of this model.
But I won't be able to get more than that.
Agreed?
All right?
So now that we've set a parametric model,
let's try to see what our goals are going to be.
OK?
So now we have a sample and a statistical model.
And we want to estimate the parameter theta,
and I could say, well, you know what?
I don't have time for this analysis.
Collecting data is going to take me a while.
So I'm just going to mmm--
and I'm going to say that mu over sigma is 4.
And I'm just going to give it to you.
And maybe you will tell me, yeah,
it's not very good, right?
So we need some measure of performance
of a given parameter.
We need to be able to evaluate if eyeballing the problem
is worse than actually collecting
a large amount of theta.
We need to know if even if I come up with an estimator that
actually sort of uses the data, does it
make an efficient use of the data?
Would I actually need 10 times more observations
to achieve the same accuracy?
To be able to answer these questions,
well, I need to define what accuracy means.
And accuracy is something that sort of makes sense.
It says, well, I want theta.
I have to be close to theta.
And the theta is a random variable.
So I'm going to have to understand
what it means for a random variable
to be close to a deterministic number.
And so, what is a parameter estimator, right?
So I have an estimator, and I said it's a random variable.
And the formal definition--
so an estimator is a measurable function of the data.
So when I write theta hat, and that
will typically be my notation for an estimator, right?
I should really write theta hat of x1 xn.
OK?
That's what an estimator is.
If you want to know an estimator is,
this is a measurable function of the data.
And it's actually also known as a statistic.
And you know, if you're interested in,
you know, I see every day I think when I have like,
you know, a dinner with normal people.
And they say I'm a statistician.
Oh, yeah, I really like baseball.
And they talk to me about batting averages.
That's not what I do.
But for them, that's what it is, and that's
because in a way, that's what a statistic is.
A batting average is a statistic.
OK, and so here are some examples.
You can take the average xn bar.
You can take the maximum of your observation.
That's the statistics.
You can take the first one.
You can take the first one plus log of 1
plus the absolute value of the last one.
You can do whatever you want that will be an estimator.
Some of them are clearly going to be bad.
But that's still a statistic, and you can do this.
Now, when I say measurable, I always have--
so you know, graduate students sometimes
ask me like, yeah, how do I know if this estimator is measurable
or not.
And usually, my answer is, well, if I give you data,
can you compute it.
And they say, yeah, I'm like, well, then it's measurable.
That's a very good rule to check if you can actually--
if something is actually measurable.
When is this thing non-measurable?
It's when it's implicitly defined.
OK, and in particular, the things
that give you problems are--
sup or inf.
Anybody knows what a sup or an inf is?
It's like a max or a min.
But it's not always attained.
OK, so if I have x1.
So if I look at the infimum of the function
f of x for x on the real line and f of x, sorry,
let's say x on the 1 infinity.
And f of x is equal to 1 over x.
Right?
Then the infimum is the smallest value
we can take except that it doesn't really
take it at 0 right, because 1 over x is going to 0.
But it's never really getting there.
So we just called the inf 0.
But it's not the value that it ever takes.
And these things might actually be complicated to compute.
And so that's when you actually have problems, right?
When the limit is not--
you're not really quite reaching the limit.
You won't have this problem in general, but just so you know,
an estimator is not really anything.
It has to actually be measurable.
OK, so the first thing we want to know I mentioned it--
so an estimator is a statistic which does not depend on theta,
of course.
So if I give you the data, you have to be able to compute it.
And that probably should not require not knowing any known
parameters.
OK, so an estimator is said to be consistent.
When my data-- when I collect more and more data, this thing
is getting closer and closer to the true parameter.
All right?
And we said that eyeballing and saying that it's going to be 4
is not really something that's probably
going to be consistent.
But they can have things that are consistent
but that are converging to theta at different speeds.
OK?
And we know also that this is a random variable.
It converges to something.
And there might be some different notions
of convergence that kick in.
And actually there are.
And we say that it's weakly convergent if it converges
in probability and strongly convergent
if it converges almost [INAUDIBLE]..
OK?
And this is just vocabulary.
It won't make a big difference.
OK?
So we will typically say it's consistent with any of the two.
AUDIENCE: [INAUDIBLE].
PHILIPPE RIGOLLET: Well, so in parametric statistics,
it's actually a little difficult to come up with.
But in non-parametric ones, I could just say, if I had xi,
yi, and I know that yi is f of xi plus noise s1i.
And I know that f belongs to some class of function,
let's say--
[INAUDIBLE] class of smooth functions-- it's massive.
And now, I'm going to actually find the following estimator.
I'm going to take the average.
So I'm going to do least squares, right?
So I just check.
I'm trying to minimize the distance of each of my f of xi
to my yi.
And now, I want to find the smallest of them.
So if I look at the infimum here, then the question is--
so that could be--
well, that's not really an estimator for f.
But it's an estimator for the smallest possible value.
And so for example, this is actually
an estimator for the variance of sigma square.
This might not be attained, and this might not
be measurable if f is massive?
All right, so that's the infimum over some class f of x.
OK?
So those are all voice things that are defined implicitly.
If it's an average, for example, it's completely measurable.
OK?
Any other question?
OK, so we know that the first thing we might want to check,
and that's definitely something we want about estimators that
is consistent, because all consistency tells
us is that just as I collect more and more data,
my estimator is going to get closer
and closer to the parameter.
There's other things we can look at.
For each possible value of n-- now, right now,
I have a finite number of observations--
25.
And I want to know something about my estimator.
The first thing I want to check is maybe if in average, right?
So this is a random variable.
Is this random variable in average
going to be close to theta or not?
And so the difference how far I am from theta
is actually called the bias.
So the bias of an estimator is the expectation of theta hat
minus the value that I hope it gets, which is theta.
If this thing is equal to 0, we say that theta hat is unbiased.
And unbiased estimators are things that people
are looking for in general.
The problem is that there's lots of unbiased estimators.
And so it might be misleading to look for unbiasedness
when that's not really the only thing
you should be looking for.
OK, so what does it mean to be unbiased?
Maybe for this particular round of data
you collected, you're actually pretty far
from the true estimator.
But one thing that actually--
what it means is that if I redid this experiment over, and over,
and over again, and I averaged all the values of my estimators
that I got, then this would actually be the right--
the true parameter.
OK.
That's what it means.
If I were to repeat this experiment,
in average, I would actually get the right thing.
But you don't get to repeat the experiment.
OK, just a remark about estimators,
look at this estimator-- xn bar.
Right?
Think of the kiss example.
I'm looking at the average of my observations.
And I want to know what the expectation of this thing is.
OK?
Now, this guy is by linearity of the expectation,
it is this, right?
But my data is identically distributed.
So in particular, all the xi's have the same expectation,
right?
Everybody agrees with this.
When it's identically distributed,
they'll get the same expectation.
So what it means is that this guy's here--
they're all equal to the expectation of x1.
Right?
So what it means is that these guys--
I have the average of the same number.
So this is actually the expectation of x1.
OK?
And it's true.
In the kiss example, this was p.
And this is p--
the probability of turning your head right.
OK?
So those two things are the same.
In particular, that means that xn bar and just x1
have the same bias.
So that should probably illustrate to you
that bias is not something that really is telling you
the entire picture, Right?
I can take only one of my observations--
Bernoulli 0, 1.
This thing will have the same bias
as if I average 1,000 of them.
But the bias is really telling you where I am in average.
But it's really not telling me what fluctuations I'm getting.
And so if you want to start having fluctuations coming
into the picture, we actually have
to look at the risk or the quadratic risk
of the estimator.
And so the quadratic risk is the finest--
the expectation of the square distance between theta hat
and theta.
OK?
So let's look at this.
So the quadratic risk--
sometimes it's denoted that people
call it the l2 risk of theta hat, of course.
I'm sorry for maintaining such an ugly board.
[INAUDIBLE] this stuff.
OK, so I look at the square distance
between theta hat and theta.
This is still-- this is the function of a random variable.
So it's a random variable as well.
And now I'm looking at the expectation of this guy.
That's the definition.
I claimed that when this thing goes to 0, then
my estimator is actually going to be consistent.
Everybody agrees with this?
So if it goes to zero as n goes to infinity-- and here,
I don't need to tell you what kind of convergence I have,
because this is just the number, right?
It's an expectation.
So it's a regular, usual calculus-style convergence.
Then that implies that theta hat is actually weakly consistent.
What did I use to tell you this?
Yeah, this is the convergence in l2.
This actually is strictly equivalent.
This is by definition saying that theta hat converges in l2
to theta.
And we know that convergence in l2
implies convergence in credibility to theta.
That was the picture.
We're going up.
And this is actually equivalent to a consistency
by definition-- a weak consistency.
OK, so this is actually telling you a little more
because this guy here--
they are both unbiased.
Theta xn bar is unbiased.
X1 is unbiased.
But x1 is certainly not consistent ,
because the more data I collect, I'm not even doing anything
with it.
I'm just taking the first data point you're giving to me.
So they're both unbiased.
But this one is not consistent.
And this one we'll see is actually consistent.
xn bar is consistent.
And actually, we've seen that last time.
And that's because of the?
What guarantees the fact that xn bar is consistent?
AUDIENCE: The law of large numbers.
PHILIPPE RIGOLLET: The law of large numbers, right?
Actually, it's strongly consistent
if you have a strong [INAUDIBLE]..
OK, so just in the last two minutes,
I want to tell you a little bit about how this risk is linked
to see, quadratic risk is equal to bias
squared plus the variance.
So let's see what I mean by this?
So I'm going to forget about the absolute values
that you have a square.
I don't really need them.
If theta hat was unbiased, this thing
would be the expectation of theta hat.
It might not be the case.
So let me see how I can actually see-- put the bias in there.
Well, one way to do this is to see
that this is equal to the expectation of theta
hat minus the expectation of theta hat,
plus the expectation of theta hat minus theta.
OK?
I just removed the same and added the same thing.
So I didn't change anything.
Now, this guy is my bias, right?
So now let me expand the square.
So what I get is the expectation of the square of theta
hat minus its expectation.
I should put some square brackets--
plus two times the cross-product.
So the cross-product is what expectation
of theta hat minus the expectation of theta hat times
expectation of theta hat minus theta.
And then I have the last square.
Expectation of theta hat minus theta squared.
OK?
So square, cross-products, square.
Everybody is with me?
now this guy here--
if you pay attention, this thing is the expectation
of some random variable.
So it's a deterministic number.
Theta is the true parameter.
It's a deterministic number.
So what I can do is pull out this entire thing out
of the expectation like this and compute the expectation only
with respect to that part.
But what is the expectation of this thing?
It's zero, right?
The expectation of theta hat minus the expectation
of theta hat is 0.
So this entire thing is equal 0.
So now when I actually collect back my quadratic terms--
my two squared terms in this expansion--
what I get is that the expectation
of theta hat minus theta squared is
equal to the expectation of theta hat minus expectation
of theta hat squared plus the square of expectation
of theta hat minus theta.
Right?
So those are just the two--
the first and the last term of the previous equality?
Now, here I have the expectation of the square
of the difference between a random variable
and its expectation.
This is otherwise known as the variance, right?
So this is actually equal to the variance of theta hat.
And well, this was the bias.
We already said that's there.
So this whole thing is the bias square.
OK?
And hence the quadratic term is the sum
of the variance and the squared bias.
Why squared bias?
Well, because otherwise, you would add dollars
in dollars squared.
So you need to add dollars squared and dollars
squared so that this thing is actually homogeneous.
So if x is in dollars, then the bias is in dollars,
but the variance is in dollars squared.
OK, and the square here forced you to put everything
on the square scale.
All right, so what's nice is that if the quadratic risk goes
to 0, then since I have the sum of two positive terms,
both of them have to go to 0.
That means that my variance is going to 0--
very little fluctuations.
And my bias is also going to 0, which means that I'm actually
going to be on target once I reduce
my fluctuations, because it's one thing to reduce
the fluctuations.
But if I'm not on target, it's an issue, right?
For example, the estimator for the value 4 has no variance.
Every time I'm going to repeat the experiments,
I'm going to get 4, 4, 4, 4--
variance is 0.
But the bias is bad.
The bias is 4 minus theta.
And if theta is far from 4, that's not doing very well.
OK, so next week, we will--
we'll talk about what is a good estimate--
how estimators change if they have
high variance or low variance or high bias and low bias.
And we'll talk about confidence intervals as well.