The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high quality educational resources for free.
To make a donation or to view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu.
PROFESSOR: So we've been talking about this chi square test.
And the name chi square comes from the fact
that we build a test statistic that
has asymptotic distribution given
by the chi square distribution.
Let's just give it another shot.
OK.
This test.
Who has actually ever encountered the chi square test
outside of a stats classroom?
All right.
So some people have.
It's a fairly common test that you might encounter.
And it was essentially to test, if given
some data with a fixed probability mass function, so
a discrete distribution, you wanted
to test if the PMF was equal to a set value, p0,
or if it was different from p0.
And the way the chi square arose here
was by looking at Wald's test.
And essentially if you write-- so Wald's is the one that
has the chi square as the limiting distribution,
and if you invert the covariance matrix,
the asymptotic covariance matrix, so you compute
the Fisher information, which in this particular case
does not exist for the multinomial distribution,
but we found the trick on how to do this.
We remove the part that forbid it to be invertible,
then we found this chi square distribution.
In a way we have this test statistic,
which you might have learned as a black box, laundry list,
but going through the math which might have been slightly
unpleasant, I acknowledge, but really told you
why you should do this particular normalization.
So since some of you requested a little more practical examples
of how those things work, let me show you a couple.
The first one is you want to answer the question, well,
you know, when should I be born to be successful.
Some people believe in zodiac, and so Fortune magazine
actually collected the signs of 256 heads of the Fortune 500.
Those were taken randomly.
And they were collected there, and you
can see the count of number of CEOs that
have a particular zodiac sign.
And if this was completely uniformly distributed,
you should actually get a number that's
around 256 divided by 12, which in this case is 21.33.
And you can see that there is numbers
that are probably in the vicinity, but look at this guy.
Pisces, that's 29.
So who's Pisces here?
All right.
All right, so give me your information
and we'll meet again in 10 years.
And so basically you might want to test
if actually the fact that it's uniformly distributed
is a valid assumption.
Now this is clearly a random variable.
I pick a random CEO and I measure
what his zodiac sign is.
And I want to know, so it's a probability over, I don't know,
12 zodiac signs.
And I want to know if it's uniform or not.
Uniform sounds like it should be the status
quo, if you're reasonable.
And maybe there's actually something that moves away.
So we could do this, in view of these data is there evidence
that one is different.
Here is another example where you might want
to apply the chi square test.
So as I said, the benchmark distribution
was the uniform distribution for the zodiac sign,
and that's usually the one I give you.
1 over k, 1 over k, because well that's
sort of the zero, the central point for all distributions.
That's the point, the center of what we call the simplex.
But you can have another benchmark
that sort of makes sense.
So for example this is an actual dataset where 275 jurors were
identified, racial group were collected,
and you actually might want to know
if you know juries in this country
are actually representative of the actual population.
And so here of course, the population
is not uniformly distributed according to racial group.
And the way you actually do it is you
actually go on Wikipedia, for example,
and you look at the demographics of the United States,
and you find that the proportion of white is 72%, black is 7%,
Hispanic is 12, and other is about 9%.
So that's a total of 1.
And this is what we actually measured for some jurors.
So for this guy, you can actually
run the chi square test.
You have the estimated proportion, which
comes from this first line.
You have the tested proportion, p0,
that comes from the second line, and you
might want to check if those things actually
correspond to each other.
OK, so I'm not going to do it for you,
but I sort of invite you to do it
and test, and see how this compares
to the quantiles of the appropriate chi
square distribution and see what you can conclude from those two
things.
All right.
So this was the multinomial case.
So this is essentially what we did.
We computed the MLE under the right constraint,
and that was our test statistic that converges
to the chi square distribution.
So if you've seen it before, that's
all that was given to you.
Now we know why the normalization here
is p0 j and not p0 j squared or square root of p0 j, or even 1.
I mean it's not clear that this should
be the right normalization, but we
know that's what comes from taking
the right normalization, which comes from the Fisher
information.
All right?
OK.
The thing I wanted to move onto, so we've basically covered
chi square test.
Are there any questions about chi square test?
And for those of you who were not here on Thursday,
I'm really just--
do not pretend I just did it.
That's something we did last Thursday.
But are there any questions that arose
when you were reading your notes, things
that you didn't understand?
Yes.
AUDIENCE: Is there like a formal name?
Before we had talked about how what we call the Fisher
information [INAUDIBLE],, still has the same [INAUDIBLE]
because it's the same number.
PROFESSOR: So it's not the Fisher.
The Fisher information does not exist in this case.
And so there's no appropriate name for this.
It's the pseudoinverse of the asymptotic covariance matrix,
and that's what it is.
I don't know if I mentioned it last time,
but there's this entire field that uses--
you know, for people who really aspire to differential geometry
but are stuck in the stats department,
and there's this thing called information geometry, which
is essentially studying the manifolds associated
to the Fisher information metric, the metric that's
associated to Fisher information.
And so those of course can be lower dimensional manifolds,
not only distorts the geometry but forces everything
to live on a lower dimension, which
is what happens when your Fisher information does not exist.
And so there's a bunch of things that you
can study, what this manifold looks like, et cetera.
But no, there's no particular terminology here
about going here.
To be fair, within the scope of this class,
this is the only case where you--
multinomial case is the only case
where you typically see a lack of a Fisher information matrix.
And that's just because we have these extra constraints
that the sum of the parameters should be 1.
And if you have an extra constraint that
seems like it's actually remove one degree of freedom,
this will happen inevitably.
And so maybe what you can do is reparameterize.
So if I actually reparameterize everything function of p1
to p k minus 1, and then 1 minus the sum,
this would not have happened.
Because I have only a k-dimensional space.
So there's tricks around this to make it
exist if you want it to exist.
Any other question?
All right.
So let's move on to Student's t-test.
We mentioned it last time.
So essentially you've probably done it
more even in the homework than you've done it in lectures,
but just quickly this is essentially the test.
That's the test when we have an actual data that comes
from a normal distribution.
There is no Central Limit Theorem that exists.
This is really to account for the fact
that for smaller sample sizes, it
might be the case that it's not exactly true that when
I look at xn bar minus mu divided by-- so if I look
at xn bar minus mu divided by sigma times square root of n,
then this thing should have N 0, 1 distribution approximately.
Right?
By the Central Limit Theorem.
So that's for n large.
But if n is small, then it's still true
when the data is N mu, sigma squared,
then it's true that square root of n--
so here it's approximately.
And this is always true.
But I don't know sigma in practice, right?
Maybe mu, it comes from my, maybe mu comes from my mu
0, maybe something from the test statistic
where mu actually is here.
But for this guy I'm going to have inevitably
to find an estimator.
And now in this case, for small n, this is no longer true.
And what the t statistic is doing
is essentially telling you what the distribution of this guy
is.
So what you should say is that now this guy
has a t distribution with n minus 1 degrees of freedom.
That's basically the laundry list stats
that you would learn.
It says just look at a different table, that's what it is.
But we actually defined what a t distribution was.
And a t distribution is basically
something that has the same distribution as some N 0, 1,
divided by the square root of a chi square
with d degrees of freedom divided by d.
And that's a t distribution with d degrees of freedom.
And those two have to be independent.
And so what I need to check is that this guy over there
is of this form.
OK?
So let's look at the numerator.
Well, square root of n, xn bar minus mu.
What is the distribution of this thing?
Is it an N 0, 1?
AUDIENCE: N 0, sigma squared?
PROFESSOR: N 0, sigma squared, right.
So I'm not going to put it here.
So if I want this guy to be N 0, 1,
I need to divide by sigma, that's what we have over there.
So that's my N 0, 1 that's going to play the role of this guy
here.
So if I want to go a little further,
I need to just say, OK, now I need to have square root of n,
and I need to find something here
that looks like my square root of chi square divided
by-- yeah?
AUDIENCE: Really quick question.
The equals sign with the d on top, that's just defined as?
PROFESSOR: No, that's just the distribution.
So, I don't know.
AUDIENCE: Then never mind.
PROFESSOR: Let's just write it like that, if you want.
I mean, that's not really appropriate to have.
Usually you write only one distribution
on the right-hand inside of this little thing.
So not just this complicated function of distributions.
This is more like to explain.
OK, and so usually the thing you should
say that t is equal to this X divided by square root of Z
divided by d where X has normal distribution,
Z has chi square distribution with d degrees of freedom.
So what do we need here?
Well I need to have something which looks like my sigma hat,
right?
So somehow inevitably I'm going to need to have sigma hat.
Now of course I need to divide this by my sigma
so that my sigma goes away.
And so now this thing here--
sorry, I should move on to the right, OK.
And so this thing here, so sigma hat is square root of Sn.
And now I'm almost there.
So this thing is actually equal to square root of n.
But this thing here is actually not a--
so this thing here follows a distribution
which is actually a chi square, square root
of a chi square distribution divided by n.
Yeah, that's the square root chi square distribution
with n minus 1 degrees of freedom divided
by n, because sigma hat is equal to 1 over n sum
from i equal 1 to n, xi minus x bar squared.
And we just said that this part here
was a chi square distribution.
We didn't just say it, we said it a few lectures years back,
that this thing was a chi square distribution, and the fact
that the presence of this x bar here
was actually removing one degree of freedom from this sum.
OK, so this guy here has the same distribution
as a chi square n minus 1 divided by n.
So I need to actually still arrange this thing a little bit
to have a t distribution.
I should not see n here, but I should n minus 1.
The d is the same as this d here.
And so let me make the correction
so that this actually happens.
Well, if I actually write this to be equal to--
so if I write square root of n minus 1, as on the slide,
times xn bar minus mu divided by--
well let me write it as square root of Sn,
which is my sigma hat.
Then what this thing is actually equal to,
it follows a N 0, 1, divided by the square root
of my chi square distribution with n
minus 1 degrees of freedom.
And here the fact that I multiply
by square root of n minus 1, and I
have the square root of n here, is essentially the same
as dividing here by n minus 1.
And that's my tn distribution.
My t distribution with n minus 1 degrees of freedom.
Just by definition of what this thing is.
OK?
All right.
Yes?
AUDIENCE: Where'd you get the square root from?
PROFESSOR: This guy?
Oh sorry, that's sigma squared.
Thank you.
That's the estimator of the variance, not the estimator
of the standard deviation.
And when I want to divide it I divide by standard deviation.
Thank you.
Any other question or remark?
AUDIENCE: Shouldn't you divide by sigma squared?
The actual.
The estimator for the variance is
equal to sigma squared times chi square, right?
PROFESSOR: The estimator for the variance.
Oh yes, you're right.
So there's a sigma squared here.
Is that what you're asking?
AUDIENCE: Yeah.
PROFESSOR: Yes, absolutely.
And that's where, it get cancels here.
It gets canceled here.
OK?
So this is really a sigma squared times chi square.
OK.
So the fact that it's sigma squared
is just because I can pull out sigma
squared and just think those guys N 0, 1.
All right.
So that's my t distribution.
Now that I actually have a pivotal distribution, what I do
is that I form the statistic.
Here I called it Tn tilde.
OK.
And what is this thing?
I know that this has a pivotal distribution.
So for example, I know that the probability
that Tn tilde in absolute value exceeds some number that I'm
going to call q alpha over 2 for the t n minus 1,
is equal to alpha.
So that's basically, remember the t distribution
has the same shape as the Gaussian distribution.
What I'm finding is, for this t distribution,
some number q alpha over 2 of t n minus 1
and minus q alpha over 2 of t minus 1.
So those are different from the Gaussian one.
Such that the area under the curve
here is alpha over 2 on each side
so that the probability that my absolute value exceeds
this number is equal to alpha.
And that's what I'm going to use to reject the test.
So now my test becomes, for H0, say mu is equal to some mu 0,
versus H1, mu is not equal to mu 0.
The rejection region is going to be equal to the set on which
square root of n minus 1 times xn bar minus mu 0 this time,
divided by square root of Sn exceeds, in absolute value,
exceeds q-- sorry that's already here--
exceeds q alpha over 2 of t n minus 1.
So I reject when this thing increases.
The same as the Gaussian case, except that rather than reading
my quantiles from the Gaussian table
I read them from the Student table.
It's just the same thing.
So they're just going to be a little bit farther.
So this guy here is just going to be a little bigger
than the one for the Gaussian one,
because it's going to require me a little more evidence
in my data to be able to reject because I
have to account for the fluctuations of sigma hat.
So of course Student's test is used everywhere.
People use only t tests, right?
If you look at any data point, any output,
even if you had 500 observations,
if you look at the statistical software output
it's going to say t test.
And the reason why you see t test
is because somehow it's felt like it's not asymptotic.
You don't need to actually do, you
know, to be particularly careful.
And anyway, if n is equal to 500,
since the two curves are above each other
it's basically the same thing.
So it doesn't really change anything.
So why not use the t test?
So it's not asymptotic.
It doesn't require Central Limit Theorem to kick in.
And so in particular it be run if you have 15 observations.
Of course, the drawback of the Student test
is that it relies on the assumption
that the sample is Gaussian, and that's something
we really need to keep in mind.
If you have a small sample size, there is no magic going on.
It's not like Student t test allows you to get rid
of this asymptotic normality.
It sort of assumes that it's built in.
It assumes that your data has a Gaussian distribution.
So if you have 15 observations, what are you going to do?
You want to test if the mean is equal to 0 or not equal to 0,
but you have only 15 observations.
You have to somehow assume that your data is Gaussian.
But if the data is given to you, this is not math,
you actually have to check that it's Gaussian.
And so we're going to have to find
a test that, given some data, tells us whether it's Gaussian
or not.
If I have 15 observations, 8 of them
are equal to plus 1 and 7 of them are equal to minus 1,
then it's pretty unlikely that you're
going to be able to conclude that your data has a Gaussian
distribution.
However, if you see some sort of spread around some value,
you form a histogram maybe and it sort of
looks like it's a Gaussian, you might
want to say it's Gaussian.
And so how do we make this more quantitative?
Well, the sad answer to this question
is that there will be some tests that make it quantitative,
but here, if you think about it for one second, what is going
to be your null hypothesis?
Your null hypothesis, since it's one point,
it's going to be that it's Gaussian,
and then the alternative is going
to be that it's not Gaussian.
So what it means is that, for the first time
in your statistician life, you're
going to want to conclude that H0 is the true one.
You're definitely not going to want
to say that it's not Gaussian, because then everything you
know is sort of falling apart.
And so it's kind of a weird thing where
you're sort of going to be seeking tests
that have no power basically.
You're going to want to test that, and that's the nature.
The amount of alternatives, the number
of ways you can be not Gaussian, is so huge
that all tests are sort of bound to have very low power.
And so that's why people are pretty happy with the idea
that things are Gaussian, because it's
very hard to find a test that's going
to reject this hypothesis.
And so we're even going to find some tests that are visual,
where you're going to be able to say,
well, sort of looks Gaussian to me.
It allows you to deal with the borderline cases
pretty efficiently.
We'll see actually a particular example.
All right, so this theory of testing
whether data comes from a particular distribution
is called goodness of fit.
Is this distribution a good fit for my data?
That's the goodness of fit test.
We have just seen a goodness of fit test.
What was it?
Yeah.
The chi square test, right?
The case square test, we were given a candidate PMF
and we were testing if this was a good fit for our data.
That was a goodness of fit test.
So of course multinomial is one example,
but really what we have in the back of our mind is
I want to test if my data is Gaussian.
That's basically the usual thing.
And just like you always see t test as the standard output
from statistical software whether you ask for it or not,
there will be a test for normality
whether you ask it or not from any statistical software app.
All right.
So a goodness of fit test looks as follows.
There's a random variable X and you're
given i.i.d. copies of X, X1 to Xn,
they come from the same distribution.
And you're going to ask the following question: does X have
a standard normal distribution?
So for t distribution that's definitely
the kind of questions you may want to ask.
Does X have a uniform distribution on 0, 1?
That's different from the distribution 1
over k, 1 over k, it's the continuous notion
of uniformity.
And for example, you might want to test that--
so there's actually a nice exercise, which
is if you look at the p-values.
So we've defined what the p-values were.
And the p-value's a number between 0 and 1, right?
And you could actually ask yourself,
what is the distribution of the p-value under the null?
So the p-value is a random number.
It's the probability-- so the p-value-- let's look
at the following test.
H0, mu is equal to 0, versus H1, mu is not equal to 0.
And I know that the p-value is--
so I'm going to form what?
I'm going to look at Xn bar minus mu
times square root of n divided by-- let's say that we
know sigma for one second.
Then the p-value is the probability
that this is larger then square root of n little xn
bar minus mu, minus 0 actually in this case,
divided by sigma, where this guy is the observed.
OK.
So now you could say, well, how is that a random variable?
It's just a number.
It's just a probability of something.
But then I can view this as a function of this guy
here when I plug it back to be a random variable.
So what I mean by this is that if I look at this value
here, if I say that phi is the CDF of N 0, 1,
so the p-value is the probability
that it exceeds this.
So that's the probability that I'm either here or here.
AUDIENCE: [INAUDIBLE]
PROFESSOR: No, it's not, right?
AUDIENCE: [INAUDIBLE]
PROFESSOR: This is a big X and this is a small x.
This is just where you plug in your data.
The p-value is the probability that you
have more evidence against your null
than what you already have.
OK, so now I can write it in terms
of cumulative distribution functions.
So this is what?
This is phi of this guy, which is minus this thing here.
Well it's basically 2 times this guy,
phi of minus square root of n, Xn bar divided by sigma.
That's my p-value.
If you give me data, I'm going to compute the average
and plug it in there, and it can spit out the p-value.
Everybody agrees?
So now I can view this, if I start now looking back I say,
well, where does this data come from?
Well, it could be a random variable.
It came from the realization of this thing.
So I can try to, I can think of this value,
where now this is a random variable because I just plugged
in a random variable in here.
So now I view my p-value as a random variable.
So I keep switching from small x to large X. Everybody
agrees what I'm doing here?
So I just wrote it as a deterministic function
of some deterministic number, and now the function
stays deterministic but the number becomes random.
And so I can think of this as some statistic of my data.
And I could say, well, what is the distribution
of this random variable?
Now if my data is actually normally distributed,
so I'm actually under the null, so
under the null, that means that Xn bar times square root of n
divided by sigma has what distribution?
Normal?
Well it was sigma, I assume I knew it.
So it's N 0, 1, right?
I divided by sigma here.
OK?
So now I have this random variable.
And so my random variable is now 2 phi of minus absolute value
of a Gaussian.
And I'm actually interested in the distribution of this thing.
I could ask that.
Anybody has an idea of how you would
want to tackle this thing?
If I ask you, what is the distribution
of a random variable, how do you tackle this question?
There's basically two ways.
One is to try to find something that
looks like the expectation of h of x for all h.
And you try to write this using change of variables
and something that looks like integral of h of x p of x dx.
And then you say, well, that's the density.
If you can read this for any h, then that's
the way you would do it.
But there's a simpler way that does not
involve changing variables, et cetera,
you just try to compute the cumulative distribution
function.
So let's try to compute the probability
that 2 phi minus N 0, 1, is less than t.
And maybe we can find something we know.
OK.
Well that's equal to what?
That's the probability that a minus N 0,
well let's say that an N 0, 1--
sorry, N 0, 1 absolute value is greater than minus phi inverse
of t over 2.
And that's what?
Well, it's just the same thing that we had before.
It's equal to-- so if I look again,
this is the probability that I'm actually on this side
or that side of this number.
And this number is what?
It's minus phi of t over 2.
Why do I have a minus here?
That's fine, OK.
So it's actually not this, it's actually the probability
that my absolute value--
oh, because phi inverse.
OK.
Because phi inverse is--
so I'm going to look at t between 0
and-- so this number is ranging between 0 and 1.
So it means that this number is ranging between 0--
well, the probability that something is less than t
should be ranging between the numbers that this guy takes,
so that's between 0 and 2.
Because this thing takes values between 0 and 2.
I want to see 0 and 1, though.
AUDIENCE: Negative absolute value is always less
than [INAUDIBLE].
PROFESSOR: Yeah.
You're right, thank you.
So this is always some number which is less than 0,
so the probability that the Gaussian is less
than this number is always less than the probability
it's less than 0, which is 1/2, so t only
has to be between 0 and 1.
Thank you.
And so now for t between 0 and 1, then
this guy is actually becoming something which is positive,
for the same reason as before.
And so that's what?
That's just basically 2 times phi of phi inverse of t over 2.
That's just playing with the symmetry a little bit.
You can look at the areas under the curve.
And so what it means is that those two guys cancel.
This is the identity.
And so this is equal to t.
So which distribution has a density--
sorry, which distribution has a cumulative distribution
function which is equal to t for t between 0 and 1?
That's the uniform distribution, right?
So it means that this guy follows a uniform distribution
on the interval 0, 1.
And you could actually check that.
For any test you're going to come up with,
this is going to be the case.
Your p-value under the null will have a distribution
which is uniform.
So now if somebody shows up and says, here's my test,
it's awesome, it just works great.
I'm not going to explain to you how I built it,
it's a complicated statistics that
involve moments of order 27.
And I'm like, OK, you know, how am I
going to test that your test statistic actually makes sense?
Well one thing I can do is to run a bunch of data,
draw a bunch of samples, compute your test statistic,
compute the p-value, and check if my p-value has
a uniform distribution on the interval 0, 1.
But for that I need to have a test that,
given a bunch of observations, can tell me
whether they're actually distributed uniformly
on the interval 0, 1.
And again one thing I could do is build a histogram
and see if it looks like that of a uniform,
but I could also try to be slightly more quantitative
about this.
AUDIENCE: Why does the [INAUDIBLE] have
to be for a [INAUDIBLE]?
PROFESSOR: For two tests?
AUDIENCE: For each test.
Why does the p-value have to be normal?
I mean, uniform.
PROFESSOR: It's uniform under the null.
So because my test statistic was built under the null,
and so I have to be able to plug in the right value in there,
otherwise it's going to shift everything
for this particular test.
AUDIENCE: At the beginning while your probabilities
were of big Xn, that thing.
That thing is the p-value.
PROFESSOR: That's the p-value, right?
That's the definition of the p-value.
AUDIENCE: OK.
PROFESSOR: So it's the probability
that my test statistic exceeds what I've actually observed.
AUDIENCE: So how you run the test is basically
you have your observations and plug them
into the cumulative distribution function for a normal,
and then see if it falls under the given--
PROFESSOR: Yeah.
So my p-value is just this number
when I just plug in the values that I observe here.
That's one number.
For every dataset you're going to give me,
it's going to be one number.
Now what I can do is generate a bunch of datasets of size n,
like 200 of them.
And then I'm going to have a new sample
of say 200, which is just the sample of 200 p-values.
And I want to test if those p-values have
a uniform distribution.
OK?
Because that's the distribution they should be having.
All right?
OK.
This one we've already seen.
Does x have a PMF with 30%, 50%, and 20%?
That's something I could try to test.
That looks like your grade point distribution for this class.
Well not exactly, but that looks like it.
So all these things are known as goodness of fit tests.
The goodness of fit test is something
that you want to know if the data that you have at hand
follows the hypothesized distribution.
So it's not a parametric test.
It's not a test that says, is my mean equal to 25 or not.
Is my proportion of heads larger than 1/2 or not?
It's something that says, my distribution
this particular thing.
So I'm going to write them as goodness of fit, G-O-F here.
You don't need to have parametric modeling to do that.
So how do I work?
So if I don't have any parametric modeling,
I need to have something which is somewhat non-parametric,
something that goes beyond computing the mean
and the standard deviation, something
that computes some intrinsic non-parametric aspect
of my data.
And just like here we made this computation, what we did
is we said well, if I actually check
that the CDF of my data, that my p-value is uniform,
then I know it's uniform.
So it means that the cumulative distribution function
has an intrinsic value about it that captures
the entire distribution.
Everything I need to know about my distribution
is captured by the cumulative distribution function.
Now I have an empirical way of computing,
I have a data-driven way of computing
an estimate for the cumulative distribution function, which
is using the old statistical trick which
consists of replacing expectations by averages.
So as I said, the cumulative distribution function
for any distribution, for any random variable, is--
so F of t is the probability that X
is less than or equal to t, which
is equal to the expectation of the indicator
that X is less than or equal to t.
That's the definition of a probability.
And so here I'm just going to replace expectation
by the average.
That's my usual statistical trick.
And so my estimator Fn for--
the distribution is going to be 1 over n sum from i
equal 1 to n of these indicators.
And this is called the empirical CDF.
It's just the data version of the CDF.
So I just replaced this expectation here by an average.
Now when I sum indicators, I'm actually
counting the number of them that satisfy something.
So if you look at what this guy is,
this is the number of X i's that is less than t, right?
And so if I divide by n, it's the proportion of observations
I have that are less than t.
That's what the empirical distribution is.
That's what's written here, the number of data points
that are less than t.
And so this is going to be something
that's sort of trying to estimate one or the other.
And the law of large number actually
tells me that for any given t, if n is large enough, Fn of t
should be close to F of t.
Because it's an average.
And this entire thing, this entire statistical trick,
which consists of replacing expectations by averages,
is justified by the law of large number.
Every time we used it, that was because the law of large number
sort of guaranteed to us that the average was
close to the expectation.
OK.
So law of large numbers tell me that Fn of t converges,
so that's the strong law, says that almost surely actually
Fn of t goes to F of t.
And that's just for any given t.
Is there any question about this?
That averages converge to expectation,
that's the law of large number.
And almost surely we could say in probability
it's the same, that would be the weak law of large number.
Now this is fine.
For any given t, the average converges to the true.
It just happens that this random variable is indexed by t,
and I could do it for t equals 1 or 2 or 25,
and just check it again.
But I might want to check it for all t's at once.
And that's actually a different result.
That's called a uniform result. I
want this to hold for all t at the same time.
And it may be the case that it works for each t individually
but not for all t's at the same time.
What could happen is that for t equals 1
it converges at a certain rate, and for t equals 2
it converges at a bit of a slower rate,
and for t equals 3 at a slower rate and slower rate.
And so as t goes to infinity, the rate is going to vanish
and nothing is going to converge.
That could happen.
I could make this happen at a finite point.
There's many ways where it could make this happen.
Let's see how that could work.
I could say, well, actually no.
I still need to have this at infinity for some reason.
It turns out that this is still true uniformly,
and this is actually a much more complicated result
than the law of large number.
It's called Glivenko-Cantelli Theorem.
And the Glivenko-Cantelli Theorem
tells me that, for all t's at once, Fn converges to F.
So let me just show you quickly why
this is just a little bit stronger than the one
that we had.
If sup is confusing you, think of max.
It's just the max over an infinite set.
And so what we know is that Fn of t goes to F of t
as n goes to infinity.
And that's almost surely.
And that's the law of large numbers.
Which is equivalent to saying that Fn of t minus F of t as n
goes to infinity converges almost surely to 0, right?
This is the same thing.
Now I want this to happen for all t's at once.
So what I'm going to do-- oh, and this is actually
equivalent to this.
And so what I'm going to do is I'm going
to make it a little stronger.
So here the arrow only goes one way.
And this is where the sup for t in R of Fn of t.
And you could actually show that this happens also
almost surely.
Now maybe almost surely is a bit more
difficult to get a grasp on.
Does anybody want to see, like why this statement for this sup
is strictly stronger than the one that holds individually
for all t's?
You want to see that?
OK, so let's do that.
So forget about it almost surely for one second.
Let's just do it in probability.
The fact that Fn of t converges to F of t for all t,
in probability means that this goes to 0 as n goes
to infinity for any epsilon.
For any epsilon in t we know we have this.
That's the convergence in probability.
Now what I want is to put a sup here.
The probability that the sup is lower than epsilon,
might be actually always larger than, never go to 0
in some cases.
It could be the case that for each given t,
I can make n large enough so that this probability becomes
small.
But then maybe it's an n of t.
So this here means that for any--
maybe I shouldn't put, let me put a delta here.
So for any epsilon, for any t and for any epsilon,
there exists n, which could depend on both epsilon
and t, such that the probability that Fn t
minus F of t exceeding delta is less than epsilon t.
There exists an n and a delta.
No, that's for all delta, sorry.
So this is true.
That's what this limit statement actually means.
But it could be the case that now when I take the sup over t,
maybe that n of t is something that looks like t.
Or maybe, well, integer part of t.
It could be, right?
I don't say anything.
It's just an n that depends on t.
So if this n is just t, maybe t over epsilon,
because I want epsilon.
Something like this.
Well that means that if I want this
to hold for all t's at once, I'm going
to have to go for the n that works for all t's at once.
But there's no such n that works for all t's at once.
The only n that works is infinity.
And so I cannot make this happen for all of them.
What Glivenko-Cantelli tells you,
it's actually this is not something that holds like this.
That the n that depends on t, there's actually one largest n
that works for all the t's at once, and that's it.
OK.
So just so you know why this is actually a stronger statement,
and that's basically how it works.
Any other question?
Yeah.
AUDIENCE: So what's the position for this
to have, because the random variable have
a finite mean, finite variance?
PROFESSOR: No.
Well the random variable does have finite mean
and finite variance, because the random variable
is an indicator.
So it has everything you want.
This is one of the nicest random variables,
this is a Bernoulli random variable.
So here when I say law of large number, that this holds.
Where did I write this?
I think I erased it.
Yeah, the one over there.
This is actually the law of large numbers
for Bernoulli random variables.
They have everything you want.
They're bounded.
Yes.
AUDIENCE: So I'm having trouble understanding
the first statement.
So it says, for all epsilon and all t,
the probability of that--
PROFESSOR: So you mean this one?
AUDIENCE: Yeah.
PROFESSOR: For all epsilon and all t.
So you fix them now.
Then the probability that, sorry, that was delta.
I changed this epsilon to delta at some point.
AUDIENCE: And then what's the second line?
PROFESSOR: Oh, so then the second line says that,
so I'm just rewriting in terms of epsilon delta
what this n goes to infinity means.
So it means that for any a t and delta,
so that's the same as this guy here,
then here I'm just going back to rewriting this.
It says that for any epsilon there exists an n large
enough such that, well, n larger than this thing
basically, such that this thing is less than epsilon.
So Glivenko-Cantelli tells us that not only is this thing
a good idea pointwise, but it's also a good idea uniformly.
And all it's saying is if you actually
were happy with just this result, you should
be even happier with that result.
And both of those results only tell you one thing.
They're just telling you that the empirical CDF
is a good estimator of the CDF.
Now since those indicators are Bernoulli distributions,
I can actually do even more.
So let me get this guy here.
OK so, those guys, Fn of t, this guy
is a Bernoulli distribution.
What is the parameter of this Bernoulli distribution?
What is the probability that it takes value 1?
AUDIENCE: F of t.
PROFESSOR: F of t, right?
It's just the probability that this thing happens,
which is F of t.
So in particular the variance of this guy
is the variance of this Bernoulli.
So it's F of t 1 minus F of t.
And I can use that in my Central Limit Theorem.
And Central Limit Theorem is just
going to tell me that if I look at the average
of random variables, I remove their mean,
so I look at square root of n Fn of t,
which I could really write as xn bar, right?
That's really just an xn bar.
Minus the expectation, which is F
of t, that comes from this guy.
Now if I divide by square root of the variance, that's
my square root p1 minus p.
Then this guy, by the Central Limit Theorem,
goes to some N 0, 1.
Which is the same thing as you see there,
except that the variance was put on the other side.
OK.
Do I have the same thing uniformly in t?
Can I write something that holds uniformly in t?
Well, if you think about it for one second
it's unlikely it's going to go too well.
In the sense that it's unlikely that the supremum
of those random variables over t is going to also be a Gaussian.
And the reason is that, well actually the reason
is that this thing is actually a stochastic process indexed
by t.
A stochastic process is just a sequence in random variables
that's indexed by, let's say time.
The one that's the most famous is Brownian motion,
and it's basically a bunch of Gaussian increments.
So when you go from t to just t a little after that,
you have add some Gaussian into the thing.
And here it's basically the same thing that's happening.
And you would sort of expect, since each of this guy
is Gaussian, you would expect to see
something that looks like a Brownian motion at the end.
But it's not exactly a Brownian motion,
it's something that's called the Brownian bridge.
So if you've seen the Brownian motion, if I make
it start at 0 for example, so this is the value
of my Brownian motion.
Let's write it.
So this is one path, one realization of Brownian motion.
Let's call it w of t as t increases.
So let's say it starts at 0 and looks like something like this.
So that's what Brownian motion looks like.
It's just something that's pretty nasty.
I mean it looks pretty nasty, it's not continuous et cetera,
but it's actually very benign in some average way.
So Brownian motion is just something,
you should view this as if I sum some random variable that
are Gaussian, and then I look at this from farther and farther,
it's going to look like this.
And so here I cannot have a Brownian motion in the n,
because what is the variance of Fn of t minus F of t at t is
equal to 1?
Sorry, at t is equal to infinity.
AUDIENCE: 0.
PROFESSOR: It's 0, right?
The variance goes from 0 at t is negative infinity,
because at negative infinity F of t is going to 0.
And as t goes to plus infinity, F of t
is going to 1, which means that the variance of this guy as t
goes from negative infinity to plus infinity
is pinned to be 0 on each side.
And so my Brownian motion cannot,
when I describe a Brownian motion I'm just adding more
and more entropy to the thing and it's going all over
the place, but here what I want is that as I go back it should
go back to essentially 0.
It should be pinned down to a specific value at the n.
And that's actually called the Brownian bridge.
It's a Brownian motion that's conditioned
to come back to where it started essentially.
Now you don't need to understand Brownian bridges to understand
what I'm going to be telling you.
The only thing I want to communicate to you
is that this guy here, when I say a Brownian bridge,
I can go to any probabilist and they can tell you
all the probability properties of this stochastic process.
It can tell me the probability that it
takes any value at any point.
In particular, it can tell me--
the supremum between 0 and 1 of this guy,
it could tell me what the cumulative distribution
function of this thing is, can tell me
what the density of this thing is, can tell me everything.
So it means that if I want to compute probabilities
on this object here, which is the maximum value that this guy
can take over a certain period of time, which is basically
this random variable.
So if I look at the value here, it's
a random variable that fluctuates.
It can tell me where it is with hyperability, can tell me
the quantiles of this thing, which is useful
because I can build a table and use it to compute my quantiles
and form tests from it.
So that's what actually is quite nice.
It says that if I look at the square root of n Fn
hat minus sup over t, I get something
that looks like the sup of these Gaussians,
but it's not really sup of Gaussian,
it's sup of a Brownian motion.
Now there's something you should be very careful here.
I cheated a little bit.
I mean, I didn't cheat, I can do whatever I want.
But my notation might be a little confusing.
Everybody sees that this t here is not the same as this t here?
Can somebody see that?
Just because, first of all, this guy's between 0 and 1.
And this guy is in all of R.
What is this t here?
As a function of this t here?
This guy is F of this guy.
So really, if I want it to be completely transparent
and not save the keys of my keyboard,
I would read this as sup over t of Fn t minus F of t
goes to N distribution as n goes to infinity.
The supremum over t, again in R, so this guy is
for t in the entire real line, this guy
is for t in the entire real line.
But now I should write b of what?
F of t, exactly.
So really the t here is F of the original one.
And so that's a Brownian bridge, where
when t goes to infinity the Brownian bridge
goes from 0 to 1 and it looks like this.
A Brownian bridge at 0 is 0, at 1 it's 0.
And it does this.
But it doesn't stray too far because I condition
it to come back to this point.
That's what a Brownian bridge is.
OK.
So in particular, I can find a distribution for this guy.
And I can use this to build a test which is called
the Kolmogorov-Smirnov test.
The idea is the following.
It says, if I want to test some distribution
F0, some distribution that has a particular CDF F0,
and I plug it in under the null, then
this guy should have pretty much the same distribution
as the supremum of Brownian bridge.
And so if I see this to be much larger than it should
be when it's the supremum of a Brownian bridge,
I'm actually going to reject my hypothesis.
So here's the test.
I want to test whether H0, F is equal to F0,
and you will see that most of the goodness of fit tests
are formulated mathematically in terms
of the cumulative distribution function.
I could formulate them in terms of personality density
function, or just write x follows N 0, 1,
but that's the way we write it.
We formulate them in terms of cumulative distribution
function because that's what we have
a handle on through the empirical cumulative
distribution function.
And then it's versus H1, F is not equal to F0.
So now I have my empirical CDF.
And I hope that for all t's, Fn of t
should be close to F0 of t.
Let me write it like this.
I put it on the exponent because otherwise that
would be the empirical distribution function based
on zero observations.
Now I form the following test statistic.
So my test statistic is tn, which
is the supremum over t in the real line of square root
of n Fn of t minus F of t, sorry, F0 of t.
So I can compute everything.
I know this from the data, and this
is the one that comes from my null hypothesis.
As I can compute this thing.
And I know that if this is true, this
should actually be the supremum of a Brownian bridge.
Pretty much.
And so the Kolmogorov-Smirnov test is simply,
reject if this guy, tn, in absolute value,
no actually not in absolute value.
This is just already absolute valued.
Then this guy should be what?
It should be larger than the q alpha over 2 distribution
that I have.
But now rather than putting N 0, 1, or Tn,
this is here whatever notation I have for supremum
of Brownian bridge.
Just like I did for any pivotal distribution.
That was the same recipe every single time.
I formed the test statistic such that
the asymptotic distribution did not depend on anything I know,
and then I would just reject when this pivotal distribution
was larger than something.
Yes?
AUDIENCE: I'm not really sure why Brownian bridge appears.
PROFESSOR: Do you know what a Brownian bridge is, or?
AUDIENCE: Only vaguely.
PROFESSOR: OK.
So this thing here, think of it as being a Gaussian.
So for all t you have a Gaussian distribution.
Now a Brownian motion, so if I had a Brownian motion
I need to tell you what the--
so it's basically a Brownian motion
is something that looks like this.
It's some random variable that's indexed by t.
I want, say, the expectation of Xt could be equal to 0
for all t.
And what I want is that the increments
have a certain distribution.
So what I want is that the expectation of Xt minus Xs
follows some distribution which is N 0, t minus s.
So the increments are bigger as I go farther,
in terms of variability.
And I also want some covariance structure between the two.
So what I want is that the covariance between Xs and Xt
is actually equal to the minimum of s and t.
Yeah, maybe.
Yeah, that should be there.
So this is, you open a probability book, that's
what it's going to look like.
So in particular, you can see, if I put 0 here
and X0 is equal to 0, it has 0 variance.
So in particular, it means that Xt,
if I look only at the t-th one, it
has some normal distribution with variance t.
So this is something that just blows up.
So this guy here looks like it's going
to be a Brownian motion because when
I look at the left-hand side it has a normal distribution.
Now there's a bunch of other things you need to check.
It's the fact that you have this covariance, for example,
which I did not tell you.
But it sure look somewhat like that.
And in particular, when I look at the normal with mean 0
and variance here, then it's clear
that this guy does not have a variance that's
going to go to infinity just like the variance of this guy.
We know that the variance is forced to be back to 0.
And so in particular we have something
that has mean 0 always, whose variance has to be 0 at 0,
and variance-- sorry, at t equals negative infinity,
and variance 1 at t equals plus infinity.
So a variance 0 at t equals plus infinity,
and so I have to basically force it to be equal to 0 at each n.
So the Brownian motion here tends
to just go to infinity somewhere,
whereas this guy forces it to come back.
Now everything I described to you
is on the scale negative infinity to plus infinity,
but since everything depends on F of t,
I can actually just put that back
into a scale, which is 0 and 1 by a simple change of variable.
It's called change of time for the Brownian motion.
OK?
Yeah.
AUDIENCE: So does a Brownian bridge
have a variance at each point that's proportional?
Like it starts at 0 variance and then
goes to 1/4 variance in the middle
and then goes back to 0 variance?
Like in the same parabolic shape?
PROFESSOR: Yeah.
I mean, definitely.
I mean by symmetry you can probably infer all the things.
AUDIENCE: Well I can imagine Brownian bridge
with a variance that starts at 0 and stays, like,
the shape of the variance as you move along.
PROFESSOR: Yeah, so I don't know if-- there
is an explicit formula for this, and it's simple.
That's what I can tell you, but I don't know what the explicit,
off the top of my head what the explicit formula is.
AUDIENCE: But would it have to match this F
of t 1 minus F of t structure?
Or not?
PROFESSOR: Yeah.
AUDIENCE: Or does the fact that we're taking the supremum--
PROFESSOR: No.
Well the Brownian bridge, this is the supremum-- you're right.
So this will be this form for the variance for sure,
because this is only marginal distributions that
don't take-- right, the process is not just
what is the distribution at each instant t.
It's also how do those distributions interact
with each other in terms of covariance.
For the marginal distributions at each instance t,
you're right, the variance is F of t 1 minus F of t.
We're not going to escape that.
But then the covariance structure between those guys
is a little more complicated.
But yes, you're right.
For marginal that's enough.
Yeah?
AUDIENCE: So the supremum of the Brownian bridge
is a number between 0 and 10, let's just say.
PROFESSOR: Yeah, it could be infinity.
AUDIENCE: So it's not symmetrical with respect to 0,
so why are we doing all over 2?
PROFESSOR: OK.
Did say raise it?
Yeah.
Because here I didn't say the supremum of the absolute value
of a Brownian bridge, I just said the supremum
of a Brownian bridge.
But you're right, let's just do this like that.
And then it's probably cleaner.
So yeah, actually well it should be q alpha.
So this is basically, you're right.
So think of it as being one-sided.
And there's actually no symmetry for the supremum.
I mean the supremum is not symmetric around 0,
so you're right.
I should not use alpha over 2, thank you.
Any other question?
This should be alpha.
Yeah.
I mean those slides were written with 1 minus alpha
and I have not replaced all instances of 1 minus alpha
by alpha.
I mean, except this guy, tilde.
Well, depends on how you want to call it.
But this is still, the probability that Z exceeds
this guy should be alpha.
OK?
And this can be found in tables.
And we can compute the p-value just like we did before.
But we have to simulate it because it's not
going to depend on the cumulative distribution
function of a Gaussian, like it did for the usual Gaussian
test.
That's something that's more complicated,
and typically you don't even try.
You get the statistical software to do it for you.
So just let me skip a few lines.
This is what the table looks like for the Kolmogorov-Smirnov
test.
So it just tells you, what is your number of observations, n.
Then you want alpha to be equal to 5%, say.
Let's say you have nine observations.
So if square root of n absolute value of Fn of t minus F of t
exceeds this thing, you reject.
Well it's pretty clear from this test
is that it looks very nice, and I tell
you this is how you build it.
But if you think about it for one second,
it's actually really an annoying thing
to build because you have to take the supremum over t.
This depends on computing a supremum, which in practice
might be super cumbersome.
I don't want to have to compute this for all values t
and then to take the maximum of those guys.
It turns out that that's actually quite nice that we
don't have to actually do this.
What does the empirical distribution function
look like?
Well, this thing, remember Fn of t by definition was--
so let me go to the slide that's relevant.
So Fn of t looks like this.
So what it means is that when t is between two observations,
then this guy is actually keeping the same value.
So if I put my observations on the real line here.
So let's say I have one observation here,
one observation here,
and one observation here,
for simplicity.
Then this guy is basically, up to this normalization,
counting how many observations they have that are less than t.
So since I normalize by n, I know that the smallest number
here is going to be 0, and the largest number here
is going to be 1.
So let's say this looks like this.
This is the value 1.
At the value, since I take it less than or equal to,
when I'm at Xi, I'm actually counting it.
So the jump happens at Xi.
So that's the first observation, and then I jump.
By how much do I jump?
Yeah?
One over n, right?
And then this value belongs to the right.
And then I do it again.
I know it's not going to work out for me, but we'll see.
Oh no actually, I did pretty well.
This is what my cumulative distribution looks like.
Now if you look on this slide, there
is this weird notation where I start putting now
my indices in parentheses.
X parenthesis 1, X parenthesis 2, et cetera.
Those are called the ordered statistic.
It's just because it might be, when my data is given
to me I just call the first observation,
the one that's on top of the table,
but it doesn't have to be the smallest value.
So it might be that this is X1 and that this is X2,
and then this is X3, X4, and X5.
These might be my observations.
So what I do is that I call them in such a way
that this is actually, I recall this guy X1,
which is just really X3.
This is X2, X3, X4, and X5.
These are my reordered observations
in such a way that the smallest one is indexed by one
and the largest one is indexed by n.
So now this is actually quite nice,
because what I'm trying to do is to find the largest
deviation from this guy to the true cumulative distribution
function.
The true cumulative distribution function,
let's say it's Gaussian, looks like this.
It's something continuous, for a symmetric distribution
it crosses this axis at 1/2, and that's what it looks like.
And the Kolmogorov-Smirnov test is just
telling me how far do those two curves get
in the worst possible case?
So in particular here, where are they the farthest?
Clearly that's this point.
And so up to rescaling, this is the value
I'm going to be interested in.
That's how they get as far as possible from each other.
Here, something just happened, right?
The farthest distance that I got was exactly
at one of those dots.
It turns out this is enough to look at those dots.
And the reason is, well because after this dot
and until the next jump, this guy does not change,
but this guy increases.
And so the only point where they can be the farthest apart
is either to the left of a jump or to the right of a jump.
That's the only place where they can be far from each other.
And that means that only one observation.
Everybody sees that?
The farthest points, the points at which those two curves are
the farthest from each other, has
to be at one of the observations.
And so rather than looking at a sup over all possible t's,
really all I need to do is to look at a maximum
only at my observations.
I just need to check at each of those points
whether they're far.
Now here, notice that you did not,
this is not written Fn of Xi.
The reason is because I actually know what Fn of Xi is.
Fn of the i-th order observation is just
the number of jumps I've had until this observation.
So here, I know that the value of Fn is 1 over n,
here it's 2 over n, 3 over n, 4 over n, 5 over n.
So I knew that the values of Fn at my observations,
and those are actually the only values that Fn can take,
are an integer divided by n.
And that's why you see i minus 1 over n, or i over n.
This is the difference just before the jump,
and this is the difference at the jump.
So here the key message is that this is no longer
a supremum over all t's, but it's just
the maximum from 1 to n.
So I really have only two n values to compute.
This value and this value for each observation, that's 2n
total.
I look at the maximum and that's actually the value.
And it's actually equal to tn.
It's not an approximation.
Those things are equal.
That's just the only places where
those guys can be maximum.
Yes?
AUDIENCE: It seems like since the null hypothesis [INAUDIBLE]
the entire distribution of theta,
this is like strictly more powerful than just
doing it [INAUDIBLE].
PROFESSOR: It's strictly less powerful.
AUDIENCE: Strictly less powerful.
But is there, is that like a big trade-off
that we're making when we do that?
Obviously we're not certain in the first place
that we want to assume normality.
Does it make sense to [INAUDIBLE],,
the Gaussian [INAUDIBLE].
PROFESSOR: So can you, I'm not sure what
question you're asking.
AUDIENCE: So when we're doing a normal test,
we're just asking questions about the mus,
the means of our distribution.
[INAUDIBLE] This one, it seems like it
would be both at the same time.
[INAUDIBLE] Is this decreasing power [INAUDIBLE]??
PROFESSOR: So remember, here in this test
we want to conclude to H0, in the other test we typically
want to conclude to H1.
So here we actually don't want power, in a way.
And you have to also assume that doing a test on the mean
is probably not the only thing you're
going to end up doing on your data
after you actually establish that it's normally distributed.
Then you have the dataset, you've sort of
established it's normally distributed,
and then you can just run the arsenal of statistical studies.
And we're going to see regression
and all sorts of predictive things, which are not just
tests if the mean is equal to something.
Maybe you want to build a confidence interval
for the mean.
Then this is not, confidence interval is not a test.
So you're going to have to first test if it's normal,
and then see if you can actually use
the quantiles of a Gaussian distribution or a t
distribution to build this confidence interval.
So in a way you should do this as like, the flat fee
to enter the Gaussian world, and then you
can do whatever you want to do in the Gaussian world.
We'll see actually that your question goes back
to something that's a little important, is here
I said F0 is fully specified.
It's like an N 1, 5.
But I didn't say, is it normally distributed,
which is the question that everybody asks.
You're not asking, is it this particular normal distribution
with this particular mean and this particular variance.
So how would you do it in practice?
Well you would say, I'm just going
to replace the mean by the empirical mean and the variance
by the empirical variance.
But by doing that you're making a huge mistake because you
are sort of depriving your test of the possibility
to reject the Gaussian hypothesis just
based on the fact that the mean is wrong or the variance
is wrong.
You've already stuck to your data pretty well.
And so you're sort of like already
tilting the game in favor of H0 big time.
So there's actually a way to arrange for this.
OK, so this is about pivotal statistic.
We've used this word many times.
And So that's how.
I'm not going to go into this test.
It's really, this is a recipe on how you would actually
build the table that I showed you, this table.
This is basically the recipe on how to build it.
There's another recipe to build it, which is just
open a book at this page.
That's a little faster.
Or use software.
I just wanted to show you.
So let's just keep in mind, anybody has a good memory?
Let's just keep in mind this number.
This is the threshold for the Kolmogorov-Smirnov statistic.
If I have 10 observations and I want to do it at 5%,
it's about 41%.
So that's the number that it should be larger from.
So it turns out that if you want to test if it's normal, and not
just the specific normal, this number
is going to be different.
Do you think the number I'm going
to read in a table that's appropriate for this is
going to be larger or smaller?
Who says larger?
AUDIENCE: Sorry, what was the question?
PROFESSOR: So the question is, this
is the number I should see if my test was, is X, say, N 0, 5.
Right?
That's a specific distribution with a specific F0.
So that's the number, I would build
the Kolmogorov-Smirnov statistic from this.
I would perform a test and check if my Kolmogorov-Smirnov
statistic tn is larger than this number or not.
If it's larger I'm going to reject.
Now I say, actually, I don't want to test if H0 is N 0, 5,
but it's just a mu sigma squared for some mu and sigma squared.
And in particular I'm just going to plugin mu hat and sigma
hat into my F0, run the same statistic,
but compare it to a different number.
So the larger the number, the more or less
likely am I to reject?
The less likely I am to reject, right?
So if I just use that number, let's say
this is a large number, I would be more
tempted to say it's Gaussian.
And if you look at the table you would
get that if you make the appropriate correction
at the same number of observations, 10,
and the same level, you get 25% as opposed to 41%.
That means that you're actually much more likely if you
use the appropriate test to reject the fact that it's
normal, which is bad news, because that means
you don't have access to the Gaussian arsenal,
and nobody wants to do this.
So actually this is a mistake that people do a lot.
They use the Kolmogorov-Smirnov test
to test for normality without adjusting for the fact
that they've plugged in the estimated mean
and the estimated variance.
This leads to rejecting less often, right?
I mean this is almost half of the number that we had.
And then they can be happy and walk home
and say, well, I did the test and it was normal.
So this is actually a mistake that I
believe that genuinely at least a quarter of the people
do make in purpose.
They just say, well I want it to be Gaussian so I'm just
going to make my life easier.
So this is the so-called Kolmogorov Lilliefors test.
We'll talk about it, well not today for sure.
There's other statistics that you can test, that you can use.
And the idea is to say, well, we want
to know if the empirical distribution
function, the empirical CDF, is close to the true CDF.
The way we did it is by forming the difference
in looking at the worst possible distance they can be.
That's called a sup norm, or L infinity norm,
in functional analysis.
So here, this is what it looked like.
The distance between Fn and F that we measured was just
the supremum distance over all t's.
That's one way to measure distance between two functions.
But there's an infinite many ways
to measure distance between functions.
One is something we're much more familiar with,
which is the squared L2-norm.
This is nice because this has like an inner product,
it has some nice properties.
And you could actually just, rather than taking the sup,
you could just integrate the squared distance.
And this is what leads to Cramier-Von Mises test.
And then there's another one that
says, well, maybe I don't want to integrate without weights.
Maybe I want to put weights that account for the variance.
And this guy is called Anderson-Darling.
For each of these tests you can check
that the asymptotic distribution is going to be pivotal,
which means that there will be a table at the back of some book
that tells you what the statistic, the quantiles
of square root of n times this guy
are asymptotically, basically.
Yeah?
AUDIENCE: For the Kolmogorov-Smirnov test,
for the table that shows the value it has,
it has the value for different n.
But I thought we [INAUDIBLE]--
PROFESSOR: Yeah.
So that's just to show you that asymptotically it's pivotal,
and I can point you to one specific thing.
But it turns out that this thing is actually pivotal for each n.
And that's why you have this recipe to construct the entire
thing, because it's actually not true for all possible n's.
Also there's the n that shows up here.
So no actually, this is something
you should have in mind.
So basically, let me strike what I just said.
This thing you can actually, this distribution
will not depend on F0 for any particular n.
It's just not going to be a Brownian bridge
but a finite sample approximation of a Brownian
bridge, and you can simulate that just drawing samples
from it, building a histogram, and constructing
the quantiles for this guy.
AUDIENCE: No one has actually developed
a table for Brownian--
PROFESSOR: Oh, there is one.
That's the table, maybe.
Let's see if we see it at the bottom of the other table.
Yeah.
See?
Over 40, over 30.
So this is not the Kolmogorov-Smirnov,
but that's the Kolmogorov Lilliefors.
Those numbers that you see here, they
are the numbers for the asymptotic thing which is
some sort of Brownian bridge.
Yeah?
AUDIENCE: Two questions.
If I want to build the Kolmogorov-Smirnov test,
it says that F0 is required to be continuous.
PROFESSOR: Yeah.
AUDIENCE: [INAUDIBLE] If we have, like, probability
mass of a particular value.
Like some sort of data.
PROFESSOR: So then you won't have this nice picture, right?
This can happen at any point because you're
going to have discontinuities in F
and those things can happen everywhere.
And then--
AUDIENCE: Would the supremum still work?
PROFESSOR: You mean the Brownian bridge?
AUDIENCE: Yeah.
The Kolmogorov test doesn't say that you
have to be able to easily calculate the supremum.
PROFESSOR: No, no, no, but you still need it.
You still need it for--
so there's some finite sample versions of it
that you can use that are slightly more conservative,
which is in a way good news because you're
going to conclude more to H0.
And there's are some, I forget the name,
it's Kiefer-Wolfowitz, the Kiefer-Dvoretzky-Wolfowitz,
an equality which is basically like Hoeffding's inequality.
So it's basically up to bad constants
telling you the same result as the Brownian bridge result,
and those are true all the time.
But for the exact asymptotic distribution,
you need continuity.
Yes.
AUDIENCE: So just a clarification.
So when we are testing the Kolmogorov,
we shouldn't test a particular mu and sigma squared?
PROFESSOR: Well if you know what they are you can use
Kolmogorov-Smirnov, but if you don't know what they are
you're going to plug in--
as soon as you're going to estimate
the mean and the variance from the data,
you should use the one we'll see next time, which is
called Kolmogorov Lilliefors.
You don't have to think about it too much.
We'll talk about it on Thursday.
Any other question?
So we're out of time.
So I think we should stop here, and we'll resume on Thursday.