The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare continue to
offer high quality educational resources for free.
To make a donation or view additional materials from
hundreds of MIT courses, visit MIT OpenCourseWare at
ocw.mit.edu.
PROFESSOR: OK, today we are moving on with renewal
processes, and we're going to finish up talking a little
more about residual life, and we're going to generalize it
to time averages for arbitrary renewal processes.
And then finally we're going into the topic of stopping
trials, stopping times, optional stopping times--
people call them different things--
and Wald's equality.
This is, for some reason or other, normally a very tricky
thing to understand.
I think finally after many years I understand it.
And I hope that I can make it easy for you to understand it,
so some of the confusions that often come in won't come in.
And then we're going to end up talking about stopping when
you're ahead, which is a gambling strategy which says
if you're playing a fair game you play until you're $1.00
ahead, and then you stop.
And you show that with probability 1 you will
eventually become $1.00 ahead, so you have a way to beat the
bank, or beat the casino.
But of course casinos don't give you even odds, but you
have a way to win any time you get even odds.
We'll find that there's something wrong with that, but
it is an interesting topic which illustrates Wald's
equality in an interesting way.
So let's start out by reviewing a little bit.
We first talked last time about convergence with
probability 1, and we weren't talking about the strong law
then, we were just talking about a sequence of random
variables and what it means for them to converge with
probability 1.
And the theorem is that if you have a sequence of random
variables, z sub n, they converge to some number alpha.
This is slightly different than the theorem we stated
last time, but it's trivially the same.
In other words, it's the probability of the set of
sample points.
A sample point, now, is something which runs from time
0 to time infinity.
It's something that covers the whole process, covers anything
else you might want to talk about also.
It's what you do when you go from a real situation to a
model, and mathematically when you go to a model you put in
everything you're going to be
interested in at the beginning.
So, it converges to some alpha probability 1 if the set of
sequences for which the statement holds has
probability 1, and this statement is that the sample
paths converge to alpha.
The set of sample paths has converged to alpha as
probability 1.
So, it's a statement not like most of the statements in
probability where you start out talking about finite
length, talking about probabilities, and
expectation, and so forth.
And then you go to the limit with these things you're used
to talking about, which are easier to deal with.
Here it's a statement about sample paths
over the whole sequence.
You break them into two categories.
Implicitly we have shown that when you break them into two
categories those two categories in fact have
probabilities.
They are events, and one of them has probability 1.
For renewal processes, if we have a renewal process we can
use this theorem to talk about the sample averages.
In other words, in a renewal process we have a sequence of
IID random variables, x1, x2, x3, and so forth.
We have the sample sums, the sum of the time 1, the sum of
the time 2, sum of the time 3, and so forth.
And the strong law of large numbers, which comes pretty
much directly out of this convergence of probability 1
theorem, says that the probability is the set of
sample points for which the limit of the sample average,
namely sn of omega over n.
You take the sample average over all lengths for one
sample point.
That goes to a limit, that's this convergence with
probability 1, the probability that you get convergence is
equal to 1, which is what with probability 1 means.
Then we use the theorem on the top.
Let's reset a little bit.
This theorem on the top is more than just a theorem about
convergence with probability 1.
It's the reason why convergence with probability 1
is so useful.
It says if you have some function f of x, that function
is continuous at this point alpha.
You have this limit here, then the probability of the set of
omega for which the function applied to each of these
sample points as equal to alpha is also one.
AUDIENCE: [INAUDIBLE].
PROFESSOR: It should be f of alpha.
Yes, absolutely.
Sorry about that.
This is f of alpha right there.
So that probability is equal to 1.
I'm finally getting my bearings back on this.
So if I use this theorem here, where I use f of x as equal to
1 over x, then what it says for a renewal process, which
has inter-renewals x1, x2, and so forth, with a inter-renewal
process the expected inter-renewal time, which
might be infinite in general, but it has to
be bigger than 0.
It has to be bigger than 0 because renewals in 0 time are
not possible by definition.
And you've done a problem in the homework, or you will do a
problem in the homework, whether you've done it or not,
I don't know, but when you do it, it will in fact show that
the expected value of x has to be greater than 0.
We're also assuming that it's less than infinity.
So, at that point we can use this theorem which says that
when you look at, not sn of omega over n, but the inverse
of that, n over sn of omega, the limit of that is going to
be equal to f of x-bar which is 1 over x-bar.
So we have this limit here which is equal to 1 with
probability 1.
This is almost the strong law for renewal processes, and the
reason why it's not takes a couple of pages in the text.
What I'm going to try to argue here is that the couple of
pages, while you need to do mathematically, you can see
why when you do it you can see what answer
you're going to get.
And the argument is the following.
We talked about this last time a little bit.
If you take an arbitrary time t then you look at n of t for
that value of t, that's the number of arrivals you've had
in this renewal process up until time t.
And then you compare this point down here, which is the
time of the last arrival before time t, and this point
here, which is the time of the next arrival after time t, and
you look at the slopes here, this slope here
is n of t over t.
That's the thing we're interested in.
That expression there is the time average of the number of
renewals within time t, and what we're interested in is
what this becomes as t goes to infinity.
That's squeezed between this slope here, which is n of t
over s sub n of t.
This is s sub n of t.
And this point here is n of t, and this lower band here,
which is n of t divided by s sub n of t plus 1.
Now, all this argument is devoted to proving two things,
which are almost obvious and therefore I'm just going to
wave my hands about it.
What this says is when t gets very, very large, n of t
becomes very, very large also.
When n of t becomes very, very large, the difference between
this slope and this slope doesn't make a hill of beans
worth of difference, and therefore those two, that
upper band and that lower band become the same.
So with the argument as t gets large, n of t gets large, and
these two slopes become the same, you suddenly have the
statement that if you have a renewal process and the
inter-renewal time is less than infinity, it has to be
greater than 0.
Then on a sample path basis the sample paths, the number
of arrivals per unit time, converges to 1 over x-bar.
That's what this previous picture was saying to you.
It was saying that n over t over t, which is the number of
arrivals per unit time t, as t becomes very, very large on a
sample path basis that becomes equal to 1 over x-bar.
So, it says that the rate of renewals over the infinite
time horizon is 1 over x-bar with probability 1.
And what probability 1 statement again is saying that
the set of sample paths for which this is true has
probability 1.
This also implies the weak law for renewals, which says that
the limit as t approaches infinity of the probability
that n of t over t minus 1 over x-bar is greater than
epsilon goes to 0.
This, surprisingly enough, is a statement that
you hardly ever see.
You see a bunch of other renewal theorems, which this
theorem up here is probably the most important.
There's something called an elementary renewal theorem,
which is sort of a trivial statement but
kind of hard to prove.
And there's something called Blackwell's theorem, which
talks about a local period of time after time gets very,
very large.
It's talking about the number of renewals in some tiny
interval and it's proving things about stationarity over
that local time interval.
And we'll talk about that later, but this is sort of a
nice result, and that's what it is.
Let's review residual life a little bit.
The residual life of a renewal process at time t, the
residual life is the time you have to wait until the next
renewal, until the next arrival.
So residual life means, how long does it take?
If you think of these renewals as something that comes along
and kills everybody, then this residual life is the amount of
life that people still have left.
So, if you look at what that is for a particular sample
path, I have a particular sample path drawn here,
completely arbitrary.
When I look at the first arrival time here, s1 of
omega, and I look at all of the time before that, the
amount of time you have to wait until this arrival.
Remember this is one sample point.
We're not thinking in terms of an observer who comes in and
doesn't know what's going to happen in the future.
We're looking at one sample point.
We know the whole sample point.
And what we're doing we know this, we know this, we know
this, and all we're doing here saying for each time t along
this infinite path we're just plotting how long it is until
the next arrival.
And that's just a line with slope minus 1 until the next
arrival occurs.
After the next arrival and you start waiting for the arrival
after that, after that when you wait for the
arrival after that.
So on a sample path basis, the residual life is just this
sequence of isosceles triangles here.
So we looked at that, we said, if we look at the time from n
equals 1 up until the number of arrivals at time t, the
number of triangles we have is n of t omega.
The size of the area of each triangle is xi squared over 2.
The size of each triangle is xi squared over 2, and when we
get all done we want to take the average of it so we divide
each of these by t.
This quantity here is less than or equal to the integral
of y of t over t, and that's less than or equal to the same
sum where if we look at a particular time of t this sum
here is summing everything up to this point.
This sum here is summing everything up to this point.
And as t goes to infinity, this one little triangle in
here, even though this is the biggest triangle there, it
still doesn't make any difference.
As t goes to infinity, that washes out.
You have to show that, of course, but that's what we
showed last time.
When we go to the limit we find that this time average
residual life is equal to the expected value of x squared
over twice the main effect.
Now, this can't be infinite.
You can have a finite expected in a renewal time and you can
still have an infinite second moment.
If you look at the example we talked about last time where
the inter-renewal time at two possible values, either
epsilon or 1 over epsilon, in other words, it was either
enormous or it was very, very small, what we found out there
was that some of these inter-renewal intervals, a
very small fraction of them, but a very small fraction of
the inter-renewal intervals were enormously large.
And because they were enormously large, the time
average residual life turned out to be enormous also.
You think of what happens as epsilon
get smaller and smaller.
You can see intuitively why it makes sense that when you have
humongously long inter-renewal times, and you have this x
squared that occurs, because of this triangle here, I think
is possible to see why this quantity here can't become
infinite if you had this situation of a very long
tailed distributions for the inter-renewal time.
You have enormously long residual waits then, and you
have them with high probability, because if you
come into this process at some arbitrary time you're somewhat
more likely to wind up in one of these large intervals than
one of the small intervals, and that's what causes all the
trouble here.
There are similar examples here.
You can look at the age of a process, z of t is defined as
the interval between t and the previous arrival.
So, if we look at sum time t, the age at that point is the
amount of time back to the previous arrival that goes up
like a triangle, drops as soon as we get the next arrival.
It's exactly the same.
It's just the same thing looking at it backwards
instead of forwards, so the answer is the same also.
If you look at something called duration, the question
here is at a particular time t.
If we take the difference between the next arrival and
the previous arrival, that's called the duration of time t.
How big is that?
Well, obviously this is the same problem as that, and the
residual life and in fact, that's exactly the sum of the
residual life and the age.
So, it's not surprising that the time average--
oh my.
There should be a 2 down there.
So it's exactly the same situation as we had before
when we were talking about residual life.
Now we'd like to generalize this, and I hope the
generalization is almost obvious at this point.
These three quantities here are all examples of assigning
rewards to renewal processes.
And the reward at any time t, when I'm trying to do this, is
restricted to be a function of the inter-renewal period
containing t.
In other words, when you try to define what the reward is
at a given time t, it's a function only of what's going
on in a renewal period containing t.
Now, in its simplest form, I want to go
make this even simpler.
In its simplest form r of t is restricted to be a function of
the age of time t and the duration of time t.
In other words, you try to say what's the reward in a given
time t, and it's the same as these three
examples we looked at.
It's some function of how far back do you have to go to the
previous arrival, and how far forward do you have to go on
until the next arrival.
You can make this a function if you want to of any of the
three quantities residual life, age, and duration.
It seems to be intuitively a little simpler to talk about
age and duration as opposed to the other quantities.
So the time average for sample path of r of t is found by
analogy to residual life.
That's the way I'm going to do it here in class.
The notes do a little more careful job
than just by analogy.
But you start with the n-th inter-renewal interval and you
say, what is the aggregate reward I get in the n-th
inter-renewal interval?
It's a random variable, obviously.
And what it is is the integral from the previous arrival up
to the next arrival of r of t and omega dt.
This is very strange, because when we looked at this before
we talked about n of t and we talked about s sub n of t plus
1 and s sub n of t, and suddenly n of t plus 1 has
turned into n, and n of t has turned into n minus 1.
What has happened?
Well, this gives you a clue.
And it's partly how we define the intervals.
Interval 1 goes from 0 to s1 and z of t equals t, interval
n z of t is t minus s sub n minus 1.
Let me go back and show you the original figure here and I
think it will make it clear.
Now, the first arrival comes at this time here.
The first interval we're talking about goes from 0 to
time s1 of omega.
What we're interested in, in this first interval going from
0 to s1 of omega, is this interarrival time, which is
the first interarrival time.
This first interval is connected to x1 and it goes
from s sub 0, which is just at 0 up to s1.
The second arrival time goes from s1 up to s2.
When we look at n of t, n of t is a sum t here in this
interval here, and this is s.
Am I going to get totally confused writing
this or aren't I?
I think I might get totally confused, so I'm not going to
try to write it.
Because when I try to write it I'm trying to make the
comparison between n of t and n of t plus 1, which are the
things up here.
But the quantities here with the renewal periods are, this
is the first inter-renewal period, this is the second
inter-renewal period.
The second inter-renewal period goes from s sub 2 back
to s sub 1.
The n-th goes from s sub n back to s sub n minus 1.
So that's just the way it is when you count this way.
If you count it the other way you would get even more
confused because then in the interval x sub n the
inter-renewal time would be x sub n plus 1 and
that would be awful.
So, this is the way we have to do it.
Let's look at what happens with the expected
inter-renewal time, then, as a time average.
And you can think of this as a being for a sample time.
When you have an expression which is valid for a sample
time, it's also valid for a random variable, because it
maps every sample point into some value.
So, I'll just call it here r sub n, which is the amount of
reward that you pick up in the n-th inter-renewal period, so
it's the integral from the n minus first arrival up to the
n-th arrival, and we're integrating over r of t over
this whole interval.
So r of t is a function of the age and of the duration.
The age is t minus s sub n minus 1, and the duration is
just x of n.
Now that's a little weird also because before we were talking
about the duration as being statistically very, very
different from the inter-renewal period.
But it wasn't.
If you look on a sample path basis, this duration of the
n-th inter-renewal period is exactly equal to x sub n of
omega, which is, for that particular sample path, it's
the value of that inter-renewal interval.
So when we integrate this quantity now, we want to
integrate it over dt.
What do we get?
Well, we've gotten rid of this t here, we just have the n-th
duration here.
The only t appears over here, and this is r of a difference
between t and s sub n minus 1.
So, we want to do a change of variables, we want to let z be
t minus s sub n minus 1, and then we integrate z
from 0 to x sub n.
And now just imagine that you put omegas in all of these
things to let you see that you're actually dealing with
one sample function, and when you leave the omegas out
you're just dealing with the random variables that arise
because of doing that.
So when we do this integration, what we get is
the integral of r of z with this fixed x sub n integrated
against dz.
This is a function only of the random variable x sub n.
Strange.
That's the expected value of r sub n.
This unfortunately can't be reduced any further, this is
just what it is.
You have to find out what is the integral of r of zx
integrated over z, and then we have to take the expected
value over x.
If the expectation exists, you can write this in a simple
way, because this quantity here integrated
is just r sub n.
And then you take the expected value of r sub n, and you
divide by x-bar down here.
This quantity here is just the expected value of the integral
of r of zx dz.
And that integral there is just the expected
value of r sub n.
So, if you look over the entire sample path from 0 to
infinity, what you get then is that the time average reward
is just equal to the expected value of r sub n
divided by x squared.
Oops.
Lost the example.
Suppose we want to find the k-th moment of the age.
Simple thing to try to find.
I want to do that because it's simple.
If you're looking at the k-th moment of the age looked at as
a time average, what it is is the reward at time t, it's the
age at time t to the k-th power.
We want to take the age to the k-th power at time t, and then
integrate over rt and divide by the final value
and go to the limit.
So the expected value of this k-th moment over 1
inter-renewal period is this integral here, z to the k dz
times dF sub x of x.
The only place x comes in this integral is in the final point
of the integration where we are integrating up to the
point x, at which the duration ends, and then we're taking
the expected value of this.
So, we integrate this, the integral of the z to the k
from 0 to x is just x to the k plus 1 divided by k.
So when you take that integral and you look at
this, what is this?
It's the 1 over k times the expected value of the random
variable, x to the k plus first power.
That's just this taking the expected value of it, which is
you take the expected value by integrating times
the s sub x of x.
Yes?
AUDIENCE: [INAUDIBLE]
divide by k plus 1?
PROFESSOR: Yes.
I'm sorry.
My evil twin brother is speaking this morning, and
this should be a k plus 1 here.
This should be a k plus 1, and this should be a k plus 1.
And if you look at the example of the first moment, which is
the first thing we looked at today, namely finding the
expected value of the age, what we find out is that it's
expected value of x squared divided by 2 times x-bar.
So with k equal to 1 we get the expected value of x
squared over 2 times x-bar.
It's even worse because I left a 2 out when we were talking
about the expected value of age.
And to make things even worse, when I did this I went back to
check whether it was the right answer, and of course the two
mistakes cancelled each other out.
So, sorry about that.
The thing I really wanted to talk about today is stopping
trials for stochastic processes, because that's a
topic which has always caused confusion in this class.
If you look at any of the textbooks on stochastic
processes, it causes even more confusion.
People talk about it, and then they talk about it again, and
they say what we said before wasn't right, and then they
talk about it again, they say what we said before that
wasn't right, and it goes on and on and on.
And you never know what's going on.
Very often instead of looking at an entire stochastic
process over the infinite interval, you'd like to look
at a finite interval.
You'd like to look at the interval going from 0 up to
some time t.
But in almost a majority of those cases, the time t that
you want to look at is not some fixed time but some
random time which depends on something which is happening.
And when you want to look at what's going on here, it's
always tricky to visualize this because what you have is
t becomes a random variable.
That random variable is a function of the sample values
that you have up until time t.
But that seems like a circular argument, t a random variable
and it depends on t.
Things are not supposed to depend on themselves, because
when they depend on themselves you have trouble understanding
what's going on.
So, we will sort that out.
In sorting it out we're only going to look at
discrete-time processes.
So, we'll only look at sequences of random variables.
We will not assume that these random variables are IID for
this argument here, because we want to look at a much wider
range of things.
So we have some arbitrary sequence of random variables,
and the idea is you want to sit there looking at this
evolution of the sequence of random variables, and in fact,
you want to sit there looking at the sample
path of this evolution.
You want to look at x1, and you want to look at x2, and
you want to look at x3, and so forth.
And at each point along the line in terms of what you've
seen already, you want to decide whether you're going to
stop or whether you're going to continue at that point.
And if at that point you develop a rule for what you're
going to do with each point you call that a stopping rule,
so that you have a rule that tells you when you want to
stop looking at these things.
Let me give you some idea of the generality of this.
Often when we look at queuing systems we are going to have
an arrival process where arrivals keep coming in,
things get serviced one after the other, and one of the very
interesting things is whether the system ever empties out or
whether the queue just keeps building up forever.
Well, an interesting stopping rule then is when the queue
empties out.
We will see that an even more interesting stopping rule is
starting with some arbitrary first arrival at time 0.
Let's look for the time until another arrival occurs which
starts another busy period.
These stopping times, then, are going to be critical in
analyzing things like queuing systems.
They're also going to become critical in trying to analyze
these renewal systems that we're already talking about.
They will become critical to trying to take a time average
view about renewal processes instead of a, not a time
average, but an ensemble average viewpoint looking at
particular times t as opposed to taking a time average.
So we're going to look at these things in
many different ways.
In order to do that, since you're looking at a sample
function, you're observing it on each
arrival of this process.
At each observation of a sample value of a random
variable, you decide whether you want to stop or not.
So a sensible way to deal with this is to look over all time,
define a random variable J, which describes when this
sequence is to be stopped.
So for each sample value x1 of omega, x2 of omega, x3 of
omega, and so forth, if your rule is to stop the first time
you see a sample value which is equal to 0, then J is a
integer random variable whose value is the first n at which
is x of n is equal to 0 and many other examples.
So we try a 1 then, x1 of omega is observed.
A decision is made based on x1 of omega
whether or not to stop.
If we stop, J of omega equals 1.
If we don't stop, J of omega is bigger than 1, but we don't
know what it is yet.
At trial two, if we haven't stopped yet, you observe x2 of
omega, the second random variable, second sample value,
you make a decision again based on x1 of omega and x2 of
omega whether or not to stop.
If you stop, J of omega is equal to 2.
So you can visualize doing this on each trial, you don't
have to worry about whether you've already stopped, you
just have a rule that says, I want to stop at this point if
I haven't stopped before.
So the stopping event at time n is that you stop either if
your rule tells you to stop or you stop before.
And if you stop before then you're obviously stopped.
At each trial n if stopping has not yet occurred x of n is
observed and the decision based on x1 to xn is made.
If you stop, then J omega is equal to n.
So for each sample path J of n is the time or the trial at
which you stop.
So, we're going to define a stopping trial, or stopping
time, J for a sequence of random variables.
You're going to define this to be a positive integer-valued
random variable that has to be positive, because if you're
going to stop before you observe anything that's not a
very interesting thing.
So you always observe at least one thing, then you decide
whether you want to stop or you proceed, so forth.
So it's a positive integer-valued random
variable, you always stop at an integer trial, and for each
n greater than or equal to 1 the indicator random variable
the indicator of the event J equals n is a function of what
you have observed up until that point.
This part is a really critical part of it.
The decision to stop at a time n has to be a function only of
what you have already observed.
You're not allowed as an observer to peek ahead a
little bit and then decide on the basis of what you see in
the future whether you want to stop at this time or not.
One example in the notes is that you're playing poker with
somebody, and you make a bet and the other person wins.
You make the bet on the basis of what you've seen in your
cards so far, you don't make your bet on the basis of your
observation of what the other player has, and the other
player will get very angry if you try to base your decision
later on what the person you're playing with has.
So you can't do that sort of thing.
You can't peek ahead when you're doing this.
You could design random variables where, in fact, you
can look ahead, but the times where you use what people call
stopping trials are situations in which you stop based on
what has already happened.
You can generalize this, and we will generalize it later,
where you can stop on the basis of other random
variables, which also evolve in time, but you stop on the
basis of what has already happened up until time n.
We're going to visualize conducting successive trials
until sum n in which the event J equals n occurs.
Further trials then cease.
It is simpler conceptually to visualize stopping the
observation of trials after the stopping trial, but
continuing to conduct trials.
In other words, if we start talking about a stopping
process and we say we stop on the n-th trial, but we're
forbidden to even talk about the n plus
first random variable.
Now the n plus first random variable occurs on some sample
paths but doesn't occur on other sample paths.
I don't know how to deal with that, and neither do you.
And what that means is the way we're going to visualize these
processes is physically they continue forever, but we just
stop observing them at a certain point.
So, this stopping rule is the time at which we stop
observing as opposed to the time at which the random
variable ceases existing, if we define the random variable,
it has a value for all sample values, and sample values
involve all of these paths.
So you sort of have to define it that way.
One thing you would like to do is in many of these situations
you wind up never stopping.
And the trouble is, when you're investigating stopping
rules, stopping trials, you usually don't know ahead of
time whether you're ever going to stop or not.
And because of that if you don't know whether you're
going to stop or not, you can't call
it a stopping trial.
So, what one normally does is to say that J is a possibly
defective random variable.
And if it's possibly defective you mean that all sample
points are mapped into either finite J or perhaps J equals
infinity, which means that you never stop.
But you still have the same condition that we have here,
that you stop on the basis of x1, x of n.
And you also have the condition, which isn't quite
clear here, but it's clear from the fact that J is a
random variable, that the events J equals 1, J equals 2,
J equals 3, are all disjoint.
So you can't stop twice, you have to just stop once.
And once you stop you can't start again,
Now does everybody understand what is stopping trial or a
stopping rule is at this point?
If you don't understand it I think the trouble is you won't
realize you don't understand it until you start seeing some
of these examples where strange things are going on.
I guess we have to just go ahead and see
what happens then.
So the examples, which I'm not going to
develop in any detail.
A gambler goes to a casino and he gambles until he's broke.
So, he goes in with a finite amount of capital.
He has some particular system that he's playing by.
If he's lucky he gets to play for a long, long time, he gets
a lot of amusement as he loses his money, and if he's unlucky
he loses very quickly.
If you look at the odds in casinos and you apply the
strong law of large numbers, you realize that with
probability 1 you get wiped out eventually.
Because the odds are not in your favor.
Casinos wouldn't be built if the odds were in your favor.
Another example, just flip a coin until 10
successive heads appear.
10 heads in a row.
That's a rather unlikely event.
You're going to be able to figure out very easily from
this theory what's the expected amount of time until
10 successive heads appear.
It's a very easy problem.
A more important problem, this test and hypothesis with
repeated trials.
So, you have the hypothesis that a certain kind of
treatment will make patients well, and you want to know
whether to use this treatment.
So you test it on mice, or people, or what have you.
If you think it's pretty safe, you start
testing it on people.
And how many tests do you make?
Well, you originally think that as a scientist you should
plan an experiment ahead of time, and you should say, I'm
going to do 1,000 tests.
But if you're doing experiments on people and you
find out that the first 10 patients die, you're going to
stop the experiment at that point.
And if you find out that the first 100 patients all live,
well, you might continue the experiment.
But you're going to publish the results at that point
because you're going to try to get the FDA to approve this
drug, or this operation, or this what have you.
So if you view this stopping trial as the time at which
you're going to try to publish something, then, again, if you
have any sense, you are going to perform experiments, look
at the experiments as you go, and decide what you're going
to do as a function of what you've already seen.
I mean, that's the way that all
intelligent people operate.
So if science says that's not a good way to operate,
something's wrong with science.
But fortunately science doesn't say that.
Science allows you to do this.
AUDIENCE: So not every J is potentially defective?
You can come up with examples of
where it would be defective.
But not every one is necessarily [INAUDIBLE].
PROFESSOR: Yes.
If a random variable is defective it means there's
some probability that J is going to be infinite, but
there's also presumably some probability that J is equal to
1, a probability J is equal to 2, and so forth.
So we have a distribution function for J, which instead
of going up to 1 and stopping goes up to some smaller value
and stops, and then it doesn't go any further.
AUDIENCE: Not every [INAUDIBLE].
Some of these are definitely going to [INAUDIBLE].
PROFESSOR: Yes.
So this testing hypotheses is really what triggered Abraham
Wald to start trying to understand these situations,
because he found out relatively quickly that it
wasn't trivial to understand them, and all sorts of crazy
things were happening.
So he spent a lot of his time studying what happened in
these cases.
He called it sequential analysis.
You might have heard the word.
And sequential analysis is exactly the same idea.
It's looking at analyzing experiments as you go in time,
and either stopping or doing something else or changing the
experiment or what have you, depending on what happens.
Another thing is observe successive renewals in a
renewal process until s sub n is greater
than or equal to 100.
Now, this particular thing is one of the things that we're
going to use in studying renewal processes, it's why
we're studying this topic right now.
Well, we might study it now anyway, because we're going to
use it in lots of other places, but it's also why you
have to be very careful about talking about stopping time as
opposed to stopping trials.
Because if you stop an experiment at the arrival in a
renewal process at which that renewal occurs after time 100,
you don't know when you're going to stop.
It might be a very long time after 100 before that first
arrival after time 100 occurs, it might be a very short time.
So it's not a stopping time that you're defining, it's a
stopping trial.
You are defining a rule which tells you at which trial
you're going to stop as opposed to a rule that tells
you in which time you're going to stop, and we'll see this as
we go on in this.
Suppose the random variables in this process we're looking
at have a finite number of possible sample values.
Then any possibly defective stopping trial, and a stopping
trial is a set of rules for when you're going to stop
observing things, any stopping trial can be represented as a
rooted tree.
If you don't know what a rooted tree is, a rooted tree
is what I've drawn here.
So, there's no confusion here.
A rooted tree is something that starts on the left hand
side and it grows for a while.
It's like an ordinary tree.
It has a root, and it has branches that go up.
But any possibly defective stopping trial can be
represented as a rooted tree where the trial at which each
sample path stops is represented
by a terminal node.
So the example here I want to use is the same
example as in the text.
I was trying to get another example, which will be more
interesting.
The trouble is with these trees they get very big very
quickly, and therefore this is not really a practical way of
analyzing these problems.
I think it's a nice conceptual way of analyzing.
So, the experiment is x is a binary random variable, and
stopping occurs when the pattern 1, 0 first occurs.
You can certainly look at any other pattern you want to, any
much longer pattern.
But what I want to do here is to illustrate what the tree is
that corresponds to this stopping rule.
So if the first random variable has a value 1 and
then the second one has a value 0, you observe 1, 0, and
at that point the experiment is over.
You've seen the pattern 1, 0, and you stop.
So there's a big circle there.
If you observe 1 followed by a 1 followed by 0, again you
stop 1, 1, 1, followed by a 0 you stop, and so forth.
If you observe a 0 and then you observe a 1 and then you
observe a 0, that's the first time at which you've seen the
pattern 1, 0 so you stop there.
0, 1, 1, 0, you stop there, and so forth.
So you get this kind of ladder structure.
The point that I want to make is not that this particular
structure is very interesting.
It's that whatever kind of rule you decide to use for
stopping, you can express it in this way.
You can express the points at which you stop by big circles
and the points at which you don't stop as intermediate
nodes where you keep on going.
I think in some sense this takes the picture which you
have of sample values, of sample functions with the rule
that each sample value, which is often the easiest way to
express stopping rules, but in terms of random variables it
doesn't always give you the right story.
This gives you a story in terms of all possible choices
of the random variables.
The first toss has to be either a 1 or a 0.
The second toss has to be a 1 or a 0, but if it's a 0 you
stop and don't go any further.
If it's a 1 you continue further.
So you keep continuing along here, you can
continue forever here.
If you want to analyze this, it has a
rather peculiar analysis.
How long does it take until the first 1, 0 occurs?
Well, if you start out with a 1 this experiment lasts until
the first 0 appears.
Because if the 0 doesn't appear right away another 1
appears, so you keep going along this
path until a 0 appears.
If you see a 0 first and then you see a 1, you're back in
the same situation again.
You proceed until a 0 occurs.
What this is really saying is the time until you stop here
consists of an arbitrary number, possibly 0 of 0's,
followed by an arbitrary number, possibly 0 of 1's,
followed by a single 0.
So the only patterns you can have here which would lead to
these terminal nodes are some number of 0's followed by some
number of 1's, followed by a final 0.
But the example is not important, so that just tells
you what it means.
With all of this we finally have Wald's equality.
And Wald's equality, it looks like a strange thing.
I spent so much time talking about stopping rules because
all the complexity lies in the stopping rules.
Wald's equality itself is a very simple thing after you
understand that.
What Wald's equality says is, let x sub n be a sequence of
IID random variables, each would mean x-bar.
This is the kind of thing we've been
talking about a lot.
If J is a stopping trial for x sub n, n greater than or equal
to 1, in other words, if it satisfies that definition
where it never peeks ahead, and if the expected value of J
is less than infinity, then the sum s sub J equals x1, x2,
up to x sub J, the amount that the gambler has accumulated
before the gambler stops.
The sum of the stopping trial satisfied expected value of
the gain as equal to the expected value of x times the
expected number of times you play.
It sort of says that even when you use these stopping rules
to decide when to stop, there's no
way to beat the house.
If you're playing a fair game, your expected gain is equal to
the expected gain for trial times the expected number of
times you play.
And there's no way you can get around that.
Note that this is more than a [? poof ?]
and less than a proof.
I'll explain why it's more than a [? poof ?]
and less than a proof as we go.
s sub J, the random variable s sub J, it's equal to x sub 1
times the indicator function of j greater
than or equal to 1.
In other words, well, the indicator function J greater
than or equal to 1 is just universally 1.
x sub 2 times the indicator function of J greater than or
equal to 2.
In other words, s sub J includes x sub 2 if the
experiment proceeds long enough that you take the
second trial.
In other words, this quantity here, is 1 minus the
probability that you stop at the end of the first trial.
You continue here, each x sub n is included if the
experiment has not been stopped before time n.
So the experiments continue under the event that the
stopping time was greater than or equal to the J. So the
expected value of s of J is equal to the expected value of
this sum here, of x sub n times the indicator function
of J greater than or equal to n, which is equal to the sum
over n of the expected value of x sub n times this
indicator function.
By this time I hope that most of you are at least a little
bit sensitive to interchanging expectations and sums.
And the notes and the problem sets deal with that.
That's the part of the proof that I don't want to talk
about here, because it isn't really very interesting.
It's just one of these typical tedious analysis things.
Most of the time this is valid.
We will see an example where it isn't, so you'll see
what's going on.
The essence of the proof, however, the interesting part
of the proof is to show that xn and this indicator function
are independent random variables.
So you can think of that as a separate limit, that the n-th
trial and the indicator function of J greater than or
equal to n are independent of each other.
This seems a little bit weird, and it seems a little bit
weird because this includes the event that
J is equal to n.
And the event that J is equal to n is something that's
decided on the basis of the previous arrivals, or the
previous what have you.
It's highly influenced by x sub n.
So, x sub n is not independent of the indicator
function of J equals n.
So this is strange.
But we'll see how this works out.
Now, if we want to show that x sub n and this indicator
function are independent, the thing that you do is note that
the indicator function of J greater than or equal to n,
this is the set of events in which J is greater than or
equal to n.
What's the complement of that event?
The complement of the event J greater than or equal to n is
the event J less than n.
So this indicator function is 1 minus
this indicator function.
J less than n is the complement of the event J
greater than or equal to n.
When you perform the experiment either this
happens, in which case this is 1, or this happens, in which
case this is 1, and if this is 1, this is 0.
If this is 1, this is 0.
Also, the indicator function and J less than n is a
function of x1, x2, up to x sub n minus 1.
Let me spell that out a little more because it looks a little
strange the way it is.
i of J less than n is equal to i if the indicator random
variable for J equals 1.
Union with the indicator random variable
for J equals 2.
Union indicator function for J equals n minus 1.
These are all disjoint events.
You can't stop at two different times.
So if you stop before time n you either stop at time 1, or
you stop at time 2, or you stop at time n minus 1.
This event is independent.
Well not independent, it depends on x sub 1.
This event depends on x1 and x2.
This event depends on x1 up to x sub n minus 1.
OK, so this is what I mean when I say that the event J
less than n is determined by x1 up to x of n minus 1,
because each of these sub events is determined by a sub
event up there.
Since we're looking at the situation now where the x is
our IID and this event is a function of x1 x of n minus 1,
it just depends on those random variables.
These random variables are independent of x sub n so i of
J less than n is independent of x sub n and thus x sub n is
independent of J greater than or equal to n.
Now, as I said, this is very surprising, and it's very
surprising because this event, J greater than or equal to n,
typically depends very heavily on x sub n.
It depends on J equals n plus 1, and so forth.
So it's a little paradoxical and the resolution of the
paradox is that given that J is greater than or equal to n,
in other words, given that you haven't stopped before time n,
the time at which you stop is very dependent on the
observations that you make.
But whether you stopped before time n or whether you haven't
stopped before time n is independent of x sub n.
So it really is this conditioning here that makes
this a confusing issue.
But whether or not J greater than n occurs depends only on
x1 up to x sub n minus 1.
And with that it's easy enough to finish the proof, it's just
writing a few equations.
The expected value of s sub J is equal to the sum over n and
the expected value of x sub n times the indicator function
of J greater than or equal to n.
We've just shown that these are independent.
This random variable is independent of this random
variable, that's independent of the event J greater than or
equal to n, so it's independent
of the random variable.
The indicator function is J greater than or equal to n, so
we can write this way.
Now, when we have this expression each of the x sub
n's are IID, so all of these are the same quantity x-bar.
So since they're all the same quantity x-bar we can just
take it outside of the summation.
So we have x-bar times the sum of the expected value of i of
J greater than or equal to n.
Now, the expected value of this indicator function, this
indicator function is 1 when J is greater than or equal to 1,
and it's 0 otherwise.
So the expected value of this is just a probability that J
is greater than or equal to n.
So, this is equal to the expected value of x times the
sum of the probabilities that J is greater
than or equal to n.
This is really adding the elements of the complimentary
distribution function for J, and that gives us the expected
value of J. If finite or not finite gives us this anyway.
So, this really is a proof except for that one step.
Except for this interchange of expectation and summation here
which is, well, we will get an idea of what that has to do
with anything.
So let's look at an example.
Stop when you're ahead.
And stop when you're ahead is a strategy for
gambling with coins.
You toss a coin with probability of
heads equal to p.
And we want to look at all the different cases for p, whether
p is fair or biased in your favor or biased in the other
person's favor.
And the rule is that you stop whenever you're
winnings reach one.
So you keep on dumbly gambling, and gambling, and
gambling, until finally you get to the point where you're
one ahead and when you're one ahead you sigh a sigh of
relief and say, now my wife won't be angry at me, or my
husband won't divorce me, or something of that sort.
And you walk away swearing never to gamble again.
Except, when you look at this you say, aha!
If the game is fair, I am sure to win, and after I win I
might as well play again, because I'm sure to win again,
and I'm sure to win again, and I'm sure to win again, and so
forth, which is all very strange.
And we'll see why that is in a little bit.
But first let's look at the case where p is
greater than 1/2.
In other words, you have a loaded coin and you've talked
somebody else, some poor sucker, into playing with you,
and you've decided you're going to stop the first time
you're ahead.
And because p is greater than 1/2, the expected value of
your gain each time you play the game is greater than 0.
You're applying IID random variables so that eventually
your winnings, if you kept on playing forever,
would become humongous.
So at some point you have to be 1 ahead in this process of
getting to the point where you're arbitrarily large
amounts ahead.
So with probability 1, you will eventually become ahead.
We would like to know since we know that you're going to be
ahead at some point, how long has it going to
take you to get ahead?
We know that j has to be a random variable in this case,
because we know you have to win with probability 1 s sub J
equals 1 with probability 1.
Namely, your winnings up to the time J. At the time J your
winnings are 1.
So s sub J is equal to 1, the expected value of s sub J is
equal to 1, and Wald says that the expected value of J, the
expected time you have to play, is just 1 over x-bar.
Isn't that neat?
Which says it's 1 over 2p minus 1.
The expected value of x is p plus minus 1 times 1 minus p
and if you look at that it's 2p minus 1, which is positive
when p is bigger than 1/2.
After all the abstraction of Wald's equality it's nice to
look at this and solve it in a different way to see if we get
the same answer.
So we'll do that.
If J is equal to 1, that means that on the first flip of the
coin you've got heads, you became 1 up and you stopped.
So the experiment was over at time 1 if the
first toss was a head.
So if the first toss was a tail, on the other hand, what
has to happen in order for you to win?
Well, you're sitting there at minus 1 and you're going to
stop when you get to plus 1.
Every time you move up one or down one, which means if
you're ever going to get from minus 1 up to plus 1, you've
got to go through 0 in order to get there.
So let's look at the expected time that it takes to get to 0
for the first time, and then we'll look at the expected
time that it takes to get from 0 to 1 for the first time.
Well, the expected time that it takes to get from minus 1
to 0 is exactly the same as the expected time that it
takes to get from 0 to 1.
So, the equation that you then have is the expected value of
J is going to be 1.
This is the first step you have to take anyway, and with
probability 1 minus p you went to minus 1, and if you went to
minus 1 you still have J-bar left to go to get to 0 again,
and another J-bar to get to 1, so J-bar is equal to 1 plus
failing the first time it takes 2J bar to get
back up to plus 1.
If you analyze this equation, J-bar is equal to 1 plus
2J-bar minus 2Jp-bar, you work that out and it's just 1 over
2p minus 1.
So fortunately we get the same answer as we got with the
Wald's equality.
Let's go one again.
Let's look at what happens if you go to the casino and you
play a game where you win $1 or lose $1.
But the probability in casinos is somehow always tilted to be
a little less.
If you play for red or black in roulette, what happens is
that red or black are equally likely, but there's the 0 and
double 0 where the house always wins.
So your probability of winning is always just a little less
than 1/2, so that's the situation we have here.
It's still possible to win and stop.
Like if you win on the first toss, then
you're going to stop.
So J equals 1 with probability p.
J equals 3 if you lose, and then you win,
and then you win.
So J equals 3 with probability p squared times 1 minus b.
And there are lots of other situations you can look at J
equals 5 and see when you win then, and you can count all
these things up.
Let's try to find a simple way of counting them up.
Let's let theta be the probability that J is less
than infinity.
In other words, theta is the probability that you're ever
going to stop.
Sometimes in this game you won't ever stop because you're
going to lose for while, and as soon as you lose a fair
amount you're very unlikely to ever recover because you're
drifting South all the time.
Since you're drifting South, theta is going to
be less than 1.
So how do we analyze this?
Same way as before, given that J is greater than 1, in other
words, given that you lose on the first trial, the event J
less than infinity requires that at some time you go from
minus 1 back up to 1.
In other words, there's some time m at which s sub m minus
s sub 1 is equal to 1.
s sub m is the time at which you first get from minus 1
back up to 0, and then there's some time that it takes to get
from 0 up to 1, and there's some
probability that ever happened.
The probability that you ever reach 0 is equal to theta, the
probability you ever get from 0 up to plus 1 is theta again.
So theta is equal to p, which is the probability that you
win right away, plus probability you lose on the
first toss, and you ever get from minus 1 to 0 and you ever
get from 0 to 1.
So theta equals p plus 1 minus p times theta squared.
There are two solutions to this.
One is that theta it is p over 1 minus p and the other is
theta equals 1, which we know is impossible.
And thus J is defective, and Wald's equation is
inapplicable.
But the solution that we have, which is useful to know, is
that your probability of ever winning is p over 1 minus p.
Nice to know.
That's what happens when you go to a casino.
Finally, let's consider the interesting case which is p
equals 1/2.
The limit as p approaches 1/2 from below, probability that J
less than infinity goes from p over 1 minus p goes to 1.
In other words, in a fair game, and this is not obvious,
there's probability 1 that you will eventually get to the
point where s sub n is equal to 1.
You can sort of see this if you view this in terms of the
central limit theorem.
And we looked pretty hard at the central limit theorem in
the case of these binary experiments.
As n gets bigger and bigger, the standard deviation of this
grid grows as the square root of n.
Now the standard deviation is growing as the square root of
n and over a period of n you're wobbling around.
You can see that it doesn't make any sense to have the
standard deviation growing with the square root of n if
you don't at some point go from the positive half down to
the negative half.
If you always stay in the positive half after a certain
point, then that much time later you're always going to
stay in the same half again, and that doesn't work in terms
of the square root of n.
So you can convince yourself on the basis of a very tedious
argument what this very simple argument shows you.
And as soon as we start studying Markov chains with
uncountably infinite number of states, we're going to analyze
this case again.
In other words, this is a very interesting situation because
something very interesting happens here and we're going
to analyze it at least three different ways.
So if you don't believe this way, you will
believe the other way.
And what happens as p approaches 1/2 from above you
know the expected value of J approaches infinity.
So what's going to happen at p equals 1/2 as the expected
time for you to win goes to infinity, the probability of
your winning goes to 1, and Wald's equality does not hold
because the expected value of x is equal to 0, and you look
at 0 times infinity and it doesn't tell you anything.
The expected value of s sub J s sub n at every value of n is
0, but s sub n when the experiment ends is equal to 1.
So there's something very funny going on here and if you
think in practical terms though, it takes an infinite
time to make your $1.00 and more important, it requires
access to an infinite capital.
We will see that if you limit your capital, if you limit how
far South you can go, then in fact you don't win with
probability 1, because you can go there also.
Next time we will apply this to renewal processes.