The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare continue to
offer high-quality educational resources for free.
To make a donation or view additional materials from
hundreds of MIT courses, visit MIT OpenCourseWare at
ocw.mit.edu.
PROFESSOR: OK, so let's get started.
I want to talk mostly about countable state Markov chains
today, which is the new topic we started on Wednesday.
I want to talk just a little bit about the strong law proof
that was in the third problem of the quiz.
I'm not doing that because none of you understood
anything about it.
I'm doing it because all of you understood more about it
than I thought you would.
And in fact, I've always avoided saying too much about
this proof because I thought everybody was tuning them out.
And for the class here, it looks like a lot of you have
tried seriously to understand these things.
So I thought I would explain that one part of the quiz in
detail so that you'd see the parts you're missing, and so
that [INAUDIBLE] all these other proofs that we have,
talking about the strong law and the strong law for
renewals, and putting them together and all of these
things, all of them are essentially the same.
And it's just a matter of figuring out
how things fit together.
So I wanted to talk about that because it's clear that most
of you understand enough about it that it makes sense.
The situation in the quiz, which is very close to the
usual queuing situation and little theorem type things,
there's a sequence of Y sub i's.
They're IID.
There's the service times for G/G infinity queue, and n of
t, which is a renewal process for the
arrivals to the process.
We have arrivals coming in according to this renewal
process, which means the [INAUDIBLE] arrival times, X
sub i, were IID.
And we want to put those two things together.
And we want to find out what this limit is.
If it's a limit, show that it's a limit.
And hopefully show that it's a limit with probability 1.
And I think a large number of you basically realize that
this argument consisted of a bunch of steps.
Some people with more detail than others.
But the first step, which we do in all of these arguments,
is to divide and multiply by n of t of omega.
So we're starting out, looking at some particular sample
path, and start out by multiplying and dividing by n
of t of omega.
The next thing is to claim that the limit of this times
this is equal to the limit of this times the limit of this.
Almost no one recognized that as a real problem, and that is
a real problem.
It's probably the least obvious thing
in this whole problem.
I'm not saying you shouldn't have done that, because I've
been doing that in all the proofs I've been
giving you all along.
It is sort of obvious that that works.
And when you're constructing a proof, especially in a quiz
when you don't have much time, things which are almost
obvious or which look obvious, you should just go ahead and
assume them and come back later when you have time to
see whether that really makes sense.
That's the way you do research also.
You don't do research by painstakingly establishing
every point in some linear path.
What you do is you carelessly as you can and with as much
insight as you can, you jump all the way to the end, you
see where you're trying to go, you see how to get there, and
then you come back and you try to figure out what each
of the steps are.
So this is certainly a very reasonable way of solving this
problem, because it looks like this limit should be equal to
this limit times this limit.
The next step in the argument is to claim that this sum, up
to N of t of omega, over N of t of omega, as
t approaches infinity--
the argument is that t approaches infinity--
this N of t of omega goes through, one by one, a
sequence, 1, 2, 3, 4, 5, and so forth.
So this limit is equal to that limit.
I've never been able to figure out whether that's obvious or
not obvious.
It is just on the borderline between what's obvious and not
obvious, so I'm gonna prove it to you.
And then the next step is to see that N of t of omega over
t is equal to 1/X-bar with probability 1.
And this limit is equal to Y-bar with probability 1.
The first argument, this equal to 1/X-bar is because of the
strong law with renewals.
And this one over here is because of the strong law of
large numbers.
And most of you managed to get this.
And the whole argument assumes that X-bar is less than
infinity, and Y-bar is less than infinity.
Now, how do you go back and actually see that this
actually makes sense?
And that's what I want to do next.
And if you look at this, you can't do it by starting here
and working your way down to there, because there's no way
you're going to argue that this limit is equal to this
product of limit unless you know something about this and
you know something about this.
So you really have to establish that these things
have limits first before you can go back
and establish this.
In the same way, you have to know something about this
before you can establish this.
So what you have to do way after you've managed to have
the insight to jump the whole way through this thing, is to
go back and argue each of the points, but argue them in
reverse order.
And that's very often the way you do research, and it's
certainly the way you do quizzes.
So let's see where those arguments were.
Start out by letting A1 be the set of omega for which this
limit here, N of t of omega over t, is equal to 1/X-bar.
By the strong law for renewal processes, the probability of
A1 equals 1.
This is stating this in a little cleaner way, I think,
than we stated the strong law of renewals originally,
because we started out by tumbling together this
statement and this statement.
I think it's cleaner to say, start out, there's a set of
omega 1 for which this limit exists.
And what the strong law says is that that set of omega has
probability 1.
And now we have some terminology for A1, what it
actually means.
It is the set of omega for which this works.
You never have it working for all omega,
only for some omega.
Then the next step, let A2 be the set of omega for which the
limit is n goes to infinity of 1/n times the sum of Y sub i
of omega, is equal to Y-bar.
By the strong law of large numbers, the probability of A2
is equal to 1.
So now we've established there's an A1, there's an A2.
Each of them have probability 1.
On A1, one limit exists.
On A2, the other limit exists.
And we have two sets, both at probability 1.
What's the probability of the intersection of them?
It has to be 1 also.
So with that, you know that equation three is equal to
equation four for omega in the sets, A1, A2.
And also, you know that the probability of A1,
A2 is equal to 1.
So we've established this part of the argument down here.
Now we want to go up and establish this part of the
argument, which as I said, I can't convince myself that
it's necessary or not necessary.
But since I can't convince myself, I thought, in trying
to make up solutions for the quiz, I ought to actually
write a proof of it.
And I want to show you what the proof is so that, if it's
not obvious to you, you'll know exactly how to do it.
And if it is obvious, you can maybe sort out exactly why
it's obvious.
So this is an epsilon delta kind of argument.
We assume that omega is in A2.
That's the set for which the strong law of
large numbers holds.
There exists some integer, n, which is a function of both
epsilon and omega.
This is the funny thing about all of
these strong law arguments.
In almost all of them, you're dealing with
individual sample paths.
When you start saying something exists as a limit,
you're not saying that it exists as a limit for the
random variables.
You're saying it exists as a limit for a
set of sample paths.
And therefore, this epsilon here that you're gonna choose,
you need some integer there, such that this minus this is
less than epsilon.
I think I'm going to have to give up on this.
These things run out of batteries too quickly.
So we have that this difference here must be less
than epsilon if n is bigger than that m of epsilon omega.
That's simply what a limit means.
That's the definition of a limit.
The only way to define a limit sensibly is to say, for all
epsilon greater than 0, no matter how small the epsilon
is, you can always find an n big enough that this
difference here is less than epsilon.
Then if omega is also in A1, the limit of N of t of omega
has to be equal to infinity.
If you want to, you can just say, we proved in class that
the limit of t of omega is equal to infinity with
probability 1, and introduce another set, A3.
You want to do a probability 1.
But let's do it this way.
And then there has to be a t, which is also a function of
epsilon and omega, such that N of t and
omega is greater than--
that's an integer, by the way.
That's greater than or equal to m of epsilon of omega for
all t, which is greater than or equal to t
of epsilon of omega.
That says that this difference here is less than omega.
And that's true for all epsilon greater than 0.
And that says that, in fact, this limit has to exist.
This limit over here is equal to Y-bar with probability 1.
So that's what we were trying to prove here, that this limit
is the same as this limit.
So we found out what this limit is.
We found out that it exists with probability 1, namely on
the set A2.
This is equal to this, not necessarily on A2,
but on A1 and A2.
So we got into there.
Now how do we get the fact that this limit times this
limit is equal to the limit of [? these. ?]
Now we have a chance of proceeding, because we've
actually shown that this limit exists on some set with
probability 1, this limit exists on some set with
probability 1.
So we can look at that set and say, for omega in that set,
this limit exists and this limit exists.
Those limits are non-0 and they're non-infinite.
The important thing is that they're non-infinite.
And we move on from there, and to do that carefully.
And again, I'm not suggesting that I expect any of you to do
this on the quiz.
I would have been amazed if you had.
It took me quite a while to sort it out, because all these
things are tangled together.
Where am I?
I want to be in the next slide.
Ah, there we go.
Finally, we can interchange the limit of a product of two
functions--
say, f of t, g of t--
with the product of the limits.
Can we do that?
If the two functions each have finite limits, as the
functions of interests do for omega in A1, A2, then the
answer is yes.
And if you look at any book on analysis, I'm sure that
theorem is somewhere in the first couple of chapters.
But anyway, if you're the kind of person like I am who would
rather sort something out for yourself rather than look it
up, there's a trick involved in doing it.
It's this equality right here.
f of t times g of t minus ab.
What you want to do is somehow make that look like f of t
minus a, which you have some control over, and
g of t minus b.
So the identity is this is equal to f of t minus a times
g of t minus b--
we have control over that--
plus a times g of t minus b, plus b times f of t minus a.
And you multiply and add all those things together and you
see that that is just an identity.
And therefore the magnitude of f of t times g of t minus ab
is less than or equal to this.
And then you go through all the epsilon delta stuff again.
For any epsilon greater than 0, you choose a t such that
this is less than or equal to epsilon for t
greater than the t epsilon.
This is less than or equal to epsilon for t greater than or
equal to t epsilon.
And then this difference here is less than or equal to
epsilon squared plus this.
And with a little extra fiddling around, that shows
you have that f of t, g of t minus ab approaches 0 as t
gets large.
So that's the whole thing.
Now, let me reemphasize again, I did not
expect you to do that.
I did not expect you to know how to do analysis arguments
like that, because analysis is not a
prerequisite for the course.
I do want to show you that the kinds of things we've been
doing are not, in fact, impossible.
If you trace them out from beginning to end and put in
every little detail in them.
If you have to go through these kinds of arguments
again, you will in fact know how to make it precise and
know how to put all those details in.
Let's go back to countable state Markov chains.
As we've said, two states are in the same class as they
communicate.
It's the same definition as for finite state chains.
They communicate if there's some path by which you can get
from i to j, and there's some path from which you can get
from j to i.
And you can't get there in one step only, but you can get
there in some finite number of steps.
So that's the definition of two states communicating.
The theorem that we sort of proved last time is that all
states in the same class are
recurrent or all are transient.
That's the same as the theorem we have for finite state
Markov chains.
It's just a little hard to establish here.
The argument is that you assume that j is recurrent.
If j is recurrent, then the sum has
to be equal to infinity.
How do you interpret that sum there?
What is it?
P sub jj, super n, is the probability that you will be
in state j at time n given that you're in
state j at time 0.
So what we're doing is we're starting out in time 0.
This quantity here is the probability that we'll be in
state j at time n.
Since you either are or you're not, since this is also equal
to the expected value of state j at time n, given
state j at time 0.
So when you add all these up, you're adding expectations.
So this quantity here is simply the expected number of
recurrences to state j from time 1 up to time infinity.
And that number of recurrences is equal to infinity.
You remember we argued last time that the probability of
one recurrence had to be equal to 1 if it was recurrent.
If you got back to j once in finite time, you're going to
get back again in a finite time again.
You're going to get back again in finite time.
It might take a very, very long time, but it's finite,
and you have an infinite number of returns as time goes
to infinity.
So that is the consequence of j being recurrent.
For any i such that j and i communicate, there's some path
at some length m such that the probability of going from
state i to state j in m steps is greater than 0.
That's by meaning of communicate.
And there's some m and some pji of l.
Oh, for [? some m. ?]
And there's some way of getting back from j to i.
So what you're doing is going from state i to state j, and
there is some path for doing that.
You're wobbling around, returning to state j,
maybe returning to
state i along the way.
That's part of it.
And eventually there's some path for going
from j back to i again.
So this sum here is greater than or equal.
And now all I'm doing is summing up the paths which in
m steps go from i to j, and those paths which in the final
l steps go from j back to i.
And they do whatever they want to in between.
So I'm summing over the number of times they are in between.
And this sum here is summing over pjjk.
And that sum is infinite, so this sum is infinite.
So that shows that if j is recurrent, then i is recurrent
also for any i in the same class.
And you can do the same thing reversing i and j, obviously.
And if that's true for all classes that are very
recurrent, all law classes that are transient have to be
in the same class also, because a state is either
transient or it's recurrent.
If a state j is recurrent, then the
recurrence time, T sub jj.
When you read this chapter or read my notes, I apologize
because there's a huge confusion here.
And the confusion comes from the fact that there's an
extraordinary amount of notation here.
We're dealing with all the notation of finite-state
Markov chains.
We're dealing with all the notation of renewal processes.
And we're jumping back and forth between theorems for one
and theorems for the other.
And then we're inventing a lot of new notation.
And I have to rewrite that section.
But anyway, the results are all correct as far as I know.
I mean, all of you can remember notation much better
than I can.
So if I can remember this notation, you can also.
Let me put it that way.
So I can't feel too sorry for you.
I want to rewrite it because I'm feeling sorry for myself
after every year I go through this and try to re-understand
it again, and I find it very hard to do it.
So I'm going to rewrite it and get rid of
some of that notation.
We've already seen that if you have a chain like this, which
is simply the Markov chain corresponding to Bernoulli
trials, if it's Bernoulli trials with p equals 1/2, you
move up a probability 1/2, you move down
with probability 1/2.
As we said, you eventually disperse.
And as you disperse, the probability of being in any
one of these states goes to 0.
And what that means is that the individual probabilities
of the states is going to 0.
You can also see, not so easily, that you're eventually
going to return to each state with probability 1.
And I'm sorry I didn't give that definition first.
We gave it last time.
If the expected value of the renewal time is less than
infinity, then j is positive recurrent.
If T sub jj, the recurrence time, is a random variable but
it has infinite expectation, then j is not recurrent.
And finally, if none of those things happen, j is transient.
So that we went through last time.
And for p equals 1/2, and in both of these situations, the
probability of being in any state is going to 0.
The expected time of returning is going to infinity.
But with probability 1, you will return eventually.
So in both of these cases, these are both examples of no
recurrence.
Let's say more about positive recurrence and no recurrence.
Suppose, first, that i and j are both recurrent and they
both communicate with each other.
In other words, there's a path from i to j, there's a path
from j to i.
And I want to look at the renewal process
of returns to j.
You've sorted out by now, I think, that recurrence means
exactly what you think it means.
A recurrence means, starting from a state j, there's a
recurrence to j if eventually you come back to j.
And this random variable, the recurrence of random variable,
is the amount of time it takes you to get back to j once
you've been in j.
That's a random variable.
So let's look at the renewal process, starting in j, of
returning to j eventually.
This is one of the things that makes
this whole study awkward.
We have renewal processes when we start at j and we bob back
to j at various periods of time.
If we start in i and we're interested in returns to j,
then we have something called a delayed renewal process.
All the theorems about renewals apply there.
It's a little harder to see what's going on.
It's in the end of chapter four.
You should have read it, at least quickly.
But we're going to avoid those theorems and instead go
directly using the theorems of renewal processes.
But there's still places where the transitions are awkward.
So I can warn you about that.
But the renewal reward theorem, if I look at this
renewal process, I get a renewal every time I
return to state j.
But in that renewal process of returns to state j, what I'm
really interested in is returns to state i, because
what I'm trying to do here is relate how often do you go to
state i with how often do you go to state j?
So we have a little bit of a symmetry in it, because we're
starting in state j, because that gives
us a renewal process.
But now we have this renewal reward process, where we give
ourselves a reward of 1 every time we hit state i.
And we have a renewal every time we hit state j.
So how does that renewal process work?
Well, it's a renewal process just like every other one
we've studied.
It has this peculiar feature here that is a discrete time
renewal process.
And with discrete time renewal processes, as we've seen, you
can save yourself a lot of aggravation by only looking at
these discrete times.
Namely, you only look at integer times.
And now when you only look at integer times--
well, whether you look at integer times or not--
this is the fundamental theorem of renewal rewards.
If you look at the limit as t goes to infinity, there's the
integral of the rewards you pick up.
For this discrete case, this is just a summation of the
rewards that you get.
This summation here by the theorem is equal to the
expected number of rewards within one renewal period.
Namely, this is the expected number of recurrences of state
i per state j.
So in between each occurrence of state j, what's the
expected number of i's that I hit?
And that's the number that it is.
We don't know what that number is, but we could calculate it
if we wanted to.
It's not a limit of anything.
Well, it's a sort of a limit, but not very much of a limit
that's well defined.
And the theorem says that this integral is equal to that
expected value divided by the expected recurrence
time of T sub jj.
Now, we argue that this is, in fact, the number of
occurrences of state i.
So it's in the limit.
It's 1 over the expected value of the recurrence
time to state i.
So what this says is the 1 over the recurrence time to
state i is equal to the expected number of recurrences
to state i per state j divided by the expected
time in state j.
If you think about that for a minute, it's something which
is intuitively obvious.
I mean, you look at this long sequences of things.
You keep hitting j's every once in a while.
And then what you do, is you count all of
the i's that occur.
So now looking at it this way, you're going to count the
number of i's that occur in between each j.
You can't have a simultaneous i and j.
The state is o or j.
So for each recurrence period, you count the number of i's
that occur.
And what this is then saying, is the expected time between
i's is equal to the expected time between j's divided by--
if I turn this equation upside down, the expected time
between i's is equal to the expected time between j's
divided by the expected number of i's per j.
What else would you expect?
It has to be that way, right?
But this says that it indeed, is that way.
Mathematics is sometimes confusing with countable-state
chains as we've seen.
OK, so the theorem then says for i and j recurrent, either
both are positive-recurrent or both are null-recurrent.
So this is adding to the theorem we had earlier.
The theorem we had earlier says that all states within a
class are either recurrent or they're transient.
This now divides the ones that are recurrent into two
subsets, those that are null-recurrent and those that
are positive-recurrent.
It says that for states within a class, either all of them
are recurrent or all of them are--
all of them are positive-recurrent or all of
them are null-recurrent.
And this theorem shows it because this theorem says
there has to be an expected number of occurrences of state
i between each occurrence of state j.
Why is that?
Because there has to be a path from i to j.
And there has to be a path that doesn't go through i.
Because if you have a path from i that goes back to i and
then off to j, there's also this path from i to j.
So there's a path from i to j.
There's a path from j to i that does not go through i.
That has positive probability because paths are only defined
over transitions with positive probability.
So this quantity is always positive if you're talking
about two states in the same class.
So what this relationship says, along with the fact that
it's a very nice and convenient relationship--
I almost put it in the quiz for finite
state and Markov chains.
And you cam be happy I didn't, because proving it takes a
little more agility than what one might
expect at this point.
The theorem then says that if i and j are recurrent, either
both are positive-recurrent or both are null-recurrent, what
the overall theorem then says is that for every class of
states, either all of them are transient, all of them are
null-recurrent, or all of them are positive-recurrent.
And that's sort of a convenient relationship.
You can't have some states that you never get to.
Or you only get to with an infinite recurrence time in a
class and others that you keep coming back to all the time.
If there's a path from one to the other, then they have to
work the same way.
This sort of makes it obvious why that is.
And this is too sensitive.
OK.
OK , so now we want to look at steady state for
positive-recurrent chain.
Do you remember that when we looked at finite state in
Markov chains, we did all this classification stuff, and then
we went into all this matrix stuff?
And the outcome of the matrix stuff, the most important
things, were that there is a steady state.
There's always a set of probabilities such that if you
start the chain in those probabilities, the chain stays
in those probabilities.
There's always a set of pi sub i's, which are probabilities.
They all sum to 1.
They're all non-negative.
And each of them satisfy the relationship, the probability
that you're in state j at time t is equal to the probability
that you're in state i at time t minus 1 times the
probability of going from state i to j.
This is completely familiar from finite-state chains.
And this is exactly the same for
countable-state and Markov chains.
The only question is, it's now not at all sure that that
equation has a solution anymore.
And unfortunately, you can't use matrix theory to prove
that it has a solution.
So we have to find some other way of doing it.
So we look in our toolbox, which we developed
throughout the term.
And there's only one obvious thing to try, and
it's renewal theory.
So we use renewal theory.
We then want to have one other definition, which you'll see
throughout the rest of the term and every time you start
reading about Markov chains.
When you read about Markov chains in queuing kinds of
situations, which are the kinds of things that occur all
over the place, almost all of those Markov chains are
countable-state Markov chains.
And therefore, you need a convenient word to talk about
a class of states where all of the states in that class
communicate with each other.
And irreducible is the definition that we use.
An irreducible Markov chain is a Markov chain in which all
pairs of states communicate with each other.
And before, when we were talking about finite-state
Markov chains, if all states communicated with each other,
then they were are recurrent.
You had a recurrent Markov chain, end of story.
Now we've seen that you can have a Markov chain where all
the states communicate with each other.
We just had these two examples--
these two examples here where they all
communicate with each other.
But depending on what p and q are, they're either transition
transient, or they're positive-recurrent, or they're
null-recurrent.
The first one can't even be positive-recurrent, but it can
be recurrent.
And the bottom one can also be positive-recurrent.
So any Markov chain where all the states communicate with
each other--
there's a path from everything to everything else--
which is the usual situation, is called an irreducible
Markov chain.
An irreducible can now be positive-recurrent,
null-recurrent, or transient.
All the states in an irreducible Markov chain have
to be transient, or all of them have to be
positive-recurrent, or all of them have to be
null-recurrent.
You can't share these qualities over
an irreducible chain.
That's what this last theorem just said.
OK, so if a steady state exists--
namely if the solution to those equations exist, and if
the probability that X sub 0 equals i is equal to pi i.
And incidentally, in the version that got handed out,
that equation there was a little bit garbled.
That one.
Said the probability that X sub 0 was equal to pi i, which
doesn't make any sense.
If a steady-state exists and you start out in
steady-state--
namely, the starting state X sub 0 is in state i with
probability pi sub i by for every i, this is the same
trick we played for
finite-state and Markov chains.
As we go through this, I will try to explain what's the same
and what's different.
And this is completely the same.
So there's nothing new here.
Then, this situation of being in steady-state persists from
one unit of time to the next.
Namely, if you start out in steady-state, then the
probability that X sub 1 is equal to j is equal to the sum
over i of pi sub i.
That's the probability that X sub 0 is equal to i.
Times P sub i j, which by the steady-state equations, is
equal to pi sub j.
So you start out in steady-state.
After one transition, you're in steady-state again.
You're in steady-state at time 1.
Guess what, you're in state time 2.
You're in steady-state again.
And you stay in steady-state forever.
So when you iterate, the probability that you're in
state j at time X sub n is equal to pi sub j also.
This is assuming that you started out in steady-state.
So again, we need some new notation here.
Let's let N sub j of tilde be the number of visits to j in
the period 0 to t starting in steady-state.
Namely, if you start in state j, we get a renewal process to
talk about the returns to state j.
If we start in steady-state, then this first return to
state j is going to have a different set of probabilities
than all subsequent returns to state j.
So N sub j of t, tilde is now not a renewal process, but a
delayed renewal process.
So we have to deal with it a little bit differently.
But it's a very nice thing because for all t, the
expected number of returns to state j over t transitions is
equal to n times pi sub j.
Pi sub j is the probability that you will be in state j at
any time n.
And it stays the same for every n.
So if we look at the expected number of times we hit state
j, it's exactly equal to n times pi sub j.
And again, here's this awkward thing about
renewals and Markov.
Yes?
AUDIENCE: So is that sort of like an ensemble average--
PROFESSOR: Yes.
AUDIENCE: Or is the time average [INAUDIBLE]?
PROFESSOR: Well, it's an ensemble average and it's a
time average.
But the thing we're working with here is the
fact there's a time--
is the fact that it's an ensemble average, yes.
But it's convenient because it's an
exact ensemble average.
Usually, with renewal processes, things are ugly
until you start getting into the limit zone.
Here, everything is nice and clean all the time.
So we start out in steady-state and we get this
beautiful result.
It's starting in steady-state.
The expected number of visit to state j by time n--
oh, this is interesting.
That t there should be n obviously.
Well, since we have t's everywhere else, that n there
should probably be t also.
So you can fix it whichever way you want.
n's and t's are the same.
I mean, for the purposes of this lecture, let all t's be
n's and let all n's be t's.
This works for some things.
This starts in steady state, stays in steady state.
It doesn't work for renewals because it's a delayed renewal
process, so you can't talk about a renewal process
starting in state j, because you don't know that it starts
in state j.
So sometimes we want to deal with this.
Sometimes we want to deal with this.
This is the number of returns to t starting in state j.
This is the number of returns to state j over 0 to t if we
start in steady-state.
Here's a useful hack, which you can use a lot of the time.
Look at what N sub i j of t is.
It's the number of times you hit state j
starting in state i.
So let's look at it as you go for while, you hit state j for
the first time.
After hitting state j for the first time, you then go
through a number of repetitions of state j.
But after that first time you hit state j, you have a
renewal process starting then.
In other words, you have a delayed renewal process up to
the first renewal.
After that, you have all the statistics
of a renewal process.
So the idea then is N sub i j of t is 1.
Counts 1 for the first visit to j, if there are any.
Plus, N sub i j of t minus 1 for all the subsequent
recurrences from j to j.
Thus, when you look at the expected values of this, the
expected value of N sub i j of t is less than or equal to 1
for this first recurrence, for this first visit, plus the
expected value of N sub j j of some number smaller than t.
But N sub j j of t grows with t.
It's a number of visits over some interval.
And as the interval gets bigger and bigger, the number
of visits can't shrink.
So you just put the t there to make it an upper bound.
And then, when you look at starting in steady-state, what
you get is the sum overall starting states pi sub i of
the expected value of N sub i j of t.
And this is less than or equal to 1 plus the expected value
of N sub j j of t also.
So this says you can always get from N tilde of t to N sub
j j of t, by just giving up this term 1
here as an upper bound.
If you don't like that proof--
and it's not really a proof.
If you try to make it a proof, it gets kind of ugly.
It's part of the proof of theorem 4 in the text, which
is even more ugly.
Because it's mathematically clean with equations, but you
don't get any idea of why it's true from looking at it.
This you know why it's true from looking at it, but you're
not quite sure that it satisfies the equations that
you would like.
I am trying to move you from being totally dependent on
equations to being more dependent on ideas like this,
where you can see what's going on.
But I'm also urging you, after you see what's going on, to
have a way to put the equations in to see that
you're absolutely right with it.
OK, now, we come to the major theorem of countable-state and
Markov chains.
It's sort of the crucial thing that everything
else is based on.
I mean, everything beyond what we've already done.
For any irreducible Markov chain--
in other words, for any Markov chain where all the states
communicate with each other, the steady-state equations
have a solution if and only if the states are
positive-recurrent.
Now, remember, either all the states are positive-recurrent
or none of them are.
So there's nothing confusing there.
If all the states are positive-recurrent, then there
is a steady-state solution.
There is a solution to those equations.
And if the set of states are transient, or null-recurrent,
then there isn't a solution to all those equations.
If a solution exists, then the probability, the steady-state
probability is state i is 1 over the main recurrence time
to state i.
This is a relationship that we established by using renewal
theory for finite-state and Markov chains.
We're just coming back to it here.
One thing which is important here is that pi sub i is
greater than 0 for all i.
This is a property we had for finite-state
Markov chains also.
But it's a good deal more surprising here.
When you have a countable number of states, saying that
every one of them has a positive probability is--
I don't think it's entirely intuitive.
If you think about it for a long time,
it's sort of intuitive.
But it's the kind of intuitive thing that really pushes your
intuition into understanding what's going on.
So let's give a Pf of this, of the only if part.
And I will warn you about reading the
proof in the notes.
It's ugly because it just goes through a bunch of logical
relationships and equations.
You have no idea of where it's going or why.
And finally, at the end it says, QED.
I went through it.
It's correct.
But damned if I know why.
And so, anyway, that has to be rewritten.
But, anyway here's the Pf.
Start out by assuming that the steady-state equations exist.
We want to show positive-recurrence.
Pick any j and any t.
Pick any state and any time.
pi sub j is equal to the expected value of N sub j
tilde of t.
That we chose for any Markov chain at all.
If you start out in steady-state, you stay in
steady-state.
So under the assumption that we're in steady-state--
under the assumption that we start out in steady-state, we
stay in steady-state.
This pi sub j times t has to be the expected value of the
number of recurrences to state j over t time units.
And what we showed on the last slide--
you must have realized I was doing this for some reason.
This is less than or equal to 1 plus the expected recurrence
time of state j.
So pi sub j is less than or equal to 1 over t times this
expected recurrence time for state j.
And if we go to the limit as t goes to infinity, this 1 over
t dribbles away to nothingness.
So this is less than or equal to the limit of expected value
of N sub j j of t over t.
What is that?
That's the expected number, long-term rate of
visits to state j.
It's what we've shown as equal to 1 over the expected renewal
time of state j.
Now, if the sum of the pi sub j's is equal to 1, remember
what happens when you sum a countable set of numbers.
If all of them are 0, then no matter how many of them you
sum, you have 0.
And when you go to the limit, you still have 0.
So when you sum a set of countable set of non-negative
numbers, you have to have a limit.
Because it's non-decreasing.
And that sum is equal to 1.
Then somewhere along the line, you've got to find the
positive probability.
One of the [INAUDIBLE]
has to be positive.
I mean, this is almost an amusing proof because you work
so hard to prove that one of them is positive.
And then, almost for free, you get the fact that all of them
have to be positive.
So some pi j is greater than 0.
If pi j is less than or equal to this, thus the limit as t
approaches infinity of the expected value of N sub j j of
t over t is greater than 0 for that j, which says j has to be
positive-recurrent.
Which says all the states have to be positive-recurrent
because we've already shown that.
So all the states are positive-recurrent.
Then you still have to show that this inequality here is
equality, and you've got to do that by playing around with
summing up these things.
Something has been left out, we have to sum
those up over j.
And that's another mess.
I'm not going to do it here in class.
But just sort of see why this happened.
Yeah?
AUDIENCE: [INAUDIBLE].
Why do you have to show the equality?
PROFESSOR: Why do I have to the equality?
Because if I want to show that all of the pi sub i's are
positive, how do I show that?
All I've done is started out with an arbitrary--
oh, I've started out with an arbitrary j and
an arbitrary t.
Because I got the fact that this was positive-recurrent by
arguing that at least one of the pi sub
j's had to be positive.
From this I can argue that they're all
positive-recurrent, which tells me that this number is
greater than 0.
But that doesn't show me that this number is greater than 0.
But it is.
I mean, it's all right.
It all works out.
But not quite in such a simple way as you would hope.
OK, so now let's go back to what we called birth-death
chains, but look at a slightly more general version of them.
These are things that you--
I mean, queuing theory is built on these things.
Everything in queuing theory.
Or not everything, but all the things that come from a
Poisson kind of background.
All of these somehow look at the birth-death chains.
And the way a birth-death chain works is you have
arbitrary self-loops.
You have positive probabilities going from each
state to the next state up.
You have positive probabilities going from the
higher state to the lower state.
All transitions are limited from--
i can only go to i plus 1, or i, for i minus 1.
You can't make big jumps.
You can only make jumps of one step.
And other than that, it's completely general.
OK, now we go through an interesting argument.
We look at an arbitrary state i.
And for this arbitrary state i, like i equals 2, we look at
the number of transitions that go from 2 to 3.
And the number transitions that go from 3 to 2 for any
old sample path whatsoever.
And for any sample path, the number of
transitions that go up--
if we start down there, before you can come back,
you've got to go up.
So if you're on that side, you have one more up transition
than you have down transition.
If you're on that side, you have the same number of up
transitions and down transitions.
So that as you look over a longer and longer time, the
number of up transitions is effectively the same as the
number of down transitions.
If you have a steady-state, pi sub i is the fraction of time
you're in state i.
pi sub i times p sub i is the fraction of time you're going
from state i to state i plus 1.
And pi sub i by plus 1 times q sub i plus 1 is the fraction
of time you're going from state i plus 1
down to state i.
What we've just argued by the fact that sample path averages
and ensemble averages have to be equal is that pi sub i
times p sub i is equal to pi sub i plus 1 times
q sub i plus 1.
In the next slide, I will talk about whether to
believe that or not.
For the moment, let's say we believe it.
And from this equation, we see that the steady-state
probability of i plus 1 is equal to the steady-state
probability of i times p sub i over q sub i plus 1.
It says that the steady-state probability of each pi is
determined by the steady-state probability of the state
underneath it.
So you just go up.
You can calculate the steady-state of each, the
probability of each if you know the probability of the
state below it.
So if you recurse on this, pi sub i plus 1 is equal to pi
sub i times this ratio is equal to pi sub i minus 1
times this ratio times p sub i minus 1 over q sub i is equal
to pi sub i minus 2 times this triple of things.
It tells you that what you want to do is define row sub i
as the difference of these two probabilities, namely rob i,
for any state i, is the ratio of that probability to that
probability.
And this equation then turns into pi sub i plus 1 equals pi
sub i times row sub i.
If you put all those things together--
if you just paste them one after the other, the way I was
suggesting--
what you get is pi sub i is equal to pi sub 0 times this
product of terms.
The product of terms looks a little ugly.
Why don't I care about that very much?
Well, because usually, when you have a chain like this,
all the Ps are the same and all the Qs are the same--
or all the Ps are the same for some point beyond someplace,
they're are different before that.
There's always some structure to make life easy for you.
Oh, that's my computer.
It's telling me what time it is.
I'm sorry.
OK.
So pi sub i is this.
We then have to calculate pi sub 0.
Pi sub 0 is then 1 divided by the sum of all the
probabilities is pi sub 0 times all those other things.
It's 1 plus the sum here.
And now if you don't believe what I did here, and I don't
blame you for being a little bit skeptical.
If you don't believe this, then you look at this and you
say, OK, I can now go back and look at the steady state
equations themselves and I can plug this into the steady
state equations themselves.
And you will immediately see that this solution satisfies
the steady state equations.
OK.
Oh, damn.
Excuse my language.
OK.
So we have our birth-death chain with all these
transitions here.
We have our solution to it.
Note that the solution is only a function of these rows.
It's only a function of the ratio of p sub i to
Q sub i plus 1.
It doesn't depend on those self loops at all.
Isn't that peculiar?
Completely independent of what those self loops are.
Well, you'll see later that it's not totally independent
of it, but it's essentially independent of it.
And you think about that for a while and suddenly it's not
that confusing because those equations have come from
looking at up transitions and down transitions.
By looking at an up transition and a down transition at one
place here, it tells you something about the fraction
of time you're over there and the fraction of time you're
down there if you know what these steady state
probabilities are.
So if you think about it for a bit, you realize that these
steady state probabilities cannot depend that strongly on
what those self loops are.
So this all sort of makes sense.
The next thing is the expression for pi 0--
namely this thing here--
is a product of these terms.
It converges and therefore the chain is positive recurrent
because there is a solution to the steady state equation.
It converges if the row sub i's are
asymptotically less than 1.
So for example, if the row sub i's--
beyond i equals 100--
are bounded by, say, 0.9, then these terms have to go to 0
rapidly after i equals 100 and this product has to converge.
I say essentially here of all these particular cases where
the row sub i's are very close to 1, and they're converging
very slowly to 1 and who knows.
But for most of the things we do, these row sub i's are
strictly less than 1 as you move up.
And it says that you have to have steady state
probabilities.
So for most birth-death chains, it's almost immediate
to establish whether it's recurrent, positive recurrent,
or not positive recurrent.
And we'll talk more about that when we get into Markov
processes, but that's enough of it for now.
Comment on methodology.
We could check the renewal results carefully, because
what we're doing here is assuming something rather
peculiar about time averages and ensemble averages.
And sometimes you have to worry about those things, but
here, we don't have to worry about it because we have this
major theorem which tells us if steady state
probabilities exist--
and they exist because they satisfy these equations--
then you have positive recurrence.
So it says the methodology to use is not to get involved in
any deep theory, but just to see if these
equations are satisfied.
Again, good mathematicians are lazy--
good engineers are even lazier.
That's my motto of the day.
And finally, birth-death chains are going to be
particularly useful in queuing where the births are arrivals
and the deaths are departures.
OK.
Now we come to reversibility.
I'm glad we're coming to that towards the end of the lecture
because reversibility is something which I don't think
any of you guys even--
and I think this is a pretty smart class--
but I've never seen anybody who understands reversibility
the first time they think about it.
It's a very peculiar concept and the results coming from it
are peculiar, and we will have to live with it for a while.
But let's start out with the easy things--
just with a definition of what a Markov chain is.
This top equation here says that the probability of a
whole bunch of states--
X sub n plus k down to X sub n plus 1 given the stated time,
n, down to the stated time 0.
Because of the Markov condition, that has to be
equal to the probability of these terms
just given X sub n.
Namely, if you know what X sub n is, for the future, you
don't have to know what any of those previous states are.
You get that directly from where we started with the
Markov chains-- the probability of X sub n plus 1,
given all this stuff, and then you just add the
other things onto it.
Now, if you define A plus as any event which is defined in
terms of X sub n plus 1, X of n plus 2, and so forth up, and
if you define A minus as anything which is a function
of X sub n minus 1, X sub n minus 2, down to X sub 0, then
what this equations says is that the
probability of any A plus--
given X sub n and A minus--
is equal to the probability of A plus given X sub n.
And this hasn't gotten hard yet.
If you think this is hard, just wait.
If we now multiply this by the probability of A
minus given X sub n--
and what I'm trying to get at is, how do you reason about
the probabilities of earlier states
given the present state?
We're used to proceeding in time.
We're used to looking at the past for telling
what the future is.
And every once and a while, you want to look at the future
and predict what the past had to be.
It's probably more important to talk about the future given
the past, because sometimes you don't know
what the future is.
But mathematically, you have to sort that out.
So if we multiply this equation by the probability of
A minus, given X sub n, we don't know what
that is, but it exists.
It's a defined conditional probability.
Then what we get is the probability of A plus and A
minus, given X sub n, is equal to the probability of A plus,
given X sub n, times the probability of A
minus, given X sub n.
So that the probability of the future and the past, given
what's happening now, is equal to the probability of the
future, given what's happening now, times the probability the
past, given what's happening now.
Which may be a more interesting way of looking at
past and future and present than this totally
asymmetric way here.
This is a nice, symmetric way of looking at it.
And as soon as you see that this has to be true, then you
can turn around and write this the opposite way, and you see
that the probability of A minus, given X sub n, and A
plus is equal to the probability of A
minus given X sub n.
Which says that the probability of the past, given
X sub n and the future, is equal to the probability of
the past just given X sub n.
You can go from past the future or you can go from
future to past.
And incidentally, if you people have trouble trying to
think of the past and the future as
being symmetric animals--
and I do too--
everything we do with time can also be done on a line going
from left to right, or it can be done on a line
going from bottom up.
Going from bottom up, it's hard to say that this is
symmetric to this.
If you look at it on a line going from left to right, it's
kind of easy to see that this is symmetric between left to
right and right to left.
So every time you get confused about these arguments, put
them on a line and argue right to left and left to right
instead of earlier and later.
Because mathematically, it's the same thing, but it's
easier to see these symmetries.
And now, if you think of A minus as being X sub n minus
1, and you think of A plus as being X sub n plus 1, X sub n
plus 2 and so forth up, what this equation says is that the
probability of the last state in the past, given the state
now and everything in the future, is equal to the
probability of the last state in the past given X sub n.
Now, this isn't reversibility.
I'm not saying that these are special process.
This is true for any Markov chain in the world.
These relationships are always true.
This is one reason why many people view this as the real
Markov condition, as opposed to any of these other things.
They say that three events have a Markov condition
between them if there's one of them which is
in between the other.
Where you can say that the probability of the left one,
given the middle one, times the right one, given the
middle one, is equal to the probability of the left and
the right given the middle.
It says that the past and the future, given the present, are
independent of each other.
It says that as soon as you know what the present is,
everything down there is independent of
everything up there.
That's a pretty powerful condition.
And you'll see that we can do an awful lot with it, so it's
going to be important.
OK.
So let's go on with that.
By Bayes rule--
and incidentally, this is why Bayes got into so much trouble
with the other statisticians in the world.
Because the other statisticians in the world
really got emotionally upset at the idea of talking about
the past given the future.
That was almost an attack on their religion as well as all
the mathematics they knew and everything else they knew.
It was really hitting them below the belt, so to speak.
So they didn't like this.
But now, we've recognized that Bayes' Law is just the
consequence of the axioms of probability, and there's
nothing strange about it.
You write down these conditional probabilities and
that's sitting there, facing you.
But what it says here is that the probability of the state
at time n minus 1, given the state of time n, is equal to
the probability of the state of time n, given n minus 1,
times the probability of X n minus 1 divided by the
probability of X n.
In other words, you put this over in this side, and it says
the probability of X n times the probability of X n minus 1
given X n is that probability up there.
It says that the probability of A given B times the
probability of B is equal to the probability of A times the
probability of B given A. And that's just the definition of
a conditional probability, nothing more.
OK.
If the forward chain is in a steady state, then the
probability that X sub n minus 1 equals j, given X sub n
equals i, is pji times pi sub j divided by pi sub i.
These probabilities become just probabilities which
depend on i but not on n.
Now what's going on here is when you look at this
equation, it looks peculiar because normally with a Markov
chain, we start out at time 0 with some assumed probability
distribution.
And as soon as you start out with some assumed probability
distribution at time 0 and you start talking about the past
condition on the future, it gets very sticky.
Because when you talk about the past condition on the
future, you can only go back to time equals 0, and you know
what's happening down there because you have some
established probabilities at 0.
So what it says is in this equation here, it says that
the Markov chain, defined by this rule--
I guess I ought to back and look at the
previous slide for that.
This is saying the probability of the state at time n minus
1, conditional on the entire future, is equal to the
probability of X sub n minus 1 just given X sub n.
This is the Markov condition, but it's the Markov condition
turned around.
Usually we talk about the next state given the previous state
and everything before that.
Here, we're talking about the previous state given
everything after that.
So this really is the Markov condition on what we might
view as a backward chain.
But to be a Markov chain, these transition probabilities
have to be independent of n.
The transition probabilities are not going to be
independent of n if you have these arbitrary probabilities
at time 0 lousing everything up.
So you can get around this in two ways.
One way to get around it is to say let's restrict our
attention to positive recurrent processes which are
starting out in a steady state.
And if we start out in a steady state, then these
probabilities here--
looking at these probabilities here--
if you go from here down to here, you'll find out that
this does not depend on n.
And if you have an initial state which is something other
than steady state, then these will depend on it.
Let me put this down in the next chain up.
The probability of X sub n minus 1 given X sub n is going
to be independent of n if this is independent of n, which it
is, because we have a homogeneous Markov chain.
And this is independent of n and this is independent of n.
Now, this will just be the probability of pi sub i if X
sub n minus 1 is equal to i.
And this will be pi sub i if probability of X sub n is
equal to i.
So this and this will be independent of n if in fact we
start out in a steady state.
In other words, it won't be.
So what we're doing here is we normally think of a Markov
chain starting out at time 0 because how else can you get
it started?
And we think of it in forward time, and then we say, well,
we want to make it homogeneous, because we want
to make it always do the same thing in the future otherwise
it doesn't really look much like the a Markov chain.
So what we're saying is that this backward chain-- we have
backward probabilities defined now--
the backward probabilities are homogeneous if the forward
probabilities start in a steady state.
You could probably make a similar statement but say the
forward probabilities are homogeneous if the backward
probabilities start in a steady state.
But I don't know when you're going to start.
You're going to have to start it sometime in the future, and
that gets too philosophical to understand.
OK.
If we think of the chain as starting in a steady state at
time minus infinity, these are also the equations of the
homogeneous Markov chain.
We can start at time minus infinity wherever we want to--
it doesn't make any difference--
because by the time we get to state 0, we will be in steady
state, and the whole range of where we want to look at
things will be in steady state.
OK.
So aside from this issue about starting at 0 and steady state
and things like that, what we've really shown here is
that you can look at a Markov chain either going forward or
going backward.
Or look at it going rightward or going leftward.
And that's really pretty important.
OK.
That still doesn't say anything about it being
reversible.
What reversibility is--
it comes from looking at this equation here.
This says what the transition probabilities are, going
backwards, and this is the transition
probabilities going forward.
These are the steady state probabilities.
And if we define P star of ji as a backward transition
probabilities--
namely, the probability that at this time or in stage A--
given that in this next time, which to us, is the previous
time, we're in state i, is the probability of going in a
backward direction from j to i.
This gets into whether this is P star of ij or P star of ij.
But I did check it carefully, so it has to be right.
So anyway, when you substitute this in for this, the
conditions that you get is pi sub i times P star of ij is
equal to pi j times P of ji.
These are the same equations that we had for
a birth-death chain.
But now, we're not talking about birth-death chains.
Now we're talking about any old chain.
Yeah?
AUDIENCE: Doesn't this only make sense for positive
recurring chains?
PROFESSOR: Yes.
Sorry.
I should keep emphasizing that, because it only makes
sense when you can define the steady state probabilities.
Yes.
The steady state probabilities are necessary in order to even
define this P star of ji.
But once you have that steady state condition, and once you
know what the steady state probabilities are, then you
can calculate backward probabilities, you can
calculate forward probabilities, and this is a
very simple relationship that they satisfy.
It makes sense because this is a normal form.
You look at state transition probabilities and you look at
the probability of being in one state and then the
probability of going to the next state.
And the question is the next state back there
or is it over there?
And if it's a star, then it means it's back there.
And then we define a chain as being reversible if P star of
ij is equal to P sub ij, for all i and all j.
And what that means is that all birth-death chains are
reversible.
And now let me show you what that means.
If we look at arrivals and departures for a birth-death
change, sometimes you go in a self loop, so you don't go up
and you don't go down.
Other times you either go down or you go up.
We have arrivals coming in.
Arrivals correspond to upper transitions, departures
correspond to downward transitions so that when you
look at it in a normal way, you start out at time 0.
You're in state 0.
You have an arrival.
Nothing happens for a while.
You have another arrival, but this time, you have a
departure, you have another departure, and you wind up in
state 0 again.
As far states are concerned, you go from
state 0 to state 1.
You stay in state 1.
In other words, this is the difference between arrivals
and departures.
This is what the state is.
You stay in state 1.
Then you go up and you get another arrival, you get a
departure, and then you get a departure,
according to this chain.
Now, let's look at it coming in this way.
When we look at it coming backwards
in time, what happens?
We're going along here, we're in state 0,
suddenly we move up.
If we want to view this as a backward moving Markov chain,
this corresponds to an arrival of something.
This corresponds to another arrival.
This corresponds to a departure.
We go along here with nothing else happening, we get another
departure, and there we are back in a steady state again.
And for any birth-death chain, we can do this.
Because any birth-death chain we can look at as an arrival
and departure process.
We have arrivals, we have departures--
we might have states that go negative.
That would be rather awkward, but we can have that.
But now, if we know that these steady state probabilities
govern the probability of arrivals and the probabilities
of departures, if we know that we have reversibility, then
these events here have the same probability as these
events here.
It means that when we look at things going from right back
to left, it means that the things that we viewed as
departures here look like arrivals.
And what we're going to do next time is use that to prove
Burke's Theorem, which says using this idea that if you
look at the process of departures in a birth-death
chain where arrivals are all with probability P and
departures are all with probability Q, then you get
this nice set of probabilities for arrivals and departures.
Arrivals are independent of everything else--
same probability at every unit of time.
Departures are the same way.
But when you're looking from left to right, you can only
get departures when your state is greater than 0.
When you're coming in looking this way, these things that
looked like departures before are looking like arrivals.
These arrivals form a Bernoulli Process, and the
Bernoulli Process says that given the future, the
probability of a departure, at any instant of time, is
independent of everything in the future.
Now, that is not intuitive.
If you think it's intuitive, go back and think again.
Because it's not.
But anyway, I'm going to stop here.
You have this to mull over.
We will try to sort it out a little more next time.
This is something that's going to take your cooperation to
sort it out, also.