WEBVTT
00:00:00.530 --> 00:00:02.960
The following content is
provided under a Creative
00:00:02.960 --> 00:00:04.370
Commons license.
00:00:04.370 --> 00:00:07.410
Your support will help MIT
OpenCourseWare continue to
00:00:07.410 --> 00:00:11.060
offer high quality educational
resources for free.
00:00:11.060 --> 00:00:13.960
To make a donation or view
additional materials from
00:00:13.960 --> 00:00:17.890
hundreds of MIT courses, visit
MIT OpenCourseWare at
00:00:17.890 --> 00:00:19.140
ocw.mit.edu.
00:00:23.810 --> 00:00:24.270
PROFESSOR: OK.
00:00:24.270 --> 00:00:27.020
I guess we might as
well get started.
00:00:27.020 --> 00:00:32.729
We talked a little bit about
Markov processes last week.
00:00:32.729 --> 00:00:37.300
I want to review a little bit of
what we did then, and then
00:00:37.300 --> 00:00:39.570
move on pretty quickly.
00:00:39.570 --> 00:00:44.090
This is a rather strange
chapter, because
00:00:44.090 --> 00:00:46.620
it's full of notation.
00:00:46.620 --> 00:00:50.350
When I first started reviewing
it myself after not having
00:00:50.350 --> 00:00:54.550
looked at it for about a year,
I was horrified by the amount
00:00:54.550 --> 00:00:56.130
of notation.
00:00:56.130 --> 00:00:59.490
And then I realized, what we're
doing in this chapter is
00:00:59.490 --> 00:01:02.750
putting together all the
stuff we've learned
00:01:02.750 --> 00:01:04.330
from Poisson processes.
00:01:04.330 --> 00:01:08.460
There are Poisson processes
all through this
00:01:08.460 --> 00:01:12.400
Markov chains renewals.
00:01:12.400 --> 00:01:15.670
And all three of these, with
all their notation, are all
00:01:15.670 --> 00:01:17.080
sitting here.
00:01:17.080 --> 00:01:21.390
And then we get into new things,
also, so I apologize
00:01:21.390 --> 00:01:22.720
for the notation.
00:01:22.720 --> 00:01:24.910
I'm going to try to keep
it down to the
00:01:24.910 --> 00:01:27.480
minimal amount this time.
00:01:27.480 --> 00:01:33.060
And see if we can sort of get
to the bottom line as easily
00:01:33.060 --> 00:01:38.350
as we can, so that when you do
the exercises and read the
00:01:38.350 --> 00:01:41.690
notes, you can go back in
fill in some of the
00:01:41.690 --> 00:01:44.630
things that are missing.
00:01:44.630 --> 00:01:44.980
OK.
00:01:44.980 --> 00:01:49.440
So accountable state
Markov process.
00:01:49.440 --> 00:01:53.090
The easiest way to define it,
and the way we're defining it
00:01:53.090 --> 00:01:54.980
is as an extension
of accountable
00:01:54.980 --> 00:01:56.610
state Markov chain.
00:01:56.610 --> 00:01:58.980
So you start out with
accountable state Markov
00:01:58.980 --> 00:02:03.440
chain, and then along with each
step, say from state x
00:02:03.440 --> 00:02:09.720
sub n to x sub n plus 1, there
is an exponential holding
00:02:09.720 --> 00:02:12.540
time, u sub n plus 1.
00:02:12.540 --> 00:02:15.420
We said that it was a little
strange associating the
00:02:15.420 --> 00:02:18.210
holding time of the
final state rather
00:02:18.210 --> 00:02:20.240
than the initial state.
00:02:20.240 --> 00:02:23.890
But you almost had to do that,
because of many things that
00:02:23.890 --> 00:02:26.660
would get very confusing
otherwise.
00:02:26.660 --> 00:02:28.490
So we're just doing that.
00:02:28.490 --> 00:02:33.860
We start out in state x0,
which is usually given.
00:02:33.860 --> 00:02:37.400
If it's not given,
it can be random.
00:02:37.400 --> 00:02:44.660
And then we go to state x1 after
a waiting time, u sub 1.
00:02:44.660 --> 00:02:48.880
Go from x1 to x2 with a
waiting time u sub 2.
00:02:48.880 --> 00:02:54.640
The waiting time u sub i is a
function of the state we're
00:02:54.640 --> 00:02:58.440
coming from, in the sense that
it's an exponential random
00:02:58.440 --> 00:03:01.820
variable who's rate is
given by the state
00:03:01.820 --> 00:03:03.890
we're coming from.
00:03:03.890 --> 00:03:12.170
So this diagram here gives
you, really, what the
00:03:12.170 --> 00:03:16.490
dependence of this set of
random variables is.
00:03:16.490 --> 00:03:19.650
You have a sequence of random
variables here.
00:03:19.650 --> 00:03:22.190
A sequence of random
variables here.
00:03:22.190 --> 00:03:26.950
Each random variable here
conditional on the initial
00:03:26.950 --> 00:03:30.310
state that it's coming
from, is independent
00:03:30.310 --> 00:03:31.360
of everything else.
00:03:31.360 --> 00:03:35.330
And that's what this dependence
diagram means.
00:03:35.330 --> 00:03:39.670
It's a generalization of what
we talked about before.
00:03:39.670 --> 00:03:42.790
When we talked about Markov
chains before, we just had a
00:03:42.790 --> 00:03:46.350
string of states.
00:03:46.350 --> 00:03:52.540
And each state x sub n is
dependent only on the prior
00:03:52.540 --> 00:03:55.000
state x sub n minus 1.
00:03:55.000 --> 00:03:58.610
And given the prior state
x sub n minus 1, it's
00:03:58.610 --> 00:04:02.650
statistically independent of
all states before that.
00:04:02.650 --> 00:04:08.270
Here we have this more general
situation of a tree.
00:04:08.270 --> 00:04:13.030
I thought I'd better illustrate
this a little more.
00:04:13.030 --> 00:04:17.740
If you have a directed tree, the
dependencies, each random
00:04:17.740 --> 00:04:22.300
variable conditional as parent
is statistically independent
00:04:22.300 --> 00:04:24.130
of all earlier random
variables.
00:04:24.130 --> 00:04:28.450
In other words, this random
variable here conditional on
00:04:28.450 --> 00:04:32.540
this random variable is
statistically independent of
00:04:32.540 --> 00:04:36.480
this, this, this, and this.
00:04:36.480 --> 00:04:39.810
And so you can move forward in
this way, defining each random
00:04:39.810 --> 00:04:44.160
variable as being statistically
independent of
00:04:44.160 --> 00:04:47.550
everything in the past,
conditional on just one
00:04:47.550 --> 00:04:49.750
previous random variable.
00:04:49.750 --> 00:04:54.700
As an example of this, if you
look at the probability of x0,
00:04:54.700 --> 00:05:04.680
x1, x2, and u2, these five
random variables here.
00:05:04.680 --> 00:05:08.750
Probability of x0, probability
of x1 given x0.
00:05:08.750 --> 00:05:11.400
Probability of x2 given x1.
00:05:11.400 --> 00:05:15.020
Probability of u2 given x1.
00:05:15.020 --> 00:05:16.890
You can write that
out in that way.
00:05:16.890 --> 00:05:20.550
From this, you can rewrite this
in any way you want to.
00:05:20.550 --> 00:05:24.000
You take these two equations,
and you can rewrite them as a
00:05:24.000 --> 00:05:28.950
probability of x1 times the
probability of x0 given x1,
00:05:28.950 --> 00:05:31.830
times the probability of
x2 given x1, times the
00:05:31.830 --> 00:05:34.350
probability u2 given x1.
00:05:34.350 --> 00:05:37.950
In other words, what's happening
here is if you
00:05:37.950 --> 00:05:43.060
condition everything on x1,
this random variable here,
00:05:43.060 --> 00:05:47.510
this stuff is statistically
independent of this.
00:05:47.510 --> 00:05:51.280
Is statistically independent
of all of this.
00:05:51.280 --> 00:05:57.180
Given any one node in this
three, given the value of that
00:05:57.180 --> 00:06:01.030
node, everything on every set of
branches coming out from it
00:06:01.030 --> 00:06:02.280
is statistically independent.
00:06:04.870 --> 00:06:08.000
This is a remarkably
useful property.
00:06:08.000 --> 00:06:10.830
This is the Markov property
in general.
00:06:10.830 --> 00:06:13.210
I mean, Markov chains,
we only use the fact
00:06:13.210 --> 00:06:14.590
that it's on a chain.
00:06:14.590 --> 00:06:17.770
In general you use the fact
that it's on a tree.
00:06:17.770 --> 00:06:25.540
And all of this stuff can be
used in remarkable ways.
00:06:25.540 --> 00:06:28.970
I didn't know this until
probably five years ago.
00:06:28.970 --> 00:06:32.390
And suddenly when I realized it,
I think because somebody
00:06:32.390 --> 00:06:33.990
was pointing it out
in a research
00:06:33.990 --> 00:06:35.390
paper they were writing.
00:06:35.390 --> 00:06:39.380
Suddenly all sorts of things
became much, much easier.
00:06:39.380 --> 00:06:46.850
Because everything like the fact
we pointed out before,
00:06:46.850 --> 00:06:51.190
that the past is independent
of the future, given the
00:06:51.190 --> 00:06:55.020
present, that's one
example of this.
00:06:55.020 --> 00:06:57.880
But this is far more
general than that.
00:06:57.880 --> 00:07:02.500
It says that anything on this
tree, if you can start at any
00:07:02.500 --> 00:07:05.470
point on the tree, and
everything going out from
00:07:05.470 --> 00:07:09.050
there is statistically
independent, given this node
00:07:09.050 --> 00:07:10.510
on the tree.
00:07:10.510 --> 00:07:13.400
So that's a valuable thing.
00:07:13.400 --> 00:07:16.720
And so you get your condition
on any node, breaks a tree
00:07:16.720 --> 00:07:18.590
into independent sub trees.
00:07:18.590 --> 00:07:20.920
You can then go on from there
and break it down further and
00:07:20.920 --> 00:07:24.520
further and further, until you
get out to the leaves.
00:07:24.520 --> 00:07:28.750
So this independence property
is really the general thing
00:07:28.750 --> 00:07:31.540
that we refer to when we
say a set of random
00:07:31.540 --> 00:07:34.160
variables are Markov.
00:07:34.160 --> 00:07:34.670
OK.
00:07:34.670 --> 00:07:38.840
The evolution in time with a
Markov process, this diagram I
00:07:38.840 --> 00:07:41.580
find very helpful to
see what's going
00:07:41.580 --> 00:07:43.610
on in a Markov process.
00:07:43.610 --> 00:07:46.090
You have a set of states.
00:07:46.090 --> 00:07:52.440
Initially, you're in a state
x0, the state at time 0 is
00:07:52.440 --> 00:07:54.140
some given value i.
00:07:54.140 --> 00:07:57.300
This is a sample path here.
00:07:57.300 --> 00:08:01.570
The next state we'll say is
j, the next state is a k.
00:08:01.570 --> 00:08:06.890
When you're in state i, there's
some holding time,
00:08:06.890 --> 00:08:11.440
which has rate new
1, new sub i.
00:08:11.440 --> 00:08:14.190
It's an exponential random
variable which tells you how
00:08:14.190 --> 00:08:17.010
long it takes until
this transition.
00:08:17.010 --> 00:08:24.390
This transition occurs
at time s1, which is
00:08:24.390 --> 00:08:26.440
u1 is equal to s1.
00:08:26.440 --> 00:08:29.020
The next transition is at s2.
00:08:29.020 --> 00:08:31.490
Equals u1 plus u2.
00:08:31.490 --> 00:08:36.289
The next transition is at x3,
which is u1 plus u2 plus u3.
00:08:36.289 --> 00:08:39.700
Now you start to see why we've
numbered these holding times
00:08:39.700 --> 00:08:43.260
the way we have, so we can talk
about the times that each
00:08:43.260 --> 00:08:44.995
of these transitions
take place.
00:08:47.930 --> 00:08:50.890
We usually assume that the
embedded Markov chain for a
00:08:50.890 --> 00:08:55.330
Markov process, remember, the
embedded Markov chain now is
00:08:55.330 --> 00:08:57.480
just the Markov chain
itself without these
00:08:57.480 --> 00:08:59.870
holding times on it.
00:08:59.870 --> 00:09:03.070
We assume it has no self
transitions because if you're
00:09:03.070 --> 00:09:06.980
sitting in a state x of t equals
i, and suddenly there's
00:09:06.980 --> 00:09:12.200
a transition back to i again,
and you look at the process in
00:09:12.200 --> 00:09:16.480
terms of x of t, t greater
than or equal to 0.
00:09:16.480 --> 00:09:21.320
The state that you're in at
each time t, what happens?
00:09:21.320 --> 00:09:22.070
You don't see it.
00:09:22.070 --> 00:09:26.200
It's a totally invisible
transition, because you're
00:09:26.200 --> 00:09:28.260
sitting in state i.
00:09:28.260 --> 00:09:32.140
You suddenly have a transition
back to i that takes 0 time.
00:09:32.140 --> 00:09:34.120
So you stay in state i.
00:09:34.120 --> 00:09:37.460
You can put that transition
in or you can take it out.
00:09:37.460 --> 00:09:38.840
It doesn't make any
difference.
00:09:38.840 --> 00:09:43.750
It won't affect the
process at all.
00:09:43.750 --> 00:09:44.860
OK.
00:09:44.860 --> 00:09:50.120
Aside from that issue of these
self transitions, a sample
00:09:50.120 --> 00:09:56.640
path of both x sub n, each of
these x sub 0 equals i, x sub
00:09:56.640 --> 00:09:58.906
1 equals j.
00:09:58.906 --> 00:10:01.810
x sub 2 equals k.
00:10:01.810 --> 00:10:07.630
A sample path as those plus the
holding times specifies
00:10:07.630 --> 00:10:10.450
what x of t is at each
instant of time.
00:10:10.450 --> 00:10:13.840
And if you know what x of t is
at each unit of time, that
00:10:13.840 --> 00:10:16.720
tells you when the transitions
are occurring.
00:10:16.720 --> 00:10:19.100
When you know when the
transitions are occurring, you
00:10:19.100 --> 00:10:21.340
know what the these u's are.
00:10:21.340 --> 00:10:24.250
And when you see what the
transition is into, you know
00:10:24.250 --> 00:10:25.700
what the state is.
00:10:25.700 --> 00:10:30.600
So the description of a Markov
process in terms of the
00:10:30.600 --> 00:10:34.810
process, what we call a process
x sub t for all t
00:10:34.810 --> 00:10:39.540
greater than or equal to 0,
and the set of random
00:10:39.540 --> 00:10:44.110
variables, the embedded Markov
chain, and the holding times
00:10:44.110 --> 00:10:45.960
are both equivalent
to each other.
00:10:45.960 --> 00:10:48.950
This shouldn't be any surprise
to you by now, because every
00:10:48.950 --> 00:10:52.220
process we've talked to,
we've talked about.
00:10:52.220 --> 00:10:54.380
We've described in
the same way.
00:10:54.380 --> 00:10:59.840
We described the Poisson process
in multiple ways.
00:10:59.840 --> 00:11:03.480
We described Markov chains
in multiple ways.
00:11:03.480 --> 00:11:07.360
We described renewal processes
in multiple ways.
00:11:07.360 --> 00:11:11.770
And this is just another
example of that.
00:11:11.770 --> 00:11:15.920
You use whatever description you
want to after you've shown
00:11:15.920 --> 00:11:18.960
they're all equivalent.
00:11:18.960 --> 00:11:21.090
So there's really nothing
new here.
00:11:21.090 --> 00:11:22.340
Or is there?
00:11:24.760 --> 00:11:29.590
Who can see what there is about
this relationship, which
00:11:29.590 --> 00:11:33.360
is different from what we've
just been talking about?
00:11:33.360 --> 00:11:34.745
It's using one extra property.
00:11:38.850 --> 00:11:44.640
This is not just a consequence
of this, it also uses
00:11:44.640 --> 00:11:45.450
something else.
00:11:45.450 --> 00:11:48.590
And what else does it use?
00:11:48.590 --> 00:11:53.210
What I'm doing is saying x of
t at this instance of time
00:11:53.210 --> 00:11:59.900
here is dependent on given
x of t is some
00:11:59.900 --> 00:12:03.280
previous time here.
00:12:03.280 --> 00:12:07.310
The state here, given the state
here is independent of
00:12:07.310 --> 00:12:09.920
everything in the past.
00:12:09.920 --> 00:12:11.830
So what else am I using there?
00:12:15.300 --> 00:12:19.740
I'm using the memory looseness
of the Poisson process.
00:12:19.740 --> 00:12:21.900
I'm using the memory
looseness of the
00:12:21.900 --> 00:12:25.470
exponential random variable.
00:12:25.470 --> 00:12:32.400
If I'm given the state here,
and I'm conditioning on the
00:12:32.400 --> 00:12:36.580
state here, this is
an exponential
00:12:36.580 --> 00:12:38.010
random variable in here.
00:12:38.010 --> 00:12:40.950
The time the next transition
is exponential
00:12:40.950 --> 00:12:42.490
given this time here.
00:12:42.490 --> 00:12:45.280
And it doesn't matter
when the previous
00:12:45.280 --> 00:12:46.425
transition took place.
00:12:46.425 --> 00:12:50.940
So if I'm given the state at
this time here, the time to
00:12:50.940 --> 00:12:55.020
the next transition is an
exponential random variable,
00:12:55.020 --> 00:12:58.510
the same distribution as u2.
00:12:58.510 --> 00:13:03.950
So what this says is I'm using
the initial description in
00:13:03.950 --> 00:13:10.000
terms of an embedded Markov
chain plus holding times, and
00:13:10.000 --> 00:13:12.570
I'm adding to that the fact
that the holding times are
00:13:12.570 --> 00:13:15.880
exponential, and therefore
they're memory less.
00:13:15.880 --> 00:13:17.090
OK, that clear to everybody?
00:13:17.090 --> 00:13:20.690
It's vitally important
for all of this.
00:13:20.690 --> 00:13:27.820
Because it's hard to do anything
with Markov processes
00:13:27.820 --> 00:13:33.100
without realizing explicitly
that you're using the fact
00:13:33.100 --> 00:13:36.230
that these random variables
are memory less.
00:13:36.230 --> 00:13:38.710
At the end of this chapter,
there's something called semi
00:13:38.710 --> 00:13:40.700
Markov processes.
00:13:40.700 --> 00:13:43.880
The description is that
semi Markov processes
00:13:43.880 --> 00:13:45.770
are exactly the same.
00:13:45.770 --> 00:13:50.210
Semi Markov chains are exactly
the same as markup processes
00:13:50.210 --> 00:13:56.490
except, these holding times
are not exponential.
00:13:56.490 --> 00:13:58.270
They can be anything.
00:13:58.270 --> 00:14:01.700
And as soon as the holding times
can be anything, the
00:14:01.700 --> 00:14:05.110
process gets so complicated that
you hardly want to talk
00:14:05.110 --> 00:14:06.360
about it anymore.
00:14:09.140 --> 00:14:12.480
So the fact that we have these
exponential holding times is
00:14:12.480 --> 00:14:17.600
really important in terms of
getting this condition here,
00:14:17.600 --> 00:14:21.720
which lets you talk directly
about the process, instead of
00:14:21.720 --> 00:14:23.265
the embedded Markov chain.
00:14:27.120 --> 00:14:31.200
You're going to represent a
Markov process by a graph for
00:14:31.200 --> 00:14:35.560
the embedded Markov chain, and
then you give the rates on top
00:14:35.560 --> 00:14:36.320
of the nodes.
00:14:36.320 --> 00:14:42.290
So if you're in state 0, the
holding time until you enter
00:14:42.290 --> 00:14:47.380
the next state is given as
some, the rate of that
00:14:47.380 --> 00:14:50.030
exponential is given as new 0.
00:14:50.030 --> 00:14:54.330
The rate here is given
as new 1, so forth.
00:14:54.330 --> 00:14:57.940
Ultimately, we're usually
interested in this process, x
00:14:57.940 --> 00:15:01.740
of t, t greater than or equal
to 0, which is the Markov
00:15:01.740 --> 00:15:03.220
process itself.
00:15:03.220 --> 00:15:11.300
x of t is equal to xn for t in
sub n, between sub n and
00:15:11.300 --> 00:15:13.150
s sub n plus 1.
00:15:13.150 --> 00:15:15.880
What does that mean?
00:15:15.880 --> 00:15:18.800
Well, it means at this point,
we're taking the Markov
00:15:18.800 --> 00:15:23.310
process as the fundamental
thing, and we're describing it
00:15:23.310 --> 00:15:27.400
in terms of the nth
state transition.
00:15:27.400 --> 00:15:31.390
But we know that the nth state
transition takes place at time
00:15:31.390 --> 00:15:35.330
s sub n, namely it takes place
at the sum of all of these
00:15:35.330 --> 00:15:39.250
exponential holding times
up until that point.
00:15:39.250 --> 00:15:42.100
And that state stays there until
the next exponential
00:15:42.100 --> 00:15:43.220
holding time.
00:15:43.220 --> 00:15:48.830
So this really gives you the
linkage between the Markov
00:15:48.830 --> 00:15:54.310
process in this expression,
and the markup process in
00:15:54.310 --> 00:15:57.660
terms of this graphical
expression here with the
00:15:57.660 --> 00:16:01.340
embedded chain, and the
exponential holding times.
00:16:04.060 --> 00:16:04.510
OK.
00:16:04.510 --> 00:16:09.320
You can visualize a transition
from one state to another in
00:16:09.320 --> 00:16:12.410
tree very convenient ways.
00:16:12.410 --> 00:16:16.570
And these are ways that we've, I
hope we've really learned to
00:16:16.570 --> 00:16:21.690
think about from looking
at Poisson processes.
00:16:21.690 --> 00:16:26.960
You can visualize this
transition by first using the
00:16:26.960 --> 00:16:32.820
next state by these transition
probabilities in the embedded
00:16:32.820 --> 00:16:34.160
Markov chain.
00:16:34.160 --> 00:16:36.840
And then you choose a transition
time, which is
00:16:36.840 --> 00:16:40.190
exponential with
rate new sub i.
00:16:40.190 --> 00:16:43.510
Equivalently, because it's a
Poisson process, you can
00:16:43.510 --> 00:16:44.840
choose the--
00:16:44.840 --> 00:16:48.490
well, no, because these are
independent given the state.
00:16:48.490 --> 00:16:52.190
You can choose the transition
time first, and then you can
00:16:52.190 --> 00:16:55.480
choose the state, because these
are independent of each
00:16:55.480 --> 00:16:59.520
other conditional on the
state that you're in.
00:16:59.520 --> 00:17:01.970
And finally, equivalently, which
is where the Poisson
00:17:01.970 --> 00:17:05.900
process comes in, a really neat
way to think of Poisson
00:17:05.900 --> 00:17:09.940
processes is to have an
enormously large number of
00:17:09.940 --> 00:17:12.720
Poisson processes running
all the time.
00:17:12.720 --> 00:17:17.030
There's one Poisson process for
every transition in this
00:17:17.030 --> 00:17:19.460
Markov chain.
00:17:19.460 --> 00:17:21.700
So you have accountably infinite
number of Poisson
00:17:21.700 --> 00:17:25.369
processes, which sounds a little
complicated at first.
00:17:25.369 --> 00:17:29.235
But you visualize a Poisson
process for each state pair i
00:17:29.235 --> 00:17:35.480
to j, which has a rate q sub ij,
which is the rate at which
00:17:35.480 --> 00:17:39.960
transitions occur out of
state i times p sub ij.
00:17:39.960 --> 00:17:45.250
This is the rate which, when
you're in state i, you will go
00:17:45.250 --> 00:17:47.890
to state j.
00:17:47.890 --> 00:17:51.120
And this makes use of all this
stuff about splitting and
00:17:51.120 --> 00:17:54.410
combining Poisson processes.
00:17:54.410 --> 00:18:01.810
If you have a Poisson process
which has rate new sub i and
00:18:01.810 --> 00:18:05.710
you split it into a number of
Poisson processes, for each
00:18:05.710 --> 00:18:08.970
next state you might go to,
you're splitting it into
00:18:08.970 --> 00:18:15.070
Poisson processes of rate new
sub i times p sub ij.
00:18:15.070 --> 00:18:19.100
And what's happening there is
there's a little switch.
00:18:19.100 --> 00:18:23.170
The little switch has
probabilities p sub ij, and
00:18:23.170 --> 00:18:30.950
that switch is telling you which
state to go to next.
00:18:30.950 --> 00:18:34.990
All of this is totally
artificial.
00:18:34.990 --> 00:18:39.200
And I hope by this time, you are
comfortable about looking
00:18:39.200 --> 00:18:43.020
at physical things in a totally
artificial way,
00:18:43.020 --> 00:18:47.660
because that's the magic
of mathematics.
00:18:47.660 --> 00:18:50.120
If you didn't have mathematics,
you couldn't look
00:18:50.120 --> 00:18:53.220
at real things in
artificial ways.
00:18:53.220 --> 00:18:56.500
And all the science would
suddenly disappear.
00:18:56.500 --> 00:19:02.700
So what we're doing here is
defining this Markov process
00:19:02.700 --> 00:19:05.600
in this artificial way of all
of these little Poisson
00:19:05.600 --> 00:19:09.460
processes, and we now know
how they all work.
00:19:09.460 --> 00:19:14.070
When the entry to state i, the
next state is the j with an x
00:19:14.070 --> 00:19:16.790
Poisson arrival, according
to q sub ij.
00:19:16.790 --> 00:19:19.900
So all these Poisson processes
are waiting to have an
00:19:19.900 --> 00:19:21.130
arrival come out.
00:19:21.130 --> 00:19:24.140
To have a race, one of
them wins, and you
00:19:24.140 --> 00:19:27.640
go off to that state.
00:19:27.640 --> 00:19:28.500
OK, question.
00:19:28.500 --> 00:19:31.900
What's the conditional
distribution of u1 given that
00:19:31.900 --> 00:19:37.270
x0 i, and x1 equals j?
00:19:37.270 --> 00:19:42.350
And to imagine this, suppose
that there are only two places
00:19:42.350 --> 00:19:46.060
you can go from state 0.
00:19:46.060 --> 00:19:50.560
You can go into state 1
with some very large
00:19:50.560 --> 00:19:53.500
probability, say 0.999.
00:19:53.500 --> 00:19:57.010
Or you can go into state 2 with
some very, very small
00:19:57.010 --> 00:19:59.210
probability.
00:19:59.210 --> 00:20:02.670
And what that means is this
exponential random variable
00:20:02.670 --> 00:20:08.530
going from state 0 into state
2 is a very, very
00:20:08.530 --> 00:20:10.570
slow, random variable.
00:20:10.570 --> 00:20:12.110
It has a very small rate.
00:20:12.110 --> 00:20:15.970
And the exponential random
variable going into state 1 is
00:20:15.970 --> 00:20:19.580
a very, very large
random variable.
00:20:19.580 --> 00:20:24.800
So you would think that if you
go from state 0 the state x2,
00:20:24.800 --> 00:20:29.675
it means it must take a very
long time to get there.
00:20:29.675 --> 00:20:31.400
Well, that's absolutely wrong.
00:20:35.180 --> 00:20:42.440
The time that it takes to go
from x0 to x2 is this random
00:20:42.440 --> 00:20:45.960
variable, u sub i.
00:20:45.960 --> 00:20:53.560
Where u sub i is the state you
happen to be in at this point
00:20:53.560 --> 00:20:54.810
when you're in--
00:21:00.310 --> 00:21:03.350
x0 is the state that
we start in.
00:21:03.350 --> 00:21:05.510
x0 is a random variable.
00:21:05.510 --> 00:21:08.210
It has some value i.
00:21:08.210 --> 00:21:11.665
With this value i, there's an
exponential random variable
00:21:11.665 --> 00:21:13.870
that determines how long
it takes you to
00:21:13.870 --> 00:21:15.640
get to the next state.
00:21:15.640 --> 00:21:19.700
This random variable conditional
on x0 equals i is
00:21:19.700 --> 00:21:23.270
independent of which state
you happen to go to.
00:21:23.270 --> 00:21:28.730
And what that means is that the
conditional distribution
00:21:28.730 --> 00:21:34.960
of u1, given x sub 0 is equal
to i, and x sub 1 equals j.
00:21:34.960 --> 00:21:39.380
If you've had your ears at all
open for the last 10 minutes,
00:21:39.380 --> 00:21:45.710
it is exponential with rate i.
00:21:45.710 --> 00:21:47.630
With rate new sub i.
00:21:50.400 --> 00:21:54.080
These holding times and the
next states you go to are
00:21:54.080 --> 00:21:57.000
independent of each other,
conditional on where you
00:21:57.000 --> 00:21:58.460
happen to be.
00:21:58.460 --> 00:22:02.390
It's the same thing we saw back
in Poisson processes.
00:22:02.390 --> 00:22:04.950
It was confusing as
hell back then.
00:22:04.950 --> 00:22:08.280
It is still confusing as hell.
00:22:08.280 --> 00:22:12.760
If you didn't get it sorted out
in your mind then, go back
00:22:12.760 --> 00:22:16.770
and think about it again now,
and try to get your common
00:22:16.770 --> 00:22:20.010
sense, which tells you when
you go to state 2, it must
00:22:20.010 --> 00:22:23.190
take a long time to get there,
because that exponential
00:22:23.190 --> 00:22:27.720
random variable has a very
long holding time.
00:22:27.720 --> 00:22:30.020
And that just isn't true.
00:22:30.020 --> 00:22:33.210
And it wasn't true when we were
dealing with a Poisson
00:22:33.210 --> 00:22:35.410
process, which got
split either.
00:22:35.410 --> 00:22:38.970
These two things are independent
of each other.
00:22:38.970 --> 00:22:44.840
Intuitively, why that is, I
almost hesitate to try to say
00:22:44.840 --> 00:22:49.950
why it is, because it's such
a tricky statement.
00:22:49.950 --> 00:22:55.010
If you happen to go to state 2
instead of to state 1, what's
00:22:55.010 --> 00:22:59.440
happening is that all of this
time that you're waiting to
00:22:59.440 --> 00:23:04.300
have a state transition, when
you finally have this state
00:23:04.300 --> 00:23:08.800
transition, you then flip a
switch to see which state
00:23:08.800 --> 00:23:10.350
you're going to go to.
00:23:10.350 --> 00:23:13.740
And the fact that it's taken you
a long time to get there
00:23:13.740 --> 00:23:17.680
says nothing whatsoever about
what this switch is doing,
00:23:17.680 --> 00:23:21.020
because that switch is
independent of how long it
00:23:21.020 --> 00:23:24.360
takes you for the switch
to operate.
00:23:24.360 --> 00:23:29.060
And I know.
00:23:29.060 --> 00:23:32.720
It's not entirely intuitive,
and you just have to beat
00:23:32.720 --> 00:23:36.530
yourself on the head until
it becomes intuitive.
00:23:36.530 --> 00:23:38.390
I've beaten myself on
the head until I can
00:23:38.390 --> 00:23:39.630
hardly think straight.
00:23:39.630 --> 00:23:42.680
It still isn't intuitive to me,
but maybe it will become
00:23:42.680 --> 00:23:44.030
intuitive to you.
00:23:44.030 --> 00:23:47.420
I hope so.
00:23:47.420 --> 00:23:51.690
So anyway, this gives the
conditional distribution of u1
00:23:51.690 --> 00:23:54.618
given that x0 is equal to i,
and x1 is equal to-- no.
00:23:58.450 --> 00:24:03.530
This says that the exponential
rate out of state i is equal
00:24:03.530 --> 00:24:07.660
to the sum of the exponential
rates to each of the states we
00:24:07.660 --> 00:24:08.890
might be going to.
00:24:08.890 --> 00:24:10.620
We have to go to some
other state.
00:24:10.620 --> 00:24:14.110
We have no self transitions as
far as we're concerned here.
00:24:14.110 --> 00:24:16.990
Even if we had self transitions,
this formula
00:24:16.990 --> 00:24:19.380
would still be correct, but
it's easier to think of it
00:24:19.380 --> 00:24:21.730
without self transitions.
00:24:21.730 --> 00:24:25.720
p sub ij, this is the
switch probability.
00:24:25.720 --> 00:24:29.950
It's q sub ij divided
by new sub i.
00:24:29.950 --> 00:24:33.360
This is the probability you're
going to go to j given that
00:24:33.360 --> 00:24:35.550
you were in state i.
00:24:35.550 --> 00:24:42.400
The matrix of all of these cues
specifies the matrix of
00:24:42.400 --> 00:24:46.880
all of these p's, and
it specifies new.
00:24:46.880 --> 00:24:49.900
That's what this formula says.
00:24:49.900 --> 00:24:54.190
If I know what q is, I know what
new is, I know what p is.
00:24:54.190 --> 00:24:57.885
And we've already said that if
you know what p is and you
00:24:57.885 --> 00:25:00.700
know what new is, you know
what q sub ij is.
00:25:00.700 --> 00:25:04.320
So these are completely
equivalent representations of
00:25:04.320 --> 00:25:05.140
the same thing.
00:25:05.140 --> 00:25:07.760
You can work with either
one you want to.
00:25:07.760 --> 00:25:12.060
Sometimes one is useful,
sometimes the other is useful.
00:25:12.060 --> 00:25:16.250
If you look at an mm1q,
mm1q, you remember,
00:25:16.250 --> 00:25:18.720
is exponential arrivals.
00:25:18.720 --> 00:25:20.270
Exponential service time.
00:25:20.270 --> 00:25:23.970
The time of the service is
independent of when it
00:25:23.970 --> 00:25:26.450
happens, or who it happens to.
00:25:26.450 --> 00:25:29.770
These service times are just
individual exponential random
00:25:29.770 --> 00:25:31.670
variables of some rate mu.
00:25:31.670 --> 00:25:34.950
The arrivals are the
inter-arrival intervals are
00:25:34.950 --> 00:25:40.980
exponential random variables
of rate lambda.
00:25:40.980 --> 00:25:44.860
So when you're sitting in state
0, the only place you
00:25:44.860 --> 00:25:46.210
can is to state one.
00:25:46.210 --> 00:25:52.260
You're sitting there, and
if you're in state 0,
00:25:52.260 --> 00:25:53.510
the server is idle.
00:25:55.770 --> 00:25:58.880
You're waiting for the first
arrival to occur.
00:25:58.880 --> 00:26:02.140
The next thing that happens has
to be an arrival because
00:26:02.140 --> 00:26:03.850
it can't be a service.
00:26:03.850 --> 00:26:07.430
So the transition here is
with probability 1.
00:26:07.430 --> 00:26:11.010
All these other transitions are
with probability lambda
00:26:11.010 --> 00:26:14.590
divided by lambda plus mu,
because in all of these other
00:26:14.590 --> 00:26:18.720
states, you can get an arrival
or a departure.
00:26:18.720 --> 00:26:21.400
Each of them are exponential
random variables.
00:26:21.400 --> 00:26:25.040
The switch probability that
we're talking about is in
00:26:25.040 --> 00:26:29.300
lambda over lambda
plus mu to go up.
00:26:29.300 --> 00:26:32.540
Mu over lambda plus mu
to go down for all
00:26:32.540 --> 00:26:35.210
states other than 0.
00:26:35.210 --> 00:26:38.400
If you write this, this
is in terms of the
00:26:38.400 --> 00:26:40.520
embedded Markov chain.
00:26:40.520 --> 00:26:43.200
And if you write this in
terms of the transition
00:26:43.200 --> 00:26:46.760
probabilities, it
looks like this.
00:26:46.760 --> 00:26:48.860
Which is simpler?
00:26:48.860 --> 00:26:50.715
Which is more transparent?
00:26:50.715 --> 00:26:53.830
Well this, really, is
what the mm1q is.
00:26:53.830 --> 00:26:55.440
That's where we started
when we started
00:26:55.440 --> 00:26:56.810
talking about mm1q's.
00:27:02.360 --> 00:27:05.880
This in this situation is
certainly far simpler.
00:27:05.880 --> 00:27:08.260
You're giving these transition
probabilities.
00:27:08.260 --> 00:27:11.710
But don't forget that we still
have this embedded Markov
00:27:11.710 --> 00:27:14.330
chain in the background.
00:27:14.330 --> 00:27:17.430
Both these graphs have
the same information.
00:27:17.430 --> 00:27:20.250
Both these graphs have the same
information for every
00:27:20.250 --> 00:27:21.870
Markov process you want
00:27:21.870 --> 00:27:25.740
to talk about OK.
00:27:25.740 --> 00:27:27.040
Let's look at sample time
00:27:27.040 --> 00:27:29.560
approximations to Markov processes.
00:27:33.280 --> 00:27:35.700
And we already did it
in the last chapter.
00:27:35.700 --> 00:27:38.950
We just didn't talk about
it quite so much.
00:27:38.950 --> 00:27:43.580
We quantized time to increments
of delta.
00:27:43.580 --> 00:27:47.550
We viewed all Poisson processes
in a Markov process.
00:27:47.550 --> 00:27:50.570
Remember, we can view all
of these transitions as
00:27:50.570 --> 00:27:53.310
independent Poisson processes
all running
00:27:53.310 --> 00:27:55.890
away at the same time.
00:27:55.890 --> 00:27:59.330
We can view all of these Poisson
processes as Bernoulli
00:27:59.330 --> 00:28:05.455
processes with probability of
a transition from i to j in
00:28:05.455 --> 00:28:10.930
the increment delta, given as
a delta times qij, to first
00:28:10.930 --> 00:28:12.550
order in delta.
00:28:12.550 --> 00:28:17.610
So we can take this Markov
process, turn it into a rather
00:28:17.610 --> 00:28:20.530
strange kind of Bernoulli
process.
00:28:20.530 --> 00:28:24.310
For the M/M/1 queue, all that's
doing is turning into a
00:28:24.310 --> 00:28:26.700
sample time, M/M/1 process.
00:28:26.700 --> 00:28:29.680
And we can sort of think
the same way about
00:28:29.680 --> 00:28:31.820
general Markov processes.
00:28:31.820 --> 00:28:34.620
We'll see when we can
and when we can't.
00:28:34.620 --> 00:28:37.840
Since shrinking Bernoulli goes
to Poisson, we would
00:28:37.840 --> 00:28:42.430
conjecture the limiting Markov
chain as delta goes to 0 goes
00:28:42.430 --> 00:28:43.710
through a Markov process.
00:28:43.710 --> 00:28:46.030
In a sense that X of t is
00:28:46.030 --> 00:28:48.940
approximately equal to X prime.
00:28:48.940 --> 00:28:53.430
This is X prime as in
the Bernoulli domain
00:28:53.430 --> 00:28:55.340
at delta times n.
00:28:58.460 --> 00:29:00.910
You have to put self-transition
into a sample
00:29:00.910 --> 00:29:02.540
time approximation.
00:29:02.540 --> 00:29:06.280
Because if you have a very small
delta, there aren't big
00:29:06.280 --> 00:29:09.760
enough transition probabilities
going out of the
00:29:09.760 --> 00:29:12.880
chain to fill up the
probability space.
00:29:12.880 --> 00:29:15.710
So in most transition, you're
going to just have a
00:29:15.710 --> 00:29:17.210
self-transition.
00:29:17.210 --> 00:29:21.120
So you need a self-transition,
which is 1 minus delta times
00:29:21.120 --> 00:29:25.470
nu sub i, and these transitions
to other states,
00:29:25.470 --> 00:29:28.350
which are delta times
q sub ij.
00:29:28.350 --> 00:29:34.460
This has the advantage that if
you believe this, you can
00:29:34.460 --> 00:29:37.300
ignore everything we're saying
about Poisson processes
00:29:37.300 --> 00:29:39.950
because you already
know all of it.
00:29:39.950 --> 00:29:44.350
We already talked about
sample time processes.
00:29:44.350 --> 00:29:48.720
You can do this for any
old process almost.
00:29:48.720 --> 00:29:52.400
And when you do this for any old
process, you're turning it
00:29:52.400 --> 00:29:55.860
into a Markov change instead
of a Markov process.
00:29:55.860 --> 00:29:59.310
This is the same argument you
tried to use when you were a
00:29:59.310 --> 00:30:02.500
senior in high school or
freshman in college when you
00:30:02.500 --> 00:30:06.850
said, I don't have to learn
calculus, because all it is is
00:30:06.850 --> 00:30:10.180
just taking increments
to be very small and
00:30:10.180 --> 00:30:11.260
looking at a limit.
00:30:11.260 --> 00:30:14.152
So I will just ignore
all that stuff.
00:30:14.152 --> 00:30:15.940
It didn't work there.
00:30:15.940 --> 00:30:17.190
It doesn't work here.
00:30:19.690 --> 00:30:24.630
But it's a good thing to go
every time you get confused,
00:30:24.630 --> 00:30:27.460
because this you can sort
out for yourself.
00:30:27.460 --> 00:30:30.150
There is one problem here.
00:30:30.150 --> 00:30:36.410
When you start making shrieking
delta more and more,
00:30:36.410 --> 00:30:41.340
if you want to get a sample time
approximation, delta has
00:30:41.340 --> 00:30:45.660
to be smaller than 1 over the
maximum of the nu sub i's.
00:30:45.660 --> 00:30:49.600
If it's not smaller than the
maximum of the nu sub i's, the
00:30:49.600 --> 00:30:54.220
self-transition probability
here is
00:30:54.220 --> 00:30:57.110
unfortunately negative.
00:30:57.110 --> 00:31:00.370
And we don't like negative
probabilities.
00:31:00.370 --> 00:31:02.570
So you can't do that.
00:31:02.570 --> 00:31:05.910
If you have a Markov process,
has a countably infinite
00:31:05.910 --> 00:31:10.320
number of states, each of these
nu sub i's are positive.
00:31:10.320 --> 00:31:12.900
But they can approach
0 as a limit.
00:31:12.900 --> 00:31:17.240
As if they approach 0 as a
limit, you cannot describe a
00:31:17.240 --> 00:31:22.520
sample time Markov chain to go
with the Markov process.
00:31:22.520 --> 00:31:25.090
All you can do is truncate
the chain also and
00:31:25.090 --> 00:31:26.090
then see what happens.
00:31:26.090 --> 00:31:29.366
And that's often a good
way to do it.
00:31:29.366 --> 00:31:32.526
OK, so we can always do this
sample time approximation.
00:31:35.030 --> 00:31:42.640
What is nice about the sample
time approximation is that we
00:31:42.640 --> 00:31:46.170
will find it in general, if you
could use the sample time
00:31:46.170 --> 00:31:50.500
approximation, it always gives
you the exact steady state
00:31:50.500 --> 00:31:51.820
probabilities.
00:31:51.820 --> 00:31:56.010
No matter how crude you are in
this approximation, you always
00:31:56.010 --> 00:31:59.420
wind up with the exact rate
values when you're all done.
00:31:59.420 --> 00:32:01.070
I don't know why.
00:32:01.070 --> 00:32:05.210
We will essentially prove
today that that happens.
00:32:05.210 --> 00:32:06.750
But that's a nice thing.
00:32:06.750 --> 00:32:10.030
But it doesn't work with the nu
sub i's approach 0, because
00:32:10.030 --> 00:32:11.550
then you can't get
a sample time
00:32:11.550 --> 00:32:14.090
approximation to start with.
00:32:14.090 --> 00:32:17.820
OK, let's look at the embedded
chain model in the sample time
00:32:17.820 --> 00:32:20.315
model of M/M/1 queue.
00:32:20.315 --> 00:32:23.200
I hate to keep coming back
to the M/M/1 queue.
00:32:23.200 --> 00:32:27.340
But for Markov processes,
they're all so similar to each
00:32:27.340 --> 00:32:32.780
other that you might as well
get very familiar with one
00:32:32.780 --> 00:32:36.290
particular model of them,
because that one particular
00:32:36.290 --> 00:32:39.840
model tells you most of the
distinctions that you have to
00:32:39.840 --> 00:32:41.090
be careful about.
00:32:43.990 --> 00:32:44.560
Let's see.
00:32:44.560 --> 00:32:45.170
What is this?
00:32:45.170 --> 00:32:47.240
This is the embedded chain
model that we've
00:32:47.240 --> 00:32:48.680
talked about before.
00:32:48.680 --> 00:32:51.210
When you're in state 0,
the only place you can
00:32:51.210 --> 00:32:53.190
go is to state 1.
00:32:53.190 --> 00:32:57.730
When you're in state 1, you
can go down with some
00:32:57.730 --> 00:32:58.840
probability.
00:32:58.840 --> 00:33:01.590
You can go up with
some probability.
00:33:01.590 --> 00:33:05.760
And since these probabilities
have to add up to 1, it's mu
00:33:05.760 --> 00:33:09.600
over lambda plus mu and lambda
over lambda plus mu, and the
00:33:09.600 --> 00:33:13.310
same thing forever after.
00:33:13.310 --> 00:33:17.650
If we're dealing with the sample
time model, what we
00:33:17.650 --> 00:33:22.750
wind up with is we start
out with qij,
00:33:22.750 --> 00:33:24.770
which is lambda here.
00:33:27.610 --> 00:33:32.070
The time it takes to get from
state 0 to make a transition,
00:33:32.070 --> 00:33:36.100
the only place you can make a
transition to is state 1.
00:33:36.100 --> 00:33:38.730
You make those transitions
at rate lambda.
00:33:38.730 --> 00:33:43.470
So the sample time model has
this transition and discrete
00:33:43.470 --> 00:33:47.300
time with probability lambda
delta, this transition with
00:33:47.300 --> 00:33:50.520
probability mu delta
and so forth up.
00:33:50.520 --> 00:33:54.380
You need these self-transitions
in order to
00:33:54.380 --> 00:33:56.580
make things add up correctly.
00:33:56.580 --> 00:34:02.080
The steady state for the
embedded chain is pi sub 0
00:34:02.080 --> 00:34:04.880
equals 1 minus rho over 2.
00:34:04.880 --> 00:34:07.970
How do I know that?
00:34:07.970 --> 00:34:10.550
You just have to use
algebra for that.
00:34:10.550 --> 00:34:11.800
But it's very easy.
00:34:24.850 --> 00:34:28.659
I'm going to have to get
three of these things.
00:34:28.659 --> 00:34:31.880
OK, any time you have a
birth-death chain, you can
00:34:31.880 --> 00:34:33.389
find the steady state
probabilities.
00:34:54.030 --> 00:34:56.880
The probability of going this
way is equal to the
00:34:56.880 --> 00:34:58.870
probability of going this way.
00:34:58.870 --> 00:35:02.195
If you add in the probability
of the steady state that
00:35:02.195 --> 00:35:05.440
you're concerned with, steady
state transition this way, the
00:35:05.440 --> 00:35:07.300
same as the probability
of a steady state
00:35:07.300 --> 00:35:08.670
transition this way.
00:35:08.670 --> 00:35:11.630
And you remember, the reason for
this is in a birth-death
00:35:11.630 --> 00:35:16.450
chain, the total number of
transitions from here to here
00:35:16.450 --> 00:35:19.820
has to be within 1 of the number
of transitions from
00:35:19.820 --> 00:35:21.650
here to there.
00:35:21.650 --> 00:35:25.670
So that if steady state
means anything--
00:35:25.670 --> 00:35:30.440
and if these of long-term sample
space probabilities
00:35:30.440 --> 00:35:33.380
with probability 1
mean anything--
00:35:33.380 --> 00:35:35.320
this has to be true.
00:35:35.320 --> 00:35:38.600
So when you do that, this
is what you get here.
00:35:38.600 --> 00:35:42.100
This is a strange 1
minus rho over 2.
00:35:42.100 --> 00:35:45.660
It's strange because of this
strange probability one here,
00:35:45.660 --> 00:35:49.180
and this strange probability mu
over lambda plus mu here.
00:35:49.180 --> 00:35:52.530
And otherwise, everything is
symmetric, so it looks the
00:35:52.530 --> 00:35:55.490
same as this one here.
00:35:55.490 --> 00:35:58.730
For this one, the steady state
for the sample time doesn't
00:35:58.730 --> 00:36:00.420
depend on delta.
00:36:00.420 --> 00:36:04.590
And it's pi sub i prime equals 1
minus rho times rho to the i
00:36:04.590 --> 00:36:06.820
where rho equals
lambda over mu.
00:36:06.820 --> 00:36:08.620
This is what we did before.
00:36:08.620 --> 00:36:13.450
And what we found is since
transitions this way have to
00:36:13.450 --> 00:36:18.690
equal transitions this way,
these self-transitions don't
00:36:18.690 --> 00:36:21.130
make any difference here.
00:36:21.130 --> 00:36:25.200
And you get the same answer
no matter what delta is.
00:36:25.200 --> 00:36:31.070
And therefore you have pretty
much a conviction, which can't
00:36:31.070 --> 00:36:34.770
totally rely on, that you can go
to the limit as delta goes
00:36:34.770 --> 00:36:38.800
to 0 and find out what is going
on in the actual Markov
00:36:38.800 --> 00:36:40.250
process itself.
00:36:40.250 --> 00:36:45.900
You'll be very surprised with
this result if this were not
00:36:45.900 --> 00:36:50.530
the result for steady state
probabilities in some sense
00:36:50.530 --> 00:36:52.860
for the Markov process.
00:36:52.860 --> 00:36:59.790
However, the embedded chain
probabilities and these
00:36:59.790 --> 00:37:03.650
probabilities down here
are not the same.
00:37:03.650 --> 00:37:07.730
What's the difference
between them?
00:37:07.730 --> 00:37:11.700
For the embedded chain, what
you're talking about is the
00:37:11.700 --> 00:37:15.210
ratio of transitions
that go from one
00:37:15.210 --> 00:37:17.850
state to another state.
00:37:17.850 --> 00:37:21.130
When you're dealing with the
process, what you're talking
00:37:21.130 --> 00:37:26.250
about is the probability that
you will be in one state.
00:37:26.250 --> 00:37:29.030
If when you get in one state
you stay there for a long
00:37:29.030 --> 00:37:36.770
time, because the rate of
transitions out of that state
00:37:36.770 --> 00:37:40.260
is very small, so you're going
to stay there for a long time.
00:37:40.260 --> 00:37:43.940
That enhances the probability
of being in that state.
00:37:43.940 --> 00:37:48.220
You see that right here,
because pi 0 is 1
00:37:48.220 --> 00:37:50.410
minus rho over 2.
00:37:50.410 --> 00:37:58.060
And pi 0 prime is 1 minus rho.
00:37:58.060 --> 00:37:59.246
It's bigger.
00:37:59.246 --> 00:38:02.270
And it's bigger because you're
going to stay there longer,
00:38:02.270 --> 00:38:04.830
because the rate of getting out
of there is not as big as
00:38:04.830 --> 00:38:07.210
it was before.
00:38:07.210 --> 00:38:09.810
So the steady state
probabilities in the embedded
00:38:09.810 --> 00:38:15.000
chain and the steady state
probabilities and the sample
00:38:15.000 --> 00:38:17.700
time approximation
are different.
00:38:17.700 --> 00:38:20.410
And the steady state
probabilities and the sample
00:38:20.410 --> 00:38:25.340
time approximation are the same,
when you go to the limit
00:38:25.340 --> 00:38:28.080
of infinitely fine
sample time.
00:38:28.080 --> 00:38:31.440
Now what we have to do is go
back and look at renewal
00:38:31.440 --> 00:38:35.340
theory and all those that and
actually convince ourselves
00:38:35.340 --> 00:38:36.590
that this works.
00:38:39.819 --> 00:38:43.300
OK, so here we have renewals
for Markov processes.
00:38:46.380 --> 00:38:48.100
And what have we done so far?
00:38:48.100 --> 00:38:50.400
We've been looking at
the Poisson process.
00:38:50.400 --> 00:38:53.150
We've been looking
at Markov chains.
00:38:53.150 --> 00:38:57.180
And we've been trying to refer
to this new kind of process.
00:38:57.180 --> 00:39:01.460
Now we bring in the last
actor, renewal theory.
00:39:01.460 --> 00:39:06.170
And as usual, Poisson processes
gives you the easy
00:39:06.170 --> 00:39:07.900
way to look at a problem.
00:39:07.900 --> 00:39:11.070
Markov chains gives you a way
to look at the problem, when
00:39:11.070 --> 00:39:13.680
you'd rather write equations
and think about it.
00:39:13.680 --> 00:39:16.130
And we renewal theory gives
you the way to look at the
00:39:16.130 --> 00:39:20.030
problem when you really are a
glutton for punishment, and
00:39:20.030 --> 00:39:23.200
you want to spend a lot of
time thinking about it.
00:39:23.200 --> 00:39:25.040
And you don't want to write any
equations, or you don't
00:39:25.040 --> 00:39:26.900
want to write many equation.
00:39:26.900 --> 00:39:31.240
OK, an irreducible Markov
process is a Markov process
00:39:31.240 --> 00:39:35.350
for which the embedded Markov
chain is irreducible.
00:39:35.350 --> 00:39:38.840
Remember that an irreducible a
Markov chain is one where all
00:39:38.840 --> 00:39:42.270
states are in the same class.
00:39:42.270 --> 00:39:47.970
We saw that irreducible Markov
chains when we had a countably
00:39:47.970 --> 00:39:51.122
infinite number of states, that
they could be transient,
00:39:51.122 --> 00:39:53.300
the state simply wanders
off with high
00:39:53.300 --> 00:39:55.300
probability, never to return.
00:39:55.300 --> 00:39:58.670
If you have an M/M/1 queue,
and the expect the service
00:39:58.670 --> 00:40:04.340
time is bigger than the
expected time between
00:40:04.340 --> 00:40:08.970
arrivals, then gradually
the queue builds up.
00:40:08.970 --> 00:40:12.480
The queue keeps getting longer
and longer as time goes on.
00:40:12.480 --> 00:40:14.470
There isn't any steady state.
00:40:14.470 --> 00:40:17.200
Looked at another way, the
steady state probabilities are
00:40:17.200 --> 00:40:21.510
always 0, if you want to
just calculate them.
00:40:21.510 --> 00:40:26.700
So we're going to see the
irreducible Markov processes
00:40:26.700 --> 00:40:30.670
can have even more bizarre
behavior than these Markov
00:40:30.670 --> 00:40:32.220
chains can.
00:40:32.220 --> 00:40:36.590
And part of that more bizarre
behavior is infinitely many
00:40:36.590 --> 00:40:40.790
transitions in a finite time.
00:40:40.790 --> 00:40:44.540
I mean, how do you talk about
steady state when you have an
00:40:44.540 --> 00:40:48.680
infinite number of transitions
and a finite time?
00:40:48.680 --> 00:40:50.950
I mean, essentially, the
Markov process is
00:40:50.950 --> 00:40:53.060
blowing up on you.
00:40:53.060 --> 00:40:55.900
Transitions get more
and more frequent.
00:40:55.900 --> 00:40:57.790
They go off to infinity.
00:40:57.790 --> 00:40:58.890
What do you do after that?
00:40:58.890 --> 00:41:00.650
I don't know.
00:41:00.650 --> 00:41:02.010
I can write these equations.
00:41:02.010 --> 00:41:03.860
I can solve these equations.
00:41:03.860 --> 00:41:05.840
But they don't mean anything.
00:41:05.840 --> 00:41:10.170
In other words sometimes,
talking about steady state, we
00:41:10.170 --> 00:41:13.120
usually write equations
for steady state.
00:41:13.120 --> 00:41:17.720
But as we saw with countable
state Markov chains, steady
00:41:17.720 --> 00:41:19.980
state doesn't always
exist there.
00:41:19.980 --> 00:41:23.520
There it evidenced itself with
steady state probabilities
00:41:23.520 --> 00:41:27.290
that were equal to 0, which
said that as time went on,
00:41:27.290 --> 00:41:30.420
things just got very diffused
or things wandered off to
00:41:30.420 --> 00:41:32.830
infinity or something
like that.
00:41:32.830 --> 00:41:37.170
Here it's this much worse thing,
where in fact you get
00:41:37.170 --> 00:41:40.070
an infinite number of
transitions very fast.
00:41:40.070 --> 00:41:45.000
And we'll see how that happens
a little later.
00:41:45.000 --> 00:41:48.770
You might have a transition
rate which goes down to 0.
00:41:48.770 --> 00:41:53.030
The process is chugging along
and gets slower, and slower,
00:41:53.030 --> 00:41:57.070
and slower, and pretty soon
nothing happens anymore.
00:41:57.070 --> 00:42:01.250
Well, always something happens
if you wait long enough, but
00:42:01.250 --> 00:42:04.000
as you wait longer and
longer, things happen
00:42:04.000 --> 00:42:07.520
more and more slowly.
00:42:07.520 --> 00:42:10.750
So we'll see all of these
things, and we'll see
00:42:10.750 --> 00:42:12.420
how this comes out.
00:42:12.420 --> 00:42:14.650
OK, let's review briefly
accountable
00:42:14.650 --> 00:42:17.060
state Markov chains--
00:42:17.060 --> 00:42:19.730
an irreducible, that means
everything can talk to
00:42:19.730 --> 00:42:24.550
everything else; is positive
recurrent if and only if the
00:42:24.550 --> 00:42:30.110
steady state equations, pi sub
j equals the sum of pi
00:42:30.110 --> 00:42:31.510
sub i, p sub ij.
00:42:31.510 --> 00:42:32.940
Remember what this is.
00:42:36.310 --> 00:42:40.120
If you're in steady state, the
probability of being in a
00:42:40.120 --> 00:42:42.730
state j is supposed
to be pi sub j.
00:42:42.730 --> 00:42:46.470
The probability of being in a
state is supposed to be equal
00:42:46.470 --> 00:42:49.840
then to the sum of the
probabilities of being in
00:42:49.840 --> 00:42:52.310
another state and going
to that state.
00:42:52.310 --> 00:42:54.100
That's the way it has to
be if you're going to
00:42:54.100 --> 00:42:55.610
have a steady state.
00:42:55.610 --> 00:42:57.220
So this is necessary.
00:42:57.220 --> 00:43:00.200
The pi sub j's have to be
greater than or equal to 0.
00:43:00.200 --> 00:43:03.410
And the sum of the pi sub
j's is equal to 1.
00:43:03.410 --> 00:43:04.710
It has a solution.
00:43:04.710 --> 00:43:08.970
If this has a solution, it's
unique, and if pi sub i is
00:43:08.970 --> 00:43:15.680
greater than 0 for all i, if
it's positive recurrent.
00:43:15.680 --> 00:43:17.730
We saw that if it wasn't
positive recurrent, other
00:43:17.730 --> 00:43:19.340
things could happen.
00:43:19.340 --> 00:43:23.950
Also the number of visits, n
sub ij of n, remember in a
00:43:23.950 --> 00:43:29.220
Markov chain, what we talked
about when we used renewal
00:43:29.220 --> 00:43:34.200
theory was the number of visits
over a particular
00:43:34.200 --> 00:43:39.010
number transitions
from i to j.
00:43:39.010 --> 00:43:45.390
n sub ij of n is the number
of times we hit j's
00:43:45.390 --> 00:43:46.640
in the first n trials.
00:43:51.350 --> 00:43:54.120
I always do this.
00:43:54.120 --> 00:43:59.815
Please take that n sub ij of n
and write 1 over n times n sub
00:43:59.815 --> 00:44:02.340
ij of n is equal to pi j.
00:44:02.340 --> 00:44:03.550
You all know that.
00:44:03.550 --> 00:44:06.140
I know it too.
00:44:06.140 --> 00:44:10.260
I don't know why it always gets
left off of my slides.
00:44:10.260 --> 00:44:14.190
Now, we guessed for a Markov
process the fraction of time
00:44:14.190 --> 00:44:19.337
in state j should be p sub j
equals pi sub j over a nu sub
00:44:19.337 --> 00:44:25.440
j divided by the sum over i
of pi sub i over nu sub i.
00:44:25.440 --> 00:44:27.590
Perhaps I should say I guessed
that because I
00:44:27.590 --> 00:44:30.190
already know it.
00:44:30.190 --> 00:44:35.270
I want to indicate to you why
if you didn't know anything
00:44:35.270 --> 00:44:38.510
and if you weren't suspicious
by this time, you would make
00:44:38.510 --> 00:44:41.450
that guess, OK?
00:44:41.450 --> 00:44:44.490
We had this embedded
Markov chain.
00:44:44.490 --> 00:44:47.480
Over a very long period
of time, the number of
00:44:47.480 --> 00:44:53.570
transitions into state i is
going to be the number of
00:44:53.570 --> 00:44:58.090
transitions into state i
divided by n is going
00:44:58.090 --> 00:44:59.460
to be pi sub i.
00:44:59.460 --> 00:45:02.940
That's what this equation here
says, or what it would say if
00:45:02.940 --> 00:45:05.830
I had written it correctly.
00:45:05.830 --> 00:45:09.720
Now, each time we get to pi
sub i, we're going to stay
00:45:09.720 --> 00:45:11.220
there for a while.
00:45:11.220 --> 00:45:16.270
The holding time in state
i is proportional to
00:45:16.270 --> 00:45:18.490
1 over nu sub i.
00:45:18.490 --> 00:45:22.230
The rate of the next transition
is nu sub i.
00:45:22.230 --> 00:45:26.272
So the expected holding time is
going to be 1 over nu sub
00:45:26.272 --> 00:45:31.150
i, which says that the fraction
of time that we're
00:45:31.150 --> 00:45:38.510
actually in state i should be
proportional to the number of
00:45:38.510 --> 00:45:43.250
times we go into state j times
the expected holding
00:45:43.250 --> 00:45:44.930
time in state j.
00:45:44.930 --> 00:45:49.350
Now when you write p sub j
equals pi sub j over nu sub j,
00:45:49.350 --> 00:45:51.740
you have a constant there
which is missing.
00:45:51.740 --> 00:45:55.790
Because what we're doing is
we're amortizing this over
00:45:55.790 --> 00:45:57.420
some long period of time.
00:45:57.420 --> 00:46:00.520
And we don't know what the
constant of amortization is.
00:46:00.520 --> 00:46:04.120
But these probability
should add up to 1.
00:46:04.120 --> 00:46:08.610
If life is at all fair to us,
the fraction of time that we
00:46:08.610 --> 00:46:13.030
spent in each state j should be
some number which adds up
00:46:13.030 --> 00:46:15.620
to 1 as we sum over j.
00:46:15.620 --> 00:46:18.230
So this is just a normalization
factor that you
00:46:18.230 --> 00:46:21.910
need to make the p sub
j's sum up to 1.
00:46:21.910 --> 00:46:25.800
Now what this means physically,
and why it appears
00:46:25.800 --> 00:46:28.630
here, is something we have to go
through some more analysis.
00:46:28.630 --> 00:46:30.880
But this is what we
would guess if we
00:46:30.880 --> 00:46:32.270
didn't know any better.
00:46:32.270 --> 00:46:34.860
And in fact, it's pretty
much true.
00:46:34.860 --> 00:46:36.960
It's not always true, but
it's pretty much true.
00:46:39.710 --> 00:46:42.200
So now let's use renewal
theory to actually see
00:46:42.200 --> 00:46:43.790
what's going on.
00:46:43.790 --> 00:46:47.440
And here's where we need a
little more notation even.
00:46:47.440 --> 00:46:54.140
Let n sub i of t be the number
of transitions between 0 and t
00:46:54.140 --> 00:46:58.060
for a Markov process starting
in state i.
00:46:58.060 --> 00:47:01.140
I can't talk about the number of
transitions if I don't say
00:47:01.140 --> 00:47:03.610
what state we start in, because
then I don't really
00:47:03.610 --> 00:47:06.120
have a random variable.
00:47:06.120 --> 00:47:10.110
I could say let's start and
steady state, and that seems
00:47:10.110 --> 00:47:12.570
very, very appealing.
00:47:12.570 --> 00:47:14.920
I've tried to do that many
times, because it would
00:47:14.920 --> 00:47:17.470
simplify all these theorems.
00:47:17.470 --> 00:47:21.330
And it just doesn't
work, believe me.
00:47:21.330 --> 00:47:24.840
So let's take the extra pain
of saying let's start
00:47:24.840 --> 00:47:26.200
in some state i.
00:47:26.200 --> 00:47:29.220
We don't know what it is, but
we'll just assume there is
00:47:29.220 --> 00:47:31.850
some state.
00:47:31.850 --> 00:47:36.420
And the theorem says that the
limit of M sub i of t is equal
00:47:36.420 --> 00:47:37.620
to infinity.
00:47:37.620 --> 00:47:39.950
Here I don't have a 1 over
t in front of it.
00:47:39.950 --> 00:47:42.470
I've written incorrectly.
00:47:42.470 --> 00:47:45.210
And this is a very technical
theorem.
00:47:45.210 --> 00:47:48.220
We proved the same kind of
technical theorem when we were
00:47:48.220 --> 00:47:50.860
talking about Markov chains,
if you remember.
00:47:50.860 --> 00:47:52.820
We were talking about
Markov chains.
00:47:52.820 --> 00:47:59.910
We said that in some sense, an
infinite number of transitions
00:47:59.910 --> 00:48:02.520
into each one of the states
had to occur.
00:48:02.520 --> 00:48:05.560
The same kind of proof
is the proof here.
00:48:05.560 --> 00:48:07.610
It had the same kind of proof
when we were talking about
00:48:07.610 --> 00:48:09.440
renewal theory.
00:48:09.440 --> 00:48:14.670
What is going on is given any
state, given the state the
00:48:14.670 --> 00:48:19.100
transition has to occur within
finite time, because there
00:48:19.100 --> 00:48:21.620
some exponential holding
time there.
00:48:21.620 --> 00:48:26.810
So the expected amount of time
for the next transition is 1
00:48:26.810 --> 00:48:28.260
over nu sub i.
00:48:28.260 --> 00:48:32.920
And that's finite for every
i in the chain.
00:48:32.920 --> 00:48:36.540
And therefore, as you go from
one state to another, as the
00:48:36.540 --> 00:48:41.430
frog those jumping from one lily
pad to another, and each
00:48:41.430 --> 00:48:44.700
lily pad that it jumps on
there's some expected time
00:48:44.700 --> 00:48:48.410
before it moves, and therefore
assuming that it keeps moving
00:48:48.410 --> 00:48:53.500
forever and doesn't die, which
is what we assume with these
00:48:53.500 --> 00:48:57.360
Markov chains, it will
eventually go through an
00:48:57.360 --> 00:49:00.220
infinite number of steps.
00:49:00.220 --> 00:49:02.730
And the proof of that
is in the text.
00:49:02.730 --> 00:49:06.190
But it's exactly the same proof
as you've seen several
00:49:06.190 --> 00:49:09.530
times before for renewal process
in countable state
00:49:09.530 --> 00:49:11.930
Markov chains.
00:49:11.930 --> 00:49:19.160
Next theorem is to say let M sub
ij of t be the number of
00:49:19.160 --> 00:49:23.930
transitions to j, starting
in state i.
00:49:23.930 --> 00:49:26.680
We can't get rid of the
starting state.
00:49:26.680 --> 00:49:28.960
Somehow we have to
keep it in there.
00:49:28.960 --> 00:49:33.010
We have some confidence that
it's not important, that it
00:49:33.010 --> 00:49:33.850
shouldn't be there.
00:49:33.850 --> 00:49:36.840
And we're going to see it
disappear very shortly.
00:49:36.840 --> 00:49:39.970
But we have to keep it there
for the time being.
00:49:39.970 --> 00:49:45.800
So if the embedded chain is
recurrent, then n sub ij of t
00:49:45.800 --> 00:49:49.276
is a delayed renewal process.
00:49:49.276 --> 00:49:51.180
And we sort of know that.
00:49:51.180 --> 00:49:53.890
Essentially, transitions
keep occurring.
00:49:53.890 --> 00:49:57.800
So renewals in the state
j must keep occurring.
00:49:57.800 --> 00:50:01.700
And therefore, any time you go
to state j, the amount of time
00:50:01.700 --> 00:50:07.080
that it takes until you get
there again, is finite.
00:50:07.080 --> 00:50:09.970
We're not selling it to expect
at time is finite.
00:50:09.970 --> 00:50:11.520
Expect time might be infinite.
00:50:11.520 --> 00:50:13.850
We'll see lots of cases
where it is.
00:50:13.850 --> 00:50:16.210
But you've got to get
there eventually.
00:50:16.210 --> 00:50:22.190
That's the same kind of thing we
saw for renewal theory when
00:50:22.190 --> 00:50:26.610
we had renewals and the things
that could happen eventually
00:50:26.610 --> 00:50:28.910
they did happen.
00:50:28.910 --> 00:50:32.310
I don't know whether any of
you are old enough to have
00:50:32.310 --> 00:50:34.620
heard about Murphy's Law.
00:50:34.620 --> 00:50:38.240
Murphy was an Irish
American to whom
00:50:38.240 --> 00:50:40.080
awful things kept happening.
00:50:40.080 --> 00:50:43.020
And Murphy's Law says that
if something awful
00:50:43.020 --> 00:50:45.160
can happen, it will.
00:50:45.160 --> 00:50:49.080
This says if this can happen,
eventually it will happen.
00:50:49.080 --> 00:50:50.740
It doesn't say it will
happen immediately.
00:50:50.740 --> 00:50:53.850
But it says it will
happen eventually.
00:50:53.850 --> 00:50:57.340
You can think of this as
Murphy's Law, if you want to,
00:50:57.340 --> 00:50:58.590
if you're familiar with that.
00:51:01.820 --> 00:51:04.710
So we want to talk about steady
state for irreducible
00:51:04.710 --> 00:51:06.720
Markov processes.
00:51:06.720 --> 00:51:12.300
Now, let p sub j of i be the
time average fraction of time
00:51:12.300 --> 00:51:15.760
in state j for the delayed
renewal process.
00:51:15.760 --> 00:51:19.910
Remember we talked about p sub
j in terms of these sample
00:51:19.910 --> 00:51:21.050
time Markov change.
00:51:21.050 --> 00:51:23.730
And we talked about them a
little bit in terms of
00:51:23.730 --> 00:51:27.570
imagining how long you would
stay in state j if you were in
00:51:27.570 --> 00:51:29.890
some kind of steady state.
00:51:29.890 --> 00:51:32.780
Here we want to talk
about p sub j of i.
00:51:35.340 --> 00:51:37.680
In terms of strong law of
large numbers kinds of
00:51:37.680 --> 00:51:41.010
results, we want to look at the
sample path average and
00:51:41.010 --> 00:51:44.890
see the convergence with
probability one.
00:51:44.890 --> 00:51:50.330
OK, so p sub j of i is a time
average fraction of time in
00:51:50.330 --> 00:51:54.470
state j for the delayed
renewal process.
00:51:54.470 --> 00:51:57.750
Remember we said that delay
renewal processes were really
00:51:57.750 --> 00:51:59.820
the same as renewal processes.
00:51:59.820 --> 00:52:02.850
You just had this first renewal,
which really didn't
00:52:02.850 --> 00:52:05.490
make any difference.
00:52:05.490 --> 00:52:09.400
And so p sub j of i is going
to be the limit as t
00:52:09.400 --> 00:52:14.870
approaches infinity of the
reward that we pick up forever
00:52:14.870 --> 00:52:16.690
of being in state j.
00:52:16.690 --> 00:52:19.900
You get one unit of reward
whenever you're in state j, 0
00:52:19.900 --> 00:52:21.850
units when you're
anywhere else.
00:52:21.850 --> 00:52:25.520
So this is the time average
fraction of time
00:52:25.520 --> 00:52:26.890
you're in state j.
00:52:26.890 --> 00:52:29.870
This is divided by t.
00:52:29.870 --> 00:52:31.940
And the assumption is
you start in time i.
00:52:31.940 --> 00:52:34.200
So that affects this
a little bit.
00:52:34.200 --> 00:52:38.840
The picture here says whenever
you go to state j, you're
00:52:38.840 --> 00:52:47.100
going to stay in state j for
some holding time U sub n.
00:52:47.100 --> 00:52:51.562
Then you go back to 0 reward
until the next time you went
00:52:51.562 --> 00:52:52.980
to state j.
00:52:52.980 --> 00:52:55.260
Then you jump up
to reward of 1.
00:52:55.260 --> 00:52:58.000
You stay there for your holding
time until you get
00:52:58.000 --> 00:53:02.600
into some other state, a
and that keeps going
00:53:02.600 --> 00:53:04.260
on forever and ever.
00:53:04.260 --> 00:53:07.090
What does the delayed renewal
reward theorem say?
00:53:16.600 --> 00:53:22.560
It says that the expected reward
over time is going to
00:53:22.560 --> 00:53:27.040
be the expect to reward in one
renewal divided by the
00:53:27.040 --> 00:53:29.450
expected length of
the renewal path.
00:53:29.450 --> 00:53:33.500
It says it's going to be
expected value of U sub n
00:53:33.500 --> 00:53:39.130
divided by the expected time
that you stay in state j.
00:53:39.130 --> 00:53:45.490
So it's 1 over nu sub j times
the expected time
00:53:45.490 --> 00:53:46.890
you're in a state j.
00:53:49.590 --> 00:53:52.980
That's a really neat result
that connects this steady
00:53:52.980 --> 00:53:54.660
state probability.
00:53:54.660 --> 00:53:59.720
Excuse my impolite computer.
00:53:59.720 --> 00:54:04.090
This relates to fraction of time
you're in state i to the
00:54:04.090 --> 00:54:06.570
expected delay in state j.
00:54:06.570 --> 00:54:09.970
It's one of those maddening
things where you say that's
00:54:09.970 --> 00:54:13.290
great, but I don't know how to
find either of those things.
00:54:13.290 --> 00:54:14.590
So we go on.
00:54:14.590 --> 00:54:17.420
We will find them.
00:54:17.420 --> 00:54:21.180
And what we will find
is W sub j.
00:54:21.180 --> 00:54:26.780
If we can find W sub j, we'll
also know p sub j.
00:54:26.780 --> 00:54:30.510
M sub ij of t is delayed
renewal process.
00:54:30.510 --> 00:54:34.590
The strong law for renewal says
the limit as t approaches
00:54:34.590 --> 00:54:41.950
infinity of Mi j of t over t is
1 over this waiting time.
00:54:41.950 --> 00:54:44.130
This is the number of
renewals you have
00:54:44.130 --> 00:54:46.220
as t goes to infinity.
00:54:46.220 --> 00:54:49.520
This is equal to 1 over the
expected length of that
00:54:49.520 --> 00:54:50.940
renewal period.
00:54:50.940 --> 00:54:51.840
Great.
00:54:51.840 --> 00:54:55.780
So we take the limit as
t goes to infinity.
00:54:55.780 --> 00:55:01.260
Of mij of t over mi of t.
00:55:01.260 --> 00:55:06.290
This is the number of
transitions overall up to time
00:55:06.290 --> 00:55:08.340
t starting in state i.
00:55:08.340 --> 00:55:10.960
This is the number of those
transitions which
00:55:10.960 --> 00:55:13.130
go into state j.
00:55:13.130 --> 00:55:15.550
How do I talk about that?
00:55:15.550 --> 00:55:23.410
Well, this quantity up here is
the number of transitions into
00:55:23.410 --> 00:55:30.520
state j over the number of
transition that take place.
00:55:30.520 --> 00:55:35.150
n sub ij of t is the number of
transitions out of total
00:55:35.150 --> 00:55:37.210
transitions.
00:55:37.210 --> 00:55:40.550
mi of t is the total number
of transitions.
00:55:40.550 --> 00:55:42.840
mi of t goes to infinity.
00:55:42.840 --> 00:55:49.480
So this limit here goes to the
limit of n sub ij of n over n.
00:55:49.480 --> 00:55:53.210
Remember, we even proved this
very carefully in class for
00:55:53.210 --> 00:55:56.170
the last application of it.
00:55:56.170 --> 00:55:59.350
This is something we've done
many times already in
00:55:59.350 --> 00:56:02.110
different situations.
00:56:02.110 --> 00:56:05.430
And this particular time we're
doing it, we won't go through
00:56:05.430 --> 00:56:06.810
any fuss about it.
00:56:06.810 --> 00:56:11.170
It's just a limit of n
sub ij of n over n.
00:56:11.170 --> 00:56:12.090
And what is that?
00:56:12.090 --> 00:56:16.330
That's the fraction, long term
fraction of transitions that
00:56:16.330 --> 00:56:18.120
go into state j.
00:56:18.120 --> 00:56:27.490
We know that for accountable
state Markov chain, which is
00:56:27.490 --> 00:56:30.480
recurrent, which is positive
recurrent, this is
00:56:30.480 --> 00:56:32.850
equal to pi sub j.
00:56:32.850 --> 00:56:35.720
So we have that as equal
to pi sub j.
00:56:35.720 --> 00:56:38.760
We then have one over w sub j,
that's what we're trying to
00:56:38.760 --> 00:56:44.140
find is equal to the limit of
mij, of t over t, which is the
00:56:44.140 --> 00:56:49.530
limit of mij of t over mi of
t times mi of t over t.
00:56:49.530 --> 00:56:52.200
We break this into a limit of
two terms, which we've done
00:56:52.200 --> 00:56:54.130
very carefully before.
00:56:54.130 --> 00:56:58.010
And this limit here
is pi sub j.
00:56:58.010 --> 00:57:02.710
This limit here is the limit
of m sub i of t.
00:57:02.710 --> 00:57:05.904
And we have already shown that
the limit of n sub i
00:57:05.904 --> 00:57:08.260
of t is equal to--
00:57:17.700 --> 00:57:20.320
Somewhere we showed that.
00:57:20.320 --> 00:57:21.570
Yeah.
00:57:25.460 --> 00:57:30.230
1 over w sub j is equal
to pj times new sub j.
00:57:30.230 --> 00:57:34.960
That's what we proved right
down here. p sub j of i is
00:57:34.960 --> 00:57:38.000
equal to 1 over new sub
j times w sub j.
00:57:40.750 --> 00:57:43.240
Except now we're just calling
it p sub j because we've
00:57:43.240 --> 00:57:44.170
already seen it.
00:57:44.170 --> 00:57:46.950
Doesn't depend on i at all.
00:57:46.950 --> 00:57:55.190
So one over w sub j is equal
to p sub j times new sub j.
00:57:55.190 --> 00:57:59.070
This says if we know what
w sub j is, we know
00:57:59.070 --> 00:58:00.880
what p sub j is.
00:58:00.880 --> 00:58:04.520
So it looks like we're not
making any progress.
00:58:04.520 --> 00:58:07.040
So what's going on?
00:58:07.040 --> 00:58:10.710
OK, well let's look at this
a little more carefully.
00:58:10.710 --> 00:58:14.770
This quantity here is a
function only of i.
00:58:19.350 --> 00:58:26.540
1 over w sub j, over p sub
j and new sub j is a
00:58:26.540 --> 00:58:27.850
function only of j.
00:58:27.850 --> 00:58:30.100
Everything else in this
equation is a
00:58:30.100 --> 00:58:32.530
function only of j.
00:58:32.530 --> 00:58:36.890
That says that this quantity
here is independent of i, and
00:58:36.890 --> 00:58:39.450
it's also independent of j.
00:58:39.450 --> 00:58:40.820
And what is it?
00:58:40.820 --> 00:58:44.310
It's the rate at which
transitions occur.
00:58:44.310 --> 00:58:46.250
Overall transitions.
00:58:46.250 --> 00:58:51.320
If you're in steady state, this
says that looking at any
00:58:51.320 --> 00:58:55.910
state j, the total number of
transitions that occur is
00:58:55.910 --> 00:59:00.310
equal to, well, I do it
on the next page.
00:59:00.310 --> 00:59:03.550
So let's goes there.
00:59:03.550 --> 00:59:07.400
It says that p sub j is equal
to 1 over new sub j.
00:59:07.400 --> 00:59:12.580
w sub j is equal to pi j over
new j times this limit here.
00:59:15.820 --> 00:59:17.070
OK.
00:59:20.060 --> 00:59:27.990
So in fact, we now have a way
of finding p sub j for all j
00:59:27.990 --> 00:59:30.870
if we can just find this
one number here.
00:59:30.870 --> 00:59:34.460
This is independent of i, so
this is just one number, which
00:59:34.460 --> 00:59:37.920
we now know is something that
approaches some limit with
00:59:37.920 --> 00:59:40.110
probability one.
00:59:40.110 --> 00:59:44.860
So we only have one unknown
instead of this countably
00:59:44.860 --> 00:59:46.110
infinite number of unknowns.
00:59:50.410 --> 00:59:53.500
Seems like we haven't really
made any progress, because
00:59:53.500 --> 00:59:56.550
before, what we did was to
normalize these p sub js.
00:59:56.550 --> 01:00:00.800
We said they have to add up to
1, and let's normalize them.
01:00:00.800 --> 01:00:03.850
And here we're doing
the same thing.
01:00:03.850 --> 01:00:09.320
We're saying p sub j is equal
to pi sub j over new sub j
01:00:09.320 --> 01:00:11.860
with this normalization
factor in.
01:00:11.860 --> 01:00:15.970
And we're saying here that this
normalization factor has
01:00:15.970 --> 01:00:22.370
to be equal to 1 over the
sum of pi k over new k.
01:00:22.370 --> 01:00:30.290
In other words, if the p sub
j's add to 1, then the only
01:00:30.290 --> 01:00:34.510
value this didn't have is
1 over the sum of pi k
01:00:34.510 --> 01:00:35.760
times new sub k.
01:00:38.020 --> 01:00:45.290
Unfortunately, there are
examples where the sum of pi
01:00:45.290 --> 01:00:49.260
sub k over new sub k is
equal to infinity.
01:00:49.260 --> 01:00:51.200
That's very awkward.
01:00:51.200 --> 01:00:53.470
I'm going to give you an example
of that in just a
01:00:53.470 --> 01:00:56.330
minute, and you'll see
what's going on.
01:00:56.330 --> 01:01:00.900
But if pi sub k over new sub k
is equal to infinity, and the
01:01:00.900 --> 01:01:05.810
theorem is true, it says that
the limit of mi of t over t is
01:01:05.810 --> 01:01:08.275
equal to 1 over infinity,
which says it's zero.
01:01:11.430 --> 01:01:16.550
So what this is telling us is
what we sort of visualize
01:01:16.550 --> 01:01:21.170
before, but we couldn't
quite visualize it.
01:01:21.170 --> 01:01:25.690
It was saying that either these
probabilities add up to
01:01:25.690 --> 01:01:31.550
1, or if they don't add up to 1,
this quantity here doesn't
01:01:31.550 --> 01:01:32.570
make any sense.
01:01:32.570 --> 01:01:34.500
This is not approaching
a limit.
01:01:34.500 --> 01:01:36.990
The only way this can approach,
well, this can
01:01:36.990 --> 01:01:41.500
approach a limit where
the limit is 0.
01:01:41.500 --> 01:01:43.870
Because this theorem holds
whether it approaches the
01:01:43.870 --> 01:01:45.080
limit or not.
01:01:45.080 --> 01:01:49.390
So it is possible for
this limit to the 0.
01:01:49.390 --> 01:01:53.950
In this case, these p
sub js are all 0.
01:01:53.950 --> 01:01:55.470
And we've seen this kind
of thing before.
01:01:55.470 --> 01:01:58.380
We've seen that on a Markov
chain, all the steady state
01:01:58.380 --> 01:02:02.380
probabilities can be equal to
zero, and that's a sign that
01:02:02.380 --> 01:02:06.010
we're either in a transient
condition, or in a null
01:02:06.010 --> 01:02:08.110
recurrent position.
01:02:08.110 --> 01:02:13.970
Namely, the state just wonders
away, and over the long term,
01:02:13.970 --> 01:02:17.420
each state has 0 probability.
01:02:17.420 --> 01:02:20.180
And that looks like the same
kind of thing which is
01:02:20.180 --> 01:02:23.250
happening here.
01:02:23.250 --> 01:02:25.660
This looks trivial.
01:02:25.660 --> 01:02:31.140
There's a fairly long proof
in the notes doing this.
01:02:31.140 --> 01:02:35.710
The only way I can find to do
this is to truncate the chain,
01:02:35.710 --> 01:02:39.780
and then go to the limit as
the number of states gets
01:02:39.780 --> 01:02:41.210
larger and larger.
01:02:41.210 --> 01:02:45.630
And when you do that, this
theorem becomes clear.
01:02:45.630 --> 01:02:48.800
OK, let's look at an example
where the sum of pi k over new
01:02:48.800 --> 01:02:53.190
k is equal to infinity, and
see what's going on.
01:02:58.020 --> 01:03:01.790
Visualize something
like an mm1 queue.
01:03:01.790 --> 01:03:06.000
We have arrivals, and
we have a server.
01:03:06.000 --> 01:03:10.000
But as soon as the queue starts
building up, the server
01:03:10.000 --> 01:03:12.300
starts to get very rattled.
01:03:12.300 --> 01:03:15.010
And as the server gets more and
more rattled, it starts to
01:03:15.010 --> 01:03:17.850
make more and more mistakes.
01:03:17.850 --> 01:03:21.880
And as the queue builds up also,
customers come in and
01:03:21.880 --> 01:03:25.110
look at the queue, and say I'll
come back tomorrow when
01:03:25.110 --> 01:03:26.780
the queue isn't so long.
01:03:26.780 --> 01:03:31.720
So we both have this situation
where as the queue is building
01:03:31.720 --> 01:03:35.890
up, service is getting slower
and the arrival rate is
01:03:35.890 --> 01:03:37.400
getting slower.
01:03:37.400 --> 01:03:40.140
And we're assuming here to make
a nice example that the
01:03:40.140 --> 01:03:44.040
two of them build up in
exactly the same way.
01:03:44.040 --> 01:03:45.790
So that's what's
happening here.
01:03:45.790 --> 01:03:50.295
The service rate when there's
one customer being served is 2
01:03:50.295 --> 01:03:52.280
to the minus 1.
01:03:52.280 --> 01:03:54.850
The service right rate when
there are two customers in the
01:03:54.850 --> 01:03:57.680
system is 2 to the minus 2.
01:03:57.680 --> 01:03:59.830
The service rate when there
are three customers in the
01:03:59.830 --> 01:04:03.550
system is 2 to the minus 3.
01:04:03.550 --> 01:04:09.340
For each of these states, we
still have these transition
01:04:09.340 --> 01:04:11.810
probabilities for the
embedded chain.
01:04:11.810 --> 01:04:15.350
And the embedded chain, the only
way to get from here to
01:04:15.350 --> 01:04:18.350
here is with probability 1,
because that's the only
01:04:18.350 --> 01:04:21.370
transition possible here.
01:04:21.370 --> 01:04:25.250
We assume that from state 1,
you go to states 0 with
01:04:25.250 --> 01:04:27.280
probability 0.6.
01:04:27.280 --> 01:04:30.965
You go to state two with
probability 0.4.
01:04:30.965 --> 01:04:36.330
With state 1, from state 2,
you go up with probability
01:04:36.330 --> 01:04:40.530
0.4, you go down with
probability 0.6.
01:04:40.530 --> 01:04:43.780
The embedded chain
looks great.
01:04:43.780 --> 01:04:45.040
There's nothing wrong
with that.
01:04:45.040 --> 01:04:51.510
That's a perfectly stable mm1
queue type of situation.
01:04:51.510 --> 01:04:56.750
It's these damned holding
times which become very
01:04:56.750 --> 01:04:59.480
disturbing.
01:04:59.480 --> 01:05:04.600
Because if you look at pi sub
j, which is supposed to be 1
01:05:04.600 --> 01:05:06.260
minus rho times rho to the j.
01:05:06.260 --> 01:05:09.510
Rho is 2/3, it's
lambda over mu.
01:05:09.510 --> 01:05:11.590
So it's lambda over lambda
plus mu over rho
01:05:11.590 --> 01:05:14.124
plus lambda plus mu.
01:05:14.124 --> 01:05:19.440
It's 0.4 divided by
0.6, which is 2/3.
01:05:19.440 --> 01:05:24.230
If we look at pi sub j over new
sub j, it's equal to 2 to
01:05:24.230 --> 01:05:29.080
the j times 1 minus rho,
times rho to the j.
01:05:29.080 --> 01:05:34.810
It's 1 minus rho times
4/3 to the j.
01:05:34.810 --> 01:05:40.440
This quantity gets bigger and
bigger as j increases.
01:05:40.440 --> 01:05:43.550
So when you try to
sum pi i over new
01:05:43.550 --> 01:05:46.910
sub j, you get infinity.
01:05:46.910 --> 01:05:50.070
So what's going on?
01:05:50.070 --> 01:05:52.670
None of the states here have
an infinite holding time
01:05:52.670 --> 01:05:54.570
associated with them.
01:05:54.570 --> 01:05:58.490
It's just that the expected
holding time
01:05:58.490 --> 01:06:01.390
is going to be infinite.
01:06:01.390 --> 01:06:04.480
The expected number of
transitions over a long period
01:06:04.480 --> 01:06:14.050
of time, according to this
equation here, expected
01:06:14.050 --> 01:06:18.040
transitions per unit
time is going to 0.
01:06:18.040 --> 01:06:24.160
As time goes on, you keep
floating back to state 0, as
01:06:24.160 --> 01:06:26.820
far as the embedded chain
is concerned.
01:06:26.820 --> 01:06:30.680
But you're eventually going to
a steady state distribution,
01:06:30.680 --> 01:06:33.650
which is laid out over
all the states.
01:06:33.650 --> 01:06:40.310
That steady state distribution
looks very nice.
01:06:40.310 --> 01:06:42.320
That's 1 minus rho times
rho to the j.
01:06:42.320 --> 01:06:48.030
Rho is 2/3, so the probability
of being in state j is going
01:06:48.030 --> 01:06:51.680
down exponentially
as j gets big.
01:06:51.680 --> 01:06:54.570
But the time that you spend
there is going up
01:06:54.570 --> 01:06:57.510
exponentially even faster.
01:06:57.510 --> 01:07:01.740
And therefore, when we sum all
of these things up, the
01:07:01.740 --> 01:07:07.330
overall expected rate is equal
to zero, because the sum of
01:07:07.330 --> 01:07:12.220
the pi j over new j is
equal to infinity.
01:07:12.220 --> 01:07:12.600
OK.
01:07:12.600 --> 01:07:14.510
So this is one of the awful
things that are going to
01:07:14.510 --> 01:07:17.590
happen with Markov processes.
01:07:17.590 --> 01:07:19.620
We still have an
embedded chain.
01:07:19.620 --> 01:07:21.960
The embedded chain
can be stable.
01:07:21.960 --> 01:07:25.150
It can have a steady state, but
we've already found that
01:07:25.150 --> 01:07:30.070
the fraction of time in a state
is not equal to the
01:07:30.070 --> 01:07:33.390
fraction of transitions that go
into that state. pi sub j
01:07:33.390 --> 01:07:36.560
is not in general equal
to p sub j.
01:07:36.560 --> 01:07:40.740
And for this example here with
the rattled server and the
01:07:40.740 --> 01:07:47.400
discouraged customers, the
amount of time that it takes,
01:07:47.400 --> 01:07:51.285
the expected amount of time that
it takes for customers to
01:07:51.285 --> 01:07:54.770
get served is going to zero.
01:07:54.770 --> 01:07:59.280
Even though the queue
was saying stable.
01:07:59.280 --> 01:08:01.380
Does mathematics lie?
01:08:01.380 --> 01:08:01.930
I don't know.
01:08:01.930 --> 01:08:04.110
I don't think so.
01:08:04.110 --> 01:08:07.320
I've looked at this often enough
with great frustration,
01:08:07.320 --> 01:08:10.230
but I believe it
at this point.
01:08:10.230 --> 01:08:13.660
If you don't believe it, take
this chain here and truncate
01:08:13.660 --> 01:08:17.840
it, and solve the problem for
the truncated chain, and then
01:08:17.840 --> 01:08:20.220
look at what happens
as you start adding
01:08:20.220 --> 01:08:22.430
states on one by one.
01:08:22.430 --> 01:08:26.700
What happens as you start adding
states on one by one is
01:08:26.700 --> 01:08:31.460
that the rate at which this
Markov process is serving
01:08:31.460 --> 01:08:34.229
things is going to zero.
01:08:34.229 --> 01:08:38.120
So the dilemma as the number
of states becomes infinite,
01:08:38.120 --> 01:08:43.979
the rate at which things
happen is equal to 0.
01:08:43.979 --> 01:08:45.880
It's not pleasant.
01:08:45.880 --> 01:08:47.750
It's not intuitive.
01:08:47.750 --> 01:08:50.210
But that's what it is.
01:08:50.210 --> 01:08:51.460
And that can happen.
01:08:56.310 --> 01:09:00.020
Again, let's go back to the
typical case of a positive
01:09:00.020 --> 01:09:07.040
recurrent embedded chain, where
this funny sum here is
01:09:07.040 --> 01:09:09.330
less than infinity.
01:09:09.330 --> 01:09:12.710
If the sum here is less than
infinity, then you can
01:09:12.710 --> 01:09:18.479
certainly express p sub j as
pi sub j over new sub j
01:09:18.479 --> 01:09:21.890
divided by the sum over k of
p sub k over new sub k.
01:09:21.890 --> 01:09:23.620
Why can I do that?
01:09:23.620 --> 01:09:25.379
Because that's what
the formula says.
01:09:28.750 --> 01:09:32.439
I don't like to live with
formulas, but sometimes things
01:09:32.439 --> 01:09:36.520
get so dirty, the mathematics
play such awful tricks with
01:09:36.520 --> 01:09:40.180
you that you have to live with
the formulas, and just try to
01:09:40.180 --> 01:09:41.689
explain what they're doing.
01:09:41.689 --> 01:09:44.529
p sub j is equal to this.
01:09:44.529 --> 01:09:52.340
Limit of the service rate, if
this quantity is non infinite,
01:09:52.340 --> 01:09:55.570
then things get churned
out of this queueing
01:09:55.570 --> 01:09:58.820
system at some rate.
01:09:58.820 --> 01:10:03.710
And the p sub js can
be solved for.
01:10:03.710 --> 01:10:06.620
And this is the way
to solve for them.
01:10:06.620 --> 01:10:08.790
OK, so that's pretty neat.
01:10:08.790 --> 01:10:12.120
It says that if you can solve
the embedded chain, then you
01:10:12.120 --> 01:10:14.810
have a nice formula for finding
the steady state
01:10:14.810 --> 01:10:16.120
probabilities.
01:10:16.120 --> 01:10:19.430
And you have a theorem which
says that so long as this
01:10:19.430 --> 01:10:24.380
quantity is less than infinity
with probability one, the
01:10:24.380 --> 01:10:27.420
fraction of time that you
stay in state j is
01:10:27.420 --> 01:10:29.660
equal to this quantity.
01:10:29.660 --> 01:10:31.710
Well, that's not good enough.
01:10:31.710 --> 01:10:35.250
Because for the mm1 queue, we
saw that it was really a pain
01:10:35.250 --> 01:10:39.270
in the neck to solve for the
steady state equations for the
01:10:39.270 --> 01:10:41.400
embedded chain.
01:10:41.400 --> 01:10:45.520
Things looked simpler for
the process itself.
01:10:45.520 --> 01:10:51.010
So let's see if we can get those
equations back also.
01:10:51.010 --> 01:10:55.710
What we would like to do is
to solve for the p sub j's
01:10:55.710 --> 01:10:59.090
directly by using the steady
state embedded equation.
01:10:59.090 --> 01:11:03.750
Embedded equations say that pi
sub j is equal to the sum over
01:11:03.750 --> 01:11:07.330
i, pi sub i times p sub ij.
01:11:07.330 --> 01:11:09.490
The probability of going into
a state is equal to the
01:11:09.490 --> 01:11:11.000
probability of going
out of a state.
01:11:15.970 --> 01:11:23.720
If I use this formula here, pi
sub j over new sub j divided
01:11:23.720 --> 01:11:27.330
by some constant is
what p sub j is.
01:11:27.330 --> 01:11:32.435
So pi sub j is equal to
p sub j times new sub
01:11:32.435 --> 01:11:34.410
j times that constant.
01:11:34.410 --> 01:11:41.290
Here we have the p sub
j times the new sub j
01:11:41.290 --> 01:11:44.060
divided by that constant.
01:11:44.060 --> 01:11:50.490
And that's equal to this sum
here over all i divided by the
01:11:50.490 --> 01:11:51.280
same constant.
01:11:51.280 --> 01:11:53.770
So the constant cancels out.
01:11:53.770 --> 01:11:57.870
Namely, we left out that term
here, but that term is on this
01:11:57.870 --> 01:11:59.940
side, and it's on this side.
01:11:59.940 --> 01:12:03.530
So we have this equation here.
01:12:03.530 --> 01:12:07.230
If you remember, I can't
remember, but you being
01:12:07.230 --> 01:12:09.450
younger can perhaps remember.
01:12:09.450 --> 01:12:12.810
But when we were dealing with
a sample time approximation,
01:12:12.810 --> 01:12:16.600
if you leave the deltas out,
this is exactly the equation
01:12:16.600 --> 01:12:18.440
that you got.
01:12:18.440 --> 01:12:20.600
It's a nice equation.
01:12:20.600 --> 01:12:25.910
It says that the rate at which
transitions occur out of state
01:12:25.910 --> 01:12:29.300
i, the rate at which transitions
occur out of state
01:12:29.300 --> 01:12:35.490
i, out of state j, excuse me,
is p sub j times new sub j.
01:12:35.490 --> 01:12:40.330
Here's the holding time, and
there's also the probability
01:12:40.330 --> 01:12:42.230
of being there.
01:12:42.230 --> 01:12:43.380
Excuse me.
01:12:43.380 --> 01:12:45.200
Let's be more clear
about that.
01:12:45.200 --> 01:12:50.040
I'm not talking about given
you're in state j, the rate at
01:12:50.040 --> 01:12:51.870
which you get out of state j.
01:12:51.870 --> 01:12:56.700
I'm talking about the rate at
which you're in state j and
01:12:56.700 --> 01:12:58.520
you're getting out of state j.
01:12:58.520 --> 01:13:00.390
If you could make sense
out of that.
01:13:00.390 --> 01:13:02.100
That's what this is.
01:13:02.100 --> 01:13:06.090
This quantity here is the
overall rate at which you're
01:13:06.090 --> 01:13:08.100
entering state j.
01:13:08.100 --> 01:13:11.730
So these equations sort of
have the same intuitive
01:13:11.730 --> 01:13:14.390
meaning as these equations
here do.
01:13:19.450 --> 01:13:28.140
And then if you solve this
equation in the same way,
01:13:28.140 --> 01:13:30.370
what's that doing?
01:13:30.370 --> 01:13:31.610
Oh.
01:13:31.610 --> 01:13:34.550
This gets you pi sub j in
terms of the p sub j's.
01:13:34.550 --> 01:13:36.060
So there's a very nice symmetry
01:13:36.060 --> 01:13:38.880
about this set of equations.
01:13:38.880 --> 01:13:43.990
The p sub j's are found for the
pi sub j's in this way.
01:13:43.990 --> 01:13:47.340
The pi sub j's are found from
the p sub j's by this same
01:13:47.340 --> 01:13:49.010
sort of expression.
01:13:49.010 --> 01:13:52.430
The theorem then says if the
embedded chain is positive
01:13:52.430 --> 01:13:57.250
recurrent, and the sum of pi
i over new i is less than
01:13:57.250 --> 01:14:01.740
infinity, then this equation
has a unique solution.
01:14:01.740 --> 01:14:05.270
In other words, there is a
solution to the steady state
01:14:05.270 --> 01:14:08.670
process equations.
01:14:08.670 --> 01:14:16.130
And pi sub j and p sub j are
related by this, and by this.
01:14:16.130 --> 01:14:19.740
If you know the pi sub j's, you
can find the p sub j's.
01:14:19.740 --> 01:14:23.940
If you know the p sub j's, you
can find the pi sub j's.
01:14:23.940 --> 01:14:28.180
There's a fudge factor, and the
sum of pi sub i over new
01:14:28.180 --> 01:14:32.330
sub i is equal to sum of
p sub j times new sub j
01:14:32.330 --> 01:14:34.300
to the minus 1.
01:14:34.300 --> 01:14:37.660
This equation just falls out
of looking at this equation
01:14:37.660 --> 01:14:39.670
and this equation.
01:14:39.670 --> 01:14:42.210
I'm not going to do that here,
but if you just fiddle with
01:14:42.210 --> 01:14:45.650
these equations a little bit,
that's what you find.
01:14:49.554 --> 01:14:52.530
I think graduate students
love to push equations.
01:14:52.530 --> 01:14:55.310
And if you push these equations,
you will rapidly
01:14:55.310 --> 01:14:57.110
find that out.
01:14:57.110 --> 01:14:59.680
So there's no point
to doing it here.
01:14:59.680 --> 01:15:00.930
OK.
01:15:02.430 --> 01:15:05.430
You can do the opposite
thing, also.
01:15:05.430 --> 01:15:09.650
If the steady state process
equations are satisfied, and
01:15:09.650 --> 01:15:13.450
the p sub j's are all greater
than zero, and the sum of the
01:15:13.450 --> 01:15:15.910
p sub j's are equal to 1.
01:15:15.910 --> 01:15:20.010
And if these equations are less
than infinity, this is
01:15:20.010 --> 01:15:23.090
just by symmetry with
what we already did.
01:15:23.090 --> 01:15:27.730
Then pi sub j has to be equal
to p sub j times new sub j
01:15:27.730 --> 01:15:30.230
divided by this sum.
01:15:30.230 --> 01:15:32.830
And this gives the steady
state equations for the
01:15:32.830 --> 01:15:34.690
embedded chain.
01:15:34.690 --> 01:15:37.230
And this shows that the embedded
chain has to be
01:15:37.230 --> 01:15:39.980
positive recurrent, and says
that you have to be
01:15:39.980 --> 01:15:41.380
able to go both ways.
01:15:41.380 --> 01:15:44.600
So we already know that if you
can solve the steady state
01:15:44.600 --> 01:15:48.310
equations for the embedded
chain, they have to be unique,
01:15:48.310 --> 01:15:50.610
the probabilities all
have to be positive.
01:15:50.610 --> 01:15:53.580
All those neat things for
accountable state and Markov
01:15:53.580 --> 01:15:55.410
chains hold true.
01:15:55.410 --> 01:15:59.830
This says that if you can solve
that virtually identical
01:15:59.830 --> 01:16:04.590
set of equations for the
process, and you get a
01:16:04.590 --> 01:16:13.210
solution, and the sum here is
finite, then in fact you can
01:16:13.210 --> 01:16:15.310
go back the other way.
01:16:15.310 --> 01:16:17.920
And going back the other way,
you know from what we know
01:16:17.920 --> 01:16:21.530
about embedded chains that
there's a unique solution.
01:16:21.530 --> 01:16:24.460
So there has to be a unique
solution both ways.
01:16:24.460 --> 01:16:28.030
There has to be positive
recurrence both ways.
01:16:28.030 --> 01:16:31.740
So we have a complete
story at this point.
01:16:31.740 --> 01:16:34.160
I mean, you'll have to spend a
little more time putting it
01:16:34.160 --> 01:16:37.580
together, but it really
is there.
01:16:37.580 --> 01:16:42.630
If new sub j is bounded over
j, then the sum over j of p
01:16:42.630 --> 01:16:46.590
sub j, new sub j is less
than infinity.
01:16:46.590 --> 01:16:49.120
Also, the sample time
chain exists
01:16:49.120 --> 01:16:50.750
because this is bounded.
01:16:50.750 --> 01:16:54.230
It has the same steady
state solution as the
01:16:54.230 --> 01:16:57.140
Markov process solution.
01:16:57.140 --> 01:16:59.880
In other words, go back and look
at the solution for the
01:16:59.880 --> 01:17:01.550
sample time chain.
01:17:01.550 --> 01:17:05.270
Drop out the delta, and what
you will get is this set of
01:17:05.270 --> 01:17:08.410
equations here.
01:17:08.410 --> 01:17:11.880
If you have a birth death
process, it's a birth death
01:17:11.880 --> 01:17:17.190
process for both the Markov
process, and also for the
01:17:17.190 --> 01:17:18.190
embedded chain.
01:17:18.190 --> 01:17:20.500
For the embedded chain,
you know that what you
01:17:20.500 --> 01:17:24.040
have to do to get--
01:17:24.040 --> 01:17:27.660
for a birth death chain, you
know an easy way to solve the
01:17:27.660 --> 01:17:33.070
steady state equations for the
chain are to say transitions
01:17:33.070 --> 01:17:35.610
this way equal transitions
this way.
01:17:35.610 --> 01:17:38.910
It's the same for the process.
01:17:38.910 --> 01:17:42.740
The amount of time that you
spend going this way is equal
01:17:42.740 --> 01:17:46.360
to the average amount of time
you spent going this way.
01:17:46.360 --> 01:17:51.900
So it says for a birth death
process, p sub i times q sub
01:17:51.900 --> 01:17:53.800
i, i plus one.
01:17:56.450 --> 01:18:02.870
That's an i there, is equal to
p sub i plus 1 times the
01:18:02.870 --> 01:18:08.110
transition probability from
i plus 1 back to i.
01:18:08.110 --> 01:18:12.190
So this symmetry exists
almost everywhere.
01:18:12.190 --> 01:18:17.170
And then if the sum of p sub
j, new sub j is equal to
01:18:17.170 --> 01:18:20.790
infinity, that's the bad case.
01:18:20.790 --> 01:18:27.670
These equations say that
pi sub j is equal to 0
01:18:27.670 --> 01:18:28.830
everywhere.
01:18:28.830 --> 01:18:32.120
This is sort of the dual
of the situation we
01:18:32.120 --> 01:18:33.480
already looked at.
01:18:33.480 --> 01:18:38.070
In the case we already looked
at of the lazy or rattled
01:18:38.070 --> 01:18:43.050
server and the discouraged
customers, eventually the rate
01:18:43.050 --> 01:18:46.390
at which service occurred
went to 0.
01:18:46.390 --> 01:18:52.230
This situation is a situation
where as far as the embedded
01:18:52.230 --> 01:18:58.490
chain is concerned, it thinks
it's transient.
01:18:58.490 --> 01:19:02.210
As far as the process is
concerned, the process thinks
01:19:02.210 --> 01:19:03.710
it's just fine.
01:19:03.710 --> 01:19:06.750
But then you look at the
process, and you find out that
01:19:06.750 --> 01:19:10.600
what's happening is that an
infinite number of transitions
01:19:10.600 --> 01:19:15.170
are taking place in
a finite time.
01:19:15.170 --> 01:19:19.480
Markov process people call these
irregular processes.
01:19:19.480 --> 01:19:22.306
Here's a picture of it
on the next slide.
01:19:25.360 --> 01:19:29.250
Essentially, the embedded chain
for a hyperactive birth
01:19:29.250 --> 01:19:30.480
death chain.
01:19:30.480 --> 01:19:34.400
As you go to higher states, the
rate at which transitions
01:19:34.400 --> 01:19:38.340
take place gets higher
and higher.
01:19:38.340 --> 01:19:42.420
And for this particular example
here, you can solve
01:19:42.420 --> 01:19:44.460
those process equations.
01:19:44.460 --> 01:19:47.400
The process equations
look just fine.
01:19:47.400 --> 01:19:51.110
This is where you have to be
careful with Markov processes.
01:19:51.110 --> 01:19:55.360
Because you can solve these
Markov process equations, and
01:19:55.360 --> 01:19:59.930
you get things that look fine,
when actually there isn't any
01:19:59.930 --> 01:20:02.700
steady state behavior at all.
01:20:02.700 --> 01:20:05.280
It's even worse than the
rate going to zero.
01:20:05.280 --> 01:20:07.790
When the rate goes to infinity,
you can have an
01:20:07.790 --> 01:20:12.560
infinite number of transitions
taking place in a finite time.
01:20:12.560 --> 01:20:14.310
And then nothing happens.
01:20:14.310 --> 01:20:16.800
Well, I don't know whether
something happens or not.
01:20:16.800 --> 01:20:21.625
I can't visualize what happens
after the thing has exploded.
01:20:24.640 --> 01:20:26.020
Except essentially, at
that point, there's
01:20:26.020 --> 01:20:28.440
nothing nice going on.
01:20:28.440 --> 01:20:32.920
And you have to say that even
though the process equations
01:20:32.920 --> 01:20:36.910
have a steady state solution,
there is no steady state in
01:20:36.910 --> 01:20:40.230
terms of over the long period of
time, this is the fraction
01:20:40.230 --> 01:20:42.410
of time you spend in state j.
01:20:42.410 --> 01:20:45.730
Because that's not the
way it's behaving.
01:20:45.730 --> 01:20:46.980
OK.
01:20:49.250 --> 01:20:51.450
I mean, you can see this
when you look at
01:20:51.450 --> 01:20:53.490
the embedded chain.
01:20:53.490 --> 01:20:56.350
The embedded chain
is transient.
01:20:56.350 --> 01:20:58.590
You're moving up with
probability 0.6.
01:20:58.590 --> 01:21:01.020
You're moving down with
probability 0.4.
01:21:01.020 --> 01:21:03.990
So you keep moving up.
01:21:03.990 --> 01:21:10.440
When you look at the process in
terms of the process with
01:21:10.440 --> 01:21:16.960
the transition rates q sub ij,
the rates going up are always
01:21:16.960 --> 01:21:18.550
less than the rates
going down.
01:21:18.550 --> 01:21:25.520
This is because as you
move up in state,
01:21:25.520 --> 01:21:28.130
you act so much faster.
01:21:28.130 --> 01:21:33.720
The transition rates are higher
in higher states, and
01:21:33.720 --> 01:21:37.750
therefore the transition rates
down are higher, and the
01:21:37.750 --> 01:21:43.900
transition rate down from state
2 to state 1 is bigger
01:21:43.900 --> 01:21:49.110
than the transition right
up from i to i plus 1.
01:21:49.110 --> 01:21:54.000
And it looks stable, as far as
the process is concerned.
01:21:54.000 --> 01:21:58.010
This is one example where you
can't look at the process and
01:21:58.010 --> 01:22:01.190
find out anything from it
without also looking at the
01:22:01.190 --> 01:22:04.570
embedded chain, and looking at
how many transitions you're
01:22:04.570 --> 01:22:07.070
getting per unit time.
01:22:07.070 --> 01:22:10.080
OK, so that's it.
01:22:10.080 --> 01:22:11.330
Thank you.