1
00:00:00,530 --> 00:00:02,960
The following content is
provided under a Creative
2
00:00:02,960 --> 00:00:04,370
Commons license.
3
00:00:04,370 --> 00:00:07,410
Your support will help MIT
OpenCourseWare continue to
4
00:00:07,410 --> 00:00:11,060
offer high quality educational
resources for free.
5
00:00:11,060 --> 00:00:13,960
To make a donation or view
additional materials from
6
00:00:13,960 --> 00:00:17,890
hundreds of MIT courses, visit
MIT OpenCourseWare at
7
00:00:17,890 --> 00:00:19,140
ocw.mit.edu.
8
00:00:23,810 --> 00:00:24,270
PROFESSOR: OK.
9
00:00:24,270 --> 00:00:27,020
I guess we might as
well get started.
10
00:00:27,020 --> 00:00:32,729
We talked a little bit about
Markov processes last week.
11
00:00:32,729 --> 00:00:37,300
I want to review a little bit of
what we did then, and then
12
00:00:37,300 --> 00:00:39,570
move on pretty quickly.
13
00:00:39,570 --> 00:00:44,090
This is a rather strange
chapter, because
14
00:00:44,090 --> 00:00:46,620
it's full of notation.
15
00:00:46,620 --> 00:00:50,350
When I first started reviewing
it myself after not having
16
00:00:50,350 --> 00:00:54,550
looked at it for about a year,
I was horrified by the amount
17
00:00:54,550 --> 00:00:56,130
of notation.
18
00:00:56,130 --> 00:00:59,490
And then I realized, what we're
doing in this chapter is
19
00:00:59,490 --> 00:01:02,750
putting together all the
stuff we've learned
20
00:01:02,750 --> 00:01:04,330
from Poisson processes.
21
00:01:04,330 --> 00:01:08,460
There are Poisson processes
all through this
22
00:01:08,460 --> 00:01:12,400
Markov chains renewals.
23
00:01:12,400 --> 00:01:15,670
And all three of these, with
all their notation, are all
24
00:01:15,670 --> 00:01:17,080
sitting here.
25
00:01:17,080 --> 00:01:21,390
And then we get into new things,
also, so I apologize
26
00:01:21,390 --> 00:01:22,720
for the notation.
27
00:01:22,720 --> 00:01:24,910
I'm going to try to keep
it down to the
28
00:01:24,910 --> 00:01:27,480
minimal amount this time.
29
00:01:27,480 --> 00:01:33,060
And see if we can sort of get
to the bottom line as easily
30
00:01:33,060 --> 00:01:38,350
as we can, so that when you do
the exercises and read the
31
00:01:38,350 --> 00:01:41,690
notes, you can go back in
fill in some of the
32
00:01:41,690 --> 00:01:44,630
things that are missing.
33
00:01:44,630 --> 00:01:44,980
OK.
34
00:01:44,980 --> 00:01:49,440
So accountable state
Markov process.
35
00:01:49,440 --> 00:01:53,090
The easiest way to define it,
and the way we're defining it
36
00:01:53,090 --> 00:01:54,980
is as an extension
of accountable
37
00:01:54,980 --> 00:01:56,610
state Markov chain.
38
00:01:56,610 --> 00:01:58,980
So you start out with
accountable state Markov
39
00:01:58,980 --> 00:02:03,440
chain, and then along with each
step, say from state x
40
00:02:03,440 --> 00:02:09,720
sub n to x sub n plus 1, there
is an exponential holding
41
00:02:09,720 --> 00:02:12,540
time, u sub n plus 1.
42
00:02:12,540 --> 00:02:15,420
We said that it was a little
strange associating the
43
00:02:15,420 --> 00:02:18,210
holding time of the
final state rather
44
00:02:18,210 --> 00:02:20,240
than the initial state.
45
00:02:20,240 --> 00:02:23,890
But you almost had to do that,
because of many things that
46
00:02:23,890 --> 00:02:26,660
would get very confusing
otherwise.
47
00:02:26,660 --> 00:02:28,490
So we're just doing that.
48
00:02:28,490 --> 00:02:33,860
We start out in state x0,
which is usually given.
49
00:02:33,860 --> 00:02:37,400
If it's not given,
it can be random.
50
00:02:37,400 --> 00:02:44,660
And then we go to state x1 after
a waiting time, u sub 1.
51
00:02:44,660 --> 00:02:48,880
Go from x1 to x2 with a
waiting time u sub 2.
52
00:02:48,880 --> 00:02:54,640
The waiting time u sub i is a
function of the state we're
53
00:02:54,640 --> 00:02:58,440
coming from, in the sense that
it's an exponential random
54
00:02:58,440 --> 00:03:01,820
variable who's rate is
given by the state
55
00:03:01,820 --> 00:03:03,890
we're coming from.
56
00:03:03,890 --> 00:03:12,170
So this diagram here gives
you, really, what the
57
00:03:12,170 --> 00:03:16,490
dependence of this set of
random variables is.
58
00:03:16,490 --> 00:03:19,650
You have a sequence of random
variables here.
59
00:03:19,650 --> 00:03:22,190
A sequence of random
variables here.
60
00:03:22,190 --> 00:03:26,950
Each random variable here
conditional on the initial
61
00:03:26,950 --> 00:03:30,310
state that it's coming
from, is independent
62
00:03:30,310 --> 00:03:31,360
of everything else.
63
00:03:31,360 --> 00:03:35,330
And that's what this dependence
diagram means.
64
00:03:35,330 --> 00:03:39,670
It's a generalization of what
we talked about before.
65
00:03:39,670 --> 00:03:42,790
When we talked about Markov
chains before, we just had a
66
00:03:42,790 --> 00:03:46,350
string of states.
67
00:03:46,350 --> 00:03:52,540
And each state x sub n is
dependent only on the prior
68
00:03:52,540 --> 00:03:55,000
state x sub n minus 1.
69
00:03:55,000 --> 00:03:58,610
And given the prior state
x sub n minus 1, it's
70
00:03:58,610 --> 00:04:02,650
statistically independent of
all states before that.
71
00:04:02,650 --> 00:04:08,270
Here we have this more general
situation of a tree.
72
00:04:08,270 --> 00:04:13,030
I thought I'd better illustrate
this a little more.
73
00:04:13,030 --> 00:04:17,740
If you have a directed tree, the
dependencies, each random
74
00:04:17,740 --> 00:04:22,300
variable conditional as parent
is statistically independent
75
00:04:22,300 --> 00:04:24,130
of all earlier random
variables.
76
00:04:24,130 --> 00:04:28,450
In other words, this random
variable here conditional on
77
00:04:28,450 --> 00:04:32,540
this random variable is
statistically independent of
78
00:04:32,540 --> 00:04:36,480
this, this, this, and this.
79
00:04:36,480 --> 00:04:39,810
And so you can move forward in
this way, defining each random
80
00:04:39,810 --> 00:04:44,160
variable as being statistically
independent of
81
00:04:44,160 --> 00:04:47,550
everything in the past,
conditional on just one
82
00:04:47,550 --> 00:04:49,750
previous random variable.
83
00:04:49,750 --> 00:04:54,700
As an example of this, if you
look at the probability of x0,
84
00:04:54,700 --> 00:05:04,680
x1, x2, and u2, these five
random variables here.
85
00:05:04,680 --> 00:05:08,750
Probability of x0, probability
of x1 given x0.
86
00:05:08,750 --> 00:05:11,400
Probability of x2 given x1.
87
00:05:11,400 --> 00:05:15,020
Probability of u2 given x1.
88
00:05:15,020 --> 00:05:16,890
You can write that
out in that way.
89
00:05:16,890 --> 00:05:20,550
From this, you can rewrite this
in any way you want to.
90
00:05:20,550 --> 00:05:24,000
You take these two equations,
and you can rewrite them as a
91
00:05:24,000 --> 00:05:28,950
probability of x1 times the
probability of x0 given x1,
92
00:05:28,950 --> 00:05:31,830
times the probability of
x2 given x1, times the
93
00:05:31,830 --> 00:05:34,350
probability u2 given x1.
94
00:05:34,350 --> 00:05:37,950
In other words, what's happening
here is if you
95
00:05:37,950 --> 00:05:43,060
condition everything on x1,
this random variable here,
96
00:05:43,060 --> 00:05:47,510
this stuff is statistically
independent of this.
97
00:05:47,510 --> 00:05:51,280
Is statistically independent
of all of this.
98
00:05:51,280 --> 00:05:57,180
Given any one node in this
three, given the value of that
99
00:05:57,180 --> 00:06:01,030
node, everything on every set of
branches coming out from it
100
00:06:01,030 --> 00:06:02,280
is statistically independent.
101
00:06:04,870 --> 00:06:08,000
This is a remarkably
useful property.
102
00:06:08,000 --> 00:06:10,830
This is the Markov property
in general.
103
00:06:10,830 --> 00:06:13,210
I mean, Markov chains,
we only use the fact
104
00:06:13,210 --> 00:06:14,590
that it's on a chain.
105
00:06:14,590 --> 00:06:17,770
In general you use the fact
that it's on a tree.
106
00:06:17,770 --> 00:06:25,540
And all of this stuff can be
used in remarkable ways.
107
00:06:25,540 --> 00:06:28,970
I didn't know this until
probably five years ago.
108
00:06:28,970 --> 00:06:32,390
And suddenly when I realized it,
I think because somebody
109
00:06:32,390 --> 00:06:33,990
was pointing it out
in a research
110
00:06:33,990 --> 00:06:35,390
paper they were writing.
111
00:06:35,390 --> 00:06:39,380
Suddenly all sorts of things
became much, much easier.
112
00:06:39,380 --> 00:06:46,850
Because everything like the fact
we pointed out before,
113
00:06:46,850 --> 00:06:51,190
that the past is independent
of the future, given the
114
00:06:51,190 --> 00:06:55,020
present, that's one
example of this.
115
00:06:55,020 --> 00:06:57,880
But this is far more
general than that.
116
00:06:57,880 --> 00:07:02,500
It says that anything on this
tree, if you can start at any
117
00:07:02,500 --> 00:07:05,470
point on the tree, and
everything going out from
118
00:07:05,470 --> 00:07:09,050
there is statistically
independent, given this node
119
00:07:09,050 --> 00:07:10,510
on the tree.
120
00:07:10,510 --> 00:07:13,400
So that's a valuable thing.
121
00:07:13,400 --> 00:07:16,720
And so you get your condition
on any node, breaks a tree
122
00:07:16,720 --> 00:07:18,590
into independent sub trees.
123
00:07:18,590 --> 00:07:20,920
You can then go on from there
and break it down further and
124
00:07:20,920 --> 00:07:24,520
further and further, until you
get out to the leaves.
125
00:07:24,520 --> 00:07:28,750
So this independence property
is really the general thing
126
00:07:28,750 --> 00:07:31,540
that we refer to when we
say a set of random
127
00:07:31,540 --> 00:07:34,160
variables are Markov.
128
00:07:34,160 --> 00:07:34,670
OK.
129
00:07:34,670 --> 00:07:38,840
The evolution in time with a
Markov process, this diagram I
130
00:07:38,840 --> 00:07:41,580
find very helpful to
see what's going
131
00:07:41,580 --> 00:07:43,610
on in a Markov process.
132
00:07:43,610 --> 00:07:46,090
You have a set of states.
133
00:07:46,090 --> 00:07:52,440
Initially, you're in a state
x0, the state at time 0 is
134
00:07:52,440 --> 00:07:54,140
some given value i.
135
00:07:54,140 --> 00:07:57,300
This is a sample path here.
136
00:07:57,300 --> 00:08:01,570
The next state we'll say is
j, the next state is a k.
137
00:08:01,570 --> 00:08:06,890
When you're in state i, there's
some holding time,
138
00:08:06,890 --> 00:08:11,440
which has rate new
1, new sub i.
139
00:08:11,440 --> 00:08:14,190
It's an exponential random
variable which tells you how
140
00:08:14,190 --> 00:08:17,010
long it takes until
this transition.
141
00:08:17,010 --> 00:08:24,390
This transition occurs
at time s1, which is
142
00:08:24,390 --> 00:08:26,440
u1 is equal to s1.
143
00:08:26,440 --> 00:08:29,020
The next transition is at s2.
144
00:08:29,020 --> 00:08:31,490
Equals u1 plus u2.
145
00:08:31,490 --> 00:08:36,289
The next transition is at x3,
which is u1 plus u2 plus u3.
146
00:08:36,289 --> 00:08:39,700
Now you start to see why we've
numbered these holding times
147
00:08:39,700 --> 00:08:43,260
the way we have, so we can talk
about the times that each
148
00:08:43,260 --> 00:08:44,995
of these transitions
take place.
149
00:08:47,930 --> 00:08:50,890
We usually assume that the
embedded Markov chain for a
150
00:08:50,890 --> 00:08:55,330
Markov process, remember, the
embedded Markov chain now is
151
00:08:55,330 --> 00:08:57,480
just the Markov chain
itself without these
152
00:08:57,480 --> 00:08:59,870
holding times on it.
153
00:08:59,870 --> 00:09:03,070
We assume it has no self
transitions because if you're
154
00:09:03,070 --> 00:09:06,980
sitting in a state x of t equals
i, and suddenly there's
155
00:09:06,980 --> 00:09:12,200
a transition back to i again,
and you look at the process in
156
00:09:12,200 --> 00:09:16,480
terms of x of t, t greater
than or equal to 0.
157
00:09:16,480 --> 00:09:21,320
The state that you're in at
each time t, what happens?
158
00:09:21,320 --> 00:09:22,070
You don't see it.
159
00:09:22,070 --> 00:09:26,200
It's a totally invisible
transition, because you're
160
00:09:26,200 --> 00:09:28,260
sitting in state i.
161
00:09:28,260 --> 00:09:32,140
You suddenly have a transition
back to i that takes 0 time.
162
00:09:32,140 --> 00:09:34,120
So you stay in state i.
163
00:09:34,120 --> 00:09:37,460
You can put that transition
in or you can take it out.
164
00:09:37,460 --> 00:09:38,840
It doesn't make any
difference.
165
00:09:38,840 --> 00:09:43,750
It won't affect the
process at all.
166
00:09:43,750 --> 00:09:44,860
OK.
167
00:09:44,860 --> 00:09:50,120
Aside from that issue of these
self transitions, a sample
168
00:09:50,120 --> 00:09:56,640
path of both x sub n, each of
these x sub 0 equals i, x sub
169
00:09:56,640 --> 00:09:58,906
1 equals j.
170
00:09:58,906 --> 00:10:01,810
x sub 2 equals k.
171
00:10:01,810 --> 00:10:07,630
A sample path as those plus the
holding times specifies
172
00:10:07,630 --> 00:10:10,450
what x of t is at each
instant of time.
173
00:10:10,450 --> 00:10:13,840
And if you know what x of t is
at each unit of time, that
174
00:10:13,840 --> 00:10:16,720
tells you when the transitions
are occurring.
175
00:10:16,720 --> 00:10:19,100
When you know when the
transitions are occurring, you
176
00:10:19,100 --> 00:10:21,340
know what the these u's are.
177
00:10:21,340 --> 00:10:24,250
And when you see what the
transition is into, you know
178
00:10:24,250 --> 00:10:25,700
what the state is.
179
00:10:25,700 --> 00:10:30,600
So the description of a Markov
process in terms of the
180
00:10:30,600 --> 00:10:34,810
process, what we call a process
x sub t for all t
181
00:10:34,810 --> 00:10:39,540
greater than or equal to 0,
and the set of random
182
00:10:39,540 --> 00:10:44,110
variables, the embedded Markov
chain, and the holding times
183
00:10:44,110 --> 00:10:45,960
are both equivalent
to each other.
184
00:10:45,960 --> 00:10:48,950
This shouldn't be any surprise
to you by now, because every
185
00:10:48,950 --> 00:10:52,220
process we've talked to,
we've talked about.
186
00:10:52,220 --> 00:10:54,380
We've described in
the same way.
187
00:10:54,380 --> 00:10:59,840
We described the Poisson process
in multiple ways.
188
00:10:59,840 --> 00:11:03,480
We described Markov chains
in multiple ways.
189
00:11:03,480 --> 00:11:07,360
We described renewal processes
in multiple ways.
190
00:11:07,360 --> 00:11:11,770
And this is just another
example of that.
191
00:11:11,770 --> 00:11:15,920
You use whatever description you
want to after you've shown
192
00:11:15,920 --> 00:11:18,960
they're all equivalent.
193
00:11:18,960 --> 00:11:21,090
So there's really nothing
new here.
194
00:11:21,090 --> 00:11:22,340
Or is there?
195
00:11:24,760 --> 00:11:29,590
Who can see what there is about
this relationship, which
196
00:11:29,590 --> 00:11:33,360
is different from what we've
just been talking about?
197
00:11:33,360 --> 00:11:34,745
It's using one extra property.
198
00:11:38,850 --> 00:11:44,640
This is not just a consequence
of this, it also uses
199
00:11:44,640 --> 00:11:45,450
something else.
200
00:11:45,450 --> 00:11:48,590
And what else does it use?
201
00:11:48,590 --> 00:11:53,210
What I'm doing is saying x of
t at this instance of time
202
00:11:53,210 --> 00:11:59,900
here is dependent on given
x of t is some
203
00:11:59,900 --> 00:12:03,280
previous time here.
204
00:12:03,280 --> 00:12:07,310
The state here, given the state
here is independent of
205
00:12:07,310 --> 00:12:09,920
everything in the past.
206
00:12:09,920 --> 00:12:11,830
So what else am I using there?
207
00:12:15,300 --> 00:12:19,740
I'm using the memory looseness
of the Poisson process.
208
00:12:19,740 --> 00:12:21,900
I'm using the memory
looseness of the
209
00:12:21,900 --> 00:12:25,470
exponential random variable.
210
00:12:25,470 --> 00:12:32,400
If I'm given the state here,
and I'm conditioning on the
211
00:12:32,400 --> 00:12:36,580
state here, this is
an exponential
212
00:12:36,580 --> 00:12:38,010
random variable in here.
213
00:12:38,010 --> 00:12:40,950
The time the next transition
is exponential
214
00:12:40,950 --> 00:12:42,490
given this time here.
215
00:12:42,490 --> 00:12:45,280
And it doesn't matter
when the previous
216
00:12:45,280 --> 00:12:46,425
transition took place.
217
00:12:46,425 --> 00:12:50,940
So if I'm given the state at
this time here, the time to
218
00:12:50,940 --> 00:12:55,020
the next transition is an
exponential random variable,
219
00:12:55,020 --> 00:12:58,510
the same distribution as u2.
220
00:12:58,510 --> 00:13:03,950
So what this says is I'm using
the initial description in
221
00:13:03,950 --> 00:13:10,000
terms of an embedded Markov
chain plus holding times, and
222
00:13:10,000 --> 00:13:12,570
I'm adding to that the fact
that the holding times are
223
00:13:12,570 --> 00:13:15,880
exponential, and therefore
they're memory less.
224
00:13:15,880 --> 00:13:17,090
OK, that clear to everybody?
225
00:13:17,090 --> 00:13:20,690
It's vitally important
for all of this.
226
00:13:20,690 --> 00:13:27,820
Because it's hard to do anything
with Markov processes
227
00:13:27,820 --> 00:13:33,100
without realizing explicitly
that you're using the fact
228
00:13:33,100 --> 00:13:36,230
that these random variables
are memory less.
229
00:13:36,230 --> 00:13:38,710
At the end of this chapter,
there's something called semi
230
00:13:38,710 --> 00:13:40,700
Markov processes.
231
00:13:40,700 --> 00:13:43,880
The description is that
semi Markov processes
232
00:13:43,880 --> 00:13:45,770
are exactly the same.
233
00:13:45,770 --> 00:13:50,210
Semi Markov chains are exactly
the same as markup processes
234
00:13:50,210 --> 00:13:56,490
except, these holding times
are not exponential.
235
00:13:56,490 --> 00:13:58,270
They can be anything.
236
00:13:58,270 --> 00:14:01,700
And as soon as the holding times
can be anything, the
237
00:14:01,700 --> 00:14:05,110
process gets so complicated that
you hardly want to talk
238
00:14:05,110 --> 00:14:06,360
about it anymore.
239
00:14:09,140 --> 00:14:12,480
So the fact that we have these
exponential holding times is
240
00:14:12,480 --> 00:14:17,600
really important in terms of
getting this condition here,
241
00:14:17,600 --> 00:14:21,720
which lets you talk directly
about the process, instead of
242
00:14:21,720 --> 00:14:23,265
the embedded Markov chain.
243
00:14:27,120 --> 00:14:31,200
You're going to represent a
Markov process by a graph for
244
00:14:31,200 --> 00:14:35,560
the embedded Markov chain, and
then you give the rates on top
245
00:14:35,560 --> 00:14:36,320
of the nodes.
246
00:14:36,320 --> 00:14:42,290
So if you're in state 0, the
holding time until you enter
247
00:14:42,290 --> 00:14:47,380
the next state is given as
some, the rate of that
248
00:14:47,380 --> 00:14:50,030
exponential is given as new 0.
249
00:14:50,030 --> 00:14:54,330
The rate here is given
as new 1, so forth.
250
00:14:54,330 --> 00:14:57,940
Ultimately, we're usually
interested in this process, x
251
00:14:57,940 --> 00:15:01,740
of t, t greater than or equal
to 0, which is the Markov
252
00:15:01,740 --> 00:15:03,220
process itself.
253
00:15:03,220 --> 00:15:11,300
x of t is equal to xn for t in
sub n, between sub n and
254
00:15:11,300 --> 00:15:13,150
s sub n plus 1.
255
00:15:13,150 --> 00:15:15,880
What does that mean?
256
00:15:15,880 --> 00:15:18,800
Well, it means at this point,
we're taking the Markov
257
00:15:18,800 --> 00:15:23,310
process as the fundamental
thing, and we're describing it
258
00:15:23,310 --> 00:15:27,400
in terms of the nth
state transition.
259
00:15:27,400 --> 00:15:31,390
But we know that the nth state
transition takes place at time
260
00:15:31,390 --> 00:15:35,330
s sub n, namely it takes place
at the sum of all of these
261
00:15:35,330 --> 00:15:39,250
exponential holding times
up until that point.
262
00:15:39,250 --> 00:15:42,100
And that state stays there until
the next exponential
263
00:15:42,100 --> 00:15:43,220
holding time.
264
00:15:43,220 --> 00:15:48,830
So this really gives you the
linkage between the Markov
265
00:15:48,830 --> 00:15:54,310
process in this expression,
and the markup process in
266
00:15:54,310 --> 00:15:57,660
terms of this graphical
expression here with the
267
00:15:57,660 --> 00:16:01,340
embedded chain, and the
exponential holding times.
268
00:16:04,060 --> 00:16:04,510
OK.
269
00:16:04,510 --> 00:16:09,320
You can visualize a transition
from one state to another in
270
00:16:09,320 --> 00:16:12,410
tree very convenient ways.
271
00:16:12,410 --> 00:16:16,570
And these are ways that we've, I
hope we've really learned to
272
00:16:16,570 --> 00:16:21,690
think about from looking
at Poisson processes.
273
00:16:21,690 --> 00:16:26,960
You can visualize this
transition by first using the
274
00:16:26,960 --> 00:16:32,820
next state by these transition
probabilities in the embedded
275
00:16:32,820 --> 00:16:34,160
Markov chain.
276
00:16:34,160 --> 00:16:36,840
And then you choose a transition
time, which is
277
00:16:36,840 --> 00:16:40,190
exponential with
rate new sub i.
278
00:16:40,190 --> 00:16:43,510
Equivalently, because it's a
Poisson process, you can
279
00:16:43,510 --> 00:16:44,840
choose the--
280
00:16:44,840 --> 00:16:48,490
well, no, because these are
independent given the state.
281
00:16:48,490 --> 00:16:52,190
You can choose the transition
time first, and then you can
282
00:16:52,190 --> 00:16:55,480
choose the state, because these
are independent of each
283
00:16:55,480 --> 00:16:59,520
other conditional on the
state that you're in.
284
00:16:59,520 --> 00:17:01,970
And finally, equivalently, which
is where the Poisson
285
00:17:01,970 --> 00:17:05,900
process comes in, a really neat
way to think of Poisson
286
00:17:05,900 --> 00:17:09,940
processes is to have an
enormously large number of
287
00:17:09,940 --> 00:17:12,720
Poisson processes running
all the time.
288
00:17:12,720 --> 00:17:17,030
There's one Poisson process for
every transition in this
289
00:17:17,030 --> 00:17:19,460
Markov chain.
290
00:17:19,460 --> 00:17:21,700
So you have accountably infinite
number of Poisson
291
00:17:21,700 --> 00:17:25,369
processes, which sounds a little
complicated at first.
292
00:17:25,369 --> 00:17:29,235
But you visualize a Poisson
process for each state pair i
293
00:17:29,235 --> 00:17:35,480
to j, which has a rate q sub ij,
which is the rate at which
294
00:17:35,480 --> 00:17:39,960
transitions occur out of
state i times p sub ij.
295
00:17:39,960 --> 00:17:45,250
This is the rate which, when
you're in state i, you will go
296
00:17:45,250 --> 00:17:47,890
to state j.
297
00:17:47,890 --> 00:17:51,120
And this makes use of all this
stuff about splitting and
298
00:17:51,120 --> 00:17:54,410
combining Poisson processes.
299
00:17:54,410 --> 00:18:01,810
If you have a Poisson process
which has rate new sub i and
300
00:18:01,810 --> 00:18:05,710
you split it into a number of
Poisson processes, for each
301
00:18:05,710 --> 00:18:08,970
next state you might go to,
you're splitting it into
302
00:18:08,970 --> 00:18:15,070
Poisson processes of rate new
sub i times p sub ij.
303
00:18:15,070 --> 00:18:19,100
And what's happening there is
there's a little switch.
304
00:18:19,100 --> 00:18:23,170
The little switch has
probabilities p sub ij, and
305
00:18:23,170 --> 00:18:30,950
that switch is telling you which
state to go to next.
306
00:18:30,950 --> 00:18:34,990
All of this is totally
artificial.
307
00:18:34,990 --> 00:18:39,200
And I hope by this time, you are
comfortable about looking
308
00:18:39,200 --> 00:18:43,020
at physical things in a totally
artificial way,
309
00:18:43,020 --> 00:18:47,660
because that's the magic
of mathematics.
310
00:18:47,660 --> 00:18:50,120
If you didn't have mathematics,
you couldn't look
311
00:18:50,120 --> 00:18:53,220
at real things in
artificial ways.
312
00:18:53,220 --> 00:18:56,500
And all the science would
suddenly disappear.
313
00:18:56,500 --> 00:19:02,700
So what we're doing here is
defining this Markov process
314
00:19:02,700 --> 00:19:05,600
in this artificial way of all
of these little Poisson
315
00:19:05,600 --> 00:19:09,460
processes, and we now know
how they all work.
316
00:19:09,460 --> 00:19:14,070
When the entry to state i, the
next state is the j with an x
317
00:19:14,070 --> 00:19:16,790
Poisson arrival, according
to q sub ij.
318
00:19:16,790 --> 00:19:19,900
So all these Poisson processes
are waiting to have an
319
00:19:19,900 --> 00:19:21,130
arrival come out.
320
00:19:21,130 --> 00:19:24,140
To have a race, one of
them wins, and you
321
00:19:24,140 --> 00:19:27,640
go off to that state.
322
00:19:27,640 --> 00:19:28,500
OK, question.
323
00:19:28,500 --> 00:19:31,900
What's the conditional
distribution of u1 given that
324
00:19:31,900 --> 00:19:37,270
x0 i, and x1 equals j?
325
00:19:37,270 --> 00:19:42,350
And to imagine this, suppose
that there are only two places
326
00:19:42,350 --> 00:19:46,060
you can go from state 0.
327
00:19:46,060 --> 00:19:50,560
You can go into state 1
with some very large
328
00:19:50,560 --> 00:19:53,500
probability, say 0.999.
329
00:19:53,500 --> 00:19:57,010
Or you can go into state 2 with
some very, very small
330
00:19:57,010 --> 00:19:59,210
probability.
331
00:19:59,210 --> 00:20:02,670
And what that means is this
exponential random variable
332
00:20:02,670 --> 00:20:08,530
going from state 0 into state
2 is a very, very
333
00:20:08,530 --> 00:20:10,570
slow, random variable.
334
00:20:10,570 --> 00:20:12,110
It has a very small rate.
335
00:20:12,110 --> 00:20:15,970
And the exponential random
variable going into state 1 is
336
00:20:15,970 --> 00:20:19,580
a very, very large
random variable.
337
00:20:19,580 --> 00:20:24,800
So you would think that if you
go from state 0 the state x2,
338
00:20:24,800 --> 00:20:29,675
it means it must take a very
long time to get there.
339
00:20:29,675 --> 00:20:31,400
Well, that's absolutely wrong.
340
00:20:35,180 --> 00:20:42,440
The time that it takes to go
from x0 to x2 is this random
341
00:20:42,440 --> 00:20:45,960
variable, u sub i.
342
00:20:45,960 --> 00:20:53,560
Where u sub i is the state you
happen to be in at this point
343
00:20:53,560 --> 00:20:54,810
when you're in--
344
00:21:00,310 --> 00:21:03,350
x0 is the state that
we start in.
345
00:21:03,350 --> 00:21:05,510
x0 is a random variable.
346
00:21:05,510 --> 00:21:08,210
It has some value i.
347
00:21:08,210 --> 00:21:11,665
With this value i, there's an
exponential random variable
348
00:21:11,665 --> 00:21:13,870
that determines how long
it takes you to
349
00:21:13,870 --> 00:21:15,640
get to the next state.
350
00:21:15,640 --> 00:21:19,700
This random variable conditional
on x0 equals i is
351
00:21:19,700 --> 00:21:23,270
independent of which state
you happen to go to.
352
00:21:23,270 --> 00:21:28,730
And what that means is that the
conditional distribution
353
00:21:28,730 --> 00:21:34,960
of u1, given x sub 0 is equal
to i, and x sub 1 equals j.
354
00:21:34,960 --> 00:21:39,380
If you've had your ears at all
open for the last 10 minutes,
355
00:21:39,380 --> 00:21:45,710
it is exponential with rate i.
356
00:21:45,710 --> 00:21:47,630
With rate new sub i.
357
00:21:50,400 --> 00:21:54,080
These holding times and the
next states you go to are
358
00:21:54,080 --> 00:21:57,000
independent of each other,
conditional on where you
359
00:21:57,000 --> 00:21:58,460
happen to be.
360
00:21:58,460 --> 00:22:02,390
It's the same thing we saw back
in Poisson processes.
361
00:22:02,390 --> 00:22:04,950
It was confusing as
hell back then.
362
00:22:04,950 --> 00:22:08,280
It is still confusing as hell.
363
00:22:08,280 --> 00:22:12,760
If you didn't get it sorted out
in your mind then, go back
364
00:22:12,760 --> 00:22:16,770
and think about it again now,
and try to get your common
365
00:22:16,770 --> 00:22:20,010
sense, which tells you when
you go to state 2, it must
366
00:22:20,010 --> 00:22:23,190
take a long time to get there,
because that exponential
367
00:22:23,190 --> 00:22:27,720
random variable has a very
long holding time.
368
00:22:27,720 --> 00:22:30,020
And that just isn't true.
369
00:22:30,020 --> 00:22:33,210
And it wasn't true when we were
dealing with a Poisson
370
00:22:33,210 --> 00:22:35,410
process, which got
split either.
371
00:22:35,410 --> 00:22:38,970
These two things are independent
of each other.
372
00:22:38,970 --> 00:22:44,840
Intuitively, why that is, I
almost hesitate to try to say
373
00:22:44,840 --> 00:22:49,950
why it is, because it's such
a tricky statement.
374
00:22:49,950 --> 00:22:55,010
If you happen to go to state 2
instead of to state 1, what's
375
00:22:55,010 --> 00:22:59,440
happening is that all of this
time that you're waiting to
376
00:22:59,440 --> 00:23:04,300
have a state transition, when
you finally have this state
377
00:23:04,300 --> 00:23:08,800
transition, you then flip a
switch to see which state
378
00:23:08,800 --> 00:23:10,350
you're going to go to.
379
00:23:10,350 --> 00:23:13,740
And the fact that it's taken you
a long time to get there
380
00:23:13,740 --> 00:23:17,680
says nothing whatsoever about
what this switch is doing,
381
00:23:17,680 --> 00:23:21,020
because that switch is
independent of how long it
382
00:23:21,020 --> 00:23:24,360
takes you for the switch
to operate.
383
00:23:24,360 --> 00:23:29,060
And I know.
384
00:23:29,060 --> 00:23:32,720
It's not entirely intuitive,
and you just have to beat
385
00:23:32,720 --> 00:23:36,530
yourself on the head until
it becomes intuitive.
386
00:23:36,530 --> 00:23:38,390
I've beaten myself on
the head until I can
387
00:23:38,390 --> 00:23:39,630
hardly think straight.
388
00:23:39,630 --> 00:23:42,680
It still isn't intuitive to me,
but maybe it will become
389
00:23:42,680 --> 00:23:44,030
intuitive to you.
390
00:23:44,030 --> 00:23:47,420
I hope so.
391
00:23:47,420 --> 00:23:51,690
So anyway, this gives the
conditional distribution of u1
392
00:23:51,690 --> 00:23:54,618
given that x0 is equal to i,
and x1 is equal to-- no.
393
00:23:58,450 --> 00:24:03,530
This says that the exponential
rate out of state i is equal
394
00:24:03,530 --> 00:24:07,660
to the sum of the exponential
rates to each of the states we
395
00:24:07,660 --> 00:24:08,890
might be going to.
396
00:24:08,890 --> 00:24:10,620
We have to go to some
other state.
397
00:24:10,620 --> 00:24:14,110
We have no self transitions as
far as we're concerned here.
398
00:24:14,110 --> 00:24:16,990
Even if we had self transitions,
this formula
399
00:24:16,990 --> 00:24:19,380
would still be correct, but
it's easier to think of it
400
00:24:19,380 --> 00:24:21,730
without self transitions.
401
00:24:21,730 --> 00:24:25,720
p sub ij, this is the
switch probability.
402
00:24:25,720 --> 00:24:29,950
It's q sub ij divided
by new sub i.
403
00:24:29,950 --> 00:24:33,360
This is the probability you're
going to go to j given that
404
00:24:33,360 --> 00:24:35,550
you were in state i.
405
00:24:35,550 --> 00:24:42,400
The matrix of all of these cues
specifies the matrix of
406
00:24:42,400 --> 00:24:46,880
all of these p's, and
it specifies new.
407
00:24:46,880 --> 00:24:49,900
That's what this formula says.
408
00:24:49,900 --> 00:24:54,190
If I know what q is, I know what
new is, I know what p is.
409
00:24:54,190 --> 00:24:57,885
And we've already said that if
you know what p is and you
410
00:24:57,885 --> 00:25:00,700
know what new is, you know
what q sub ij is.
411
00:25:00,700 --> 00:25:04,320
So these are completely
equivalent representations of
412
00:25:04,320 --> 00:25:05,140
the same thing.
413
00:25:05,140 --> 00:25:07,760
You can work with either
one you want to.
414
00:25:07,760 --> 00:25:12,060
Sometimes one is useful,
sometimes the other is useful.
415
00:25:12,060 --> 00:25:16,250
If you look at an mm1q,
mm1q, you remember,
416
00:25:16,250 --> 00:25:18,720
is exponential arrivals.
417
00:25:18,720 --> 00:25:20,270
Exponential service time.
418
00:25:20,270 --> 00:25:23,970
The time of the service is
independent of when it
419
00:25:23,970 --> 00:25:26,450
happens, or who it happens to.
420
00:25:26,450 --> 00:25:29,770
These service times are just
individual exponential random
421
00:25:29,770 --> 00:25:31,670
variables of some rate mu.
422
00:25:31,670 --> 00:25:34,950
The arrivals are the
inter-arrival intervals are
423
00:25:34,950 --> 00:25:40,980
exponential random variables
of rate lambda.
424
00:25:40,980 --> 00:25:44,860
So when you're sitting in state
0, the only place you
425
00:25:44,860 --> 00:25:46,210
can is to state one.
426
00:25:46,210 --> 00:25:52,260
You're sitting there, and
if you're in state 0,
427
00:25:52,260 --> 00:25:53,510
the server is idle.
428
00:25:55,770 --> 00:25:58,880
You're waiting for the first
arrival to occur.
429
00:25:58,880 --> 00:26:02,140
The next thing that happens has
to be an arrival because
430
00:26:02,140 --> 00:26:03,850
it can't be a service.
431
00:26:03,850 --> 00:26:07,430
So the transition here is
with probability 1.
432
00:26:07,430 --> 00:26:11,010
All these other transitions are
with probability lambda
433
00:26:11,010 --> 00:26:14,590
divided by lambda plus mu,
because in all of these other
434
00:26:14,590 --> 00:26:18,720
states, you can get an arrival
or a departure.
435
00:26:18,720 --> 00:26:21,400
Each of them are exponential
random variables.
436
00:26:21,400 --> 00:26:25,040
The switch probability that
we're talking about is in
437
00:26:25,040 --> 00:26:29,300
lambda over lambda
plus mu to go up.
438
00:26:29,300 --> 00:26:32,540
Mu over lambda plus mu
to go down for all
439
00:26:32,540 --> 00:26:35,210
states other than 0.
440
00:26:35,210 --> 00:26:38,400
If you write this, this
is in terms of the
441
00:26:38,400 --> 00:26:40,520
embedded Markov chain.
442
00:26:40,520 --> 00:26:43,200
And if you write this in
terms of the transition
443
00:26:43,200 --> 00:26:46,760
probabilities, it
looks like this.
444
00:26:46,760 --> 00:26:48,860
Which is simpler?
445
00:26:48,860 --> 00:26:50,715
Which is more transparent?
446
00:26:50,715 --> 00:26:53,830
Well this, really, is
what the mm1q is.
447
00:26:53,830 --> 00:26:55,440
That's where we started
when we started
448
00:26:55,440 --> 00:26:56,810
talking about mm1q's.
449
00:27:02,360 --> 00:27:05,880
This in this situation is
certainly far simpler.
450
00:27:05,880 --> 00:27:08,260
You're giving these transition
probabilities.
451
00:27:08,260 --> 00:27:11,710
But don't forget that we still
have this embedded Markov
452
00:27:11,710 --> 00:27:14,330
chain in the background.
453
00:27:14,330 --> 00:27:17,430
Both these graphs have
the same information.
454
00:27:17,430 --> 00:27:20,250
Both these graphs have the same
information for every
455
00:27:20,250 --> 00:27:21,870
Markov process you want
456
00:27:21,870 --> 00:27:25,740
to talk about OK.
457
00:27:25,740 --> 00:27:27,040
Let's look at sample time
458
00:27:27,040 --> 00:27:29,560
approximations to Markov processes.
459
00:27:33,280 --> 00:27:35,700
And we already did it
in the last chapter.
460
00:27:35,700 --> 00:27:38,950
We just didn't talk about
it quite so much.
461
00:27:38,950 --> 00:27:43,580
We quantized time to increments
of delta.
462
00:27:43,580 --> 00:27:47,550
We viewed all Poisson processes
in a Markov process.
463
00:27:47,550 --> 00:27:50,570
Remember, we can view all
of these transitions as
464
00:27:50,570 --> 00:27:53,310
independent Poisson processes
all running
465
00:27:53,310 --> 00:27:55,890
away at the same time.
466
00:27:55,890 --> 00:27:59,330
We can view all of these Poisson
processes as Bernoulli
467
00:27:59,330 --> 00:28:05,455
processes with probability of
a transition from i to j in
468
00:28:05,455 --> 00:28:10,930
the increment delta, given as
a delta times qij, to first
469
00:28:10,930 --> 00:28:12,550
order in delta.
470
00:28:12,550 --> 00:28:17,610
So we can take this Markov
process, turn it into a rather
471
00:28:17,610 --> 00:28:20,530
strange kind of Bernoulli
process.
472
00:28:20,530 --> 00:28:24,310
For the M/M/1 queue, all that's
doing is turning into a
473
00:28:24,310 --> 00:28:26,700
sample time, M/M/1 process.
474
00:28:26,700 --> 00:28:29,680
And we can sort of think
the same way about
475
00:28:29,680 --> 00:28:31,820
general Markov processes.
476
00:28:31,820 --> 00:28:34,620
We'll see when we can
and when we can't.
477
00:28:34,620 --> 00:28:37,840
Since shrinking Bernoulli goes
to Poisson, we would
478
00:28:37,840 --> 00:28:42,430
conjecture the limiting Markov
chain as delta goes to 0 goes
479
00:28:42,430 --> 00:28:43,710
through a Markov process.
480
00:28:43,710 --> 00:28:46,030
In a sense that X of t is
481
00:28:46,030 --> 00:28:48,940
approximately equal to X prime.
482
00:28:48,940 --> 00:28:53,430
This is X prime as in
the Bernoulli domain
483
00:28:53,430 --> 00:28:55,340
at delta times n.
484
00:28:58,460 --> 00:29:00,910
You have to put self-transition
into a sample
485
00:29:00,910 --> 00:29:02,540
time approximation.
486
00:29:02,540 --> 00:29:06,280
Because if you have a very small
delta, there aren't big
487
00:29:06,280 --> 00:29:09,760
enough transition probabilities
going out of the
488
00:29:09,760 --> 00:29:12,880
chain to fill up the
probability space.
489
00:29:12,880 --> 00:29:15,710
So in most transition, you're
going to just have a
490
00:29:15,710 --> 00:29:17,210
self-transition.
491
00:29:17,210 --> 00:29:21,120
So you need a self-transition,
which is 1 minus delta times
492
00:29:21,120 --> 00:29:25,470
nu sub i, and these transitions
to other states,
493
00:29:25,470 --> 00:29:28,350
which are delta times
q sub ij.
494
00:29:28,350 --> 00:29:34,460
This has the advantage that if
you believe this, you can
495
00:29:34,460 --> 00:29:37,300
ignore everything we're saying
about Poisson processes
496
00:29:37,300 --> 00:29:39,950
because you already
know all of it.
497
00:29:39,950 --> 00:29:44,350
We already talked about
sample time processes.
498
00:29:44,350 --> 00:29:48,720
You can do this for any
old process almost.
499
00:29:48,720 --> 00:29:52,400
And when you do this for any old
process, you're turning it
500
00:29:52,400 --> 00:29:55,860
into a Markov change instead
of a Markov process.
501
00:29:55,860 --> 00:29:59,310
This is the same argument you
tried to use when you were a
502
00:29:59,310 --> 00:30:02,500
senior in high school or
freshman in college when you
503
00:30:02,500 --> 00:30:06,850
said, I don't have to learn
calculus, because all it is is
504
00:30:06,850 --> 00:30:10,180
just taking increments
to be very small and
505
00:30:10,180 --> 00:30:11,260
looking at a limit.
506
00:30:11,260 --> 00:30:14,152
So I will just ignore
all that stuff.
507
00:30:14,152 --> 00:30:15,940
It didn't work there.
508
00:30:15,940 --> 00:30:17,190
It doesn't work here.
509
00:30:19,690 --> 00:30:24,630
But it's a good thing to go
every time you get confused,
510
00:30:24,630 --> 00:30:27,460
because this you can sort
out for yourself.
511
00:30:27,460 --> 00:30:30,150
There is one problem here.
512
00:30:30,150 --> 00:30:36,410
When you start making shrieking
delta more and more,
513
00:30:36,410 --> 00:30:41,340
if you want to get a sample time
approximation, delta has
514
00:30:41,340 --> 00:30:45,660
to be smaller than 1 over the
maximum of the nu sub i's.
515
00:30:45,660 --> 00:30:49,600
If it's not smaller than the
maximum of the nu sub i's, the
516
00:30:49,600 --> 00:30:54,220
self-transition probability
here is
517
00:30:54,220 --> 00:30:57,110
unfortunately negative.
518
00:30:57,110 --> 00:31:00,370
And we don't like negative
probabilities.
519
00:31:00,370 --> 00:31:02,570
So you can't do that.
520
00:31:02,570 --> 00:31:05,910
If you have a Markov process,
has a countably infinite
521
00:31:05,910 --> 00:31:10,320
number of states, each of these
nu sub i's are positive.
522
00:31:10,320 --> 00:31:12,900
But they can approach
0 as a limit.
523
00:31:12,900 --> 00:31:17,240
As if they approach 0 as a
limit, you cannot describe a
524
00:31:17,240 --> 00:31:22,520
sample time Markov chain to go
with the Markov process.
525
00:31:22,520 --> 00:31:25,090
All you can do is truncate
the chain also and
526
00:31:25,090 --> 00:31:26,090
then see what happens.
527
00:31:26,090 --> 00:31:29,366
And that's often a good
way to do it.
528
00:31:29,366 --> 00:31:32,526
OK, so we can always do this
sample time approximation.
529
00:31:35,030 --> 00:31:42,640
What is nice about the sample
time approximation is that we
530
00:31:42,640 --> 00:31:46,170
will find it in general, if you
could use the sample time
531
00:31:46,170 --> 00:31:50,500
approximation, it always gives
you the exact steady state
532
00:31:50,500 --> 00:31:51,820
probabilities.
533
00:31:51,820 --> 00:31:56,010
No matter how crude you are in
this approximation, you always
534
00:31:56,010 --> 00:31:59,420
wind up with the exact rate
values when you're all done.
535
00:31:59,420 --> 00:32:01,070
I don't know why.
536
00:32:01,070 --> 00:32:05,210
We will essentially prove
today that that happens.
537
00:32:05,210 --> 00:32:06,750
But that's a nice thing.
538
00:32:06,750 --> 00:32:10,030
But it doesn't work with the nu
sub i's approach 0, because
539
00:32:10,030 --> 00:32:11,550
then you can't get
a sample time
540
00:32:11,550 --> 00:32:14,090
approximation to start with.
541
00:32:14,090 --> 00:32:17,820
OK, let's look at the embedded
chain model in the sample time
542
00:32:17,820 --> 00:32:20,315
model of M/M/1 queue.
543
00:32:20,315 --> 00:32:23,200
I hate to keep coming back
to the M/M/1 queue.
544
00:32:23,200 --> 00:32:27,340
But for Markov processes,
they're all so similar to each
545
00:32:27,340 --> 00:32:32,780
other that you might as well
get very familiar with one
546
00:32:32,780 --> 00:32:36,290
particular model of them,
because that one particular
547
00:32:36,290 --> 00:32:39,840
model tells you most of the
distinctions that you have to
548
00:32:39,840 --> 00:32:41,090
be careful about.
549
00:32:43,990 --> 00:32:44,560
Let's see.
550
00:32:44,560 --> 00:32:45,170
What is this?
551
00:32:45,170 --> 00:32:47,240
This is the embedded chain
model that we've
552
00:32:47,240 --> 00:32:48,680
talked about before.
553
00:32:48,680 --> 00:32:51,210
When you're in state 0,
the only place you can
554
00:32:51,210 --> 00:32:53,190
go is to state 1.
555
00:32:53,190 --> 00:32:57,730
When you're in state 1, you
can go down with some
556
00:32:57,730 --> 00:32:58,840
probability.
557
00:32:58,840 --> 00:33:01,590
You can go up with
some probability.
558
00:33:01,590 --> 00:33:05,760
And since these probabilities
have to add up to 1, it's mu
559
00:33:05,760 --> 00:33:09,600
over lambda plus mu and lambda
over lambda plus mu, and the
560
00:33:09,600 --> 00:33:13,310
same thing forever after.
561
00:33:13,310 --> 00:33:17,650
If we're dealing with the sample
time model, what we
562
00:33:17,650 --> 00:33:22,750
wind up with is we start
out with qij,
563
00:33:22,750 --> 00:33:24,770
which is lambda here.
564
00:33:27,610 --> 00:33:32,070
The time it takes to get from
state 0 to make a transition,
565
00:33:32,070 --> 00:33:36,100
the only place you can make a
transition to is state 1.
566
00:33:36,100 --> 00:33:38,730
You make those transitions
at rate lambda.
567
00:33:38,730 --> 00:33:43,470
So the sample time model has
this transition and discrete
568
00:33:43,470 --> 00:33:47,300
time with probability lambda
delta, this transition with
569
00:33:47,300 --> 00:33:50,520
probability mu delta
and so forth up.
570
00:33:50,520 --> 00:33:54,380
You need these self-transitions
in order to
571
00:33:54,380 --> 00:33:56,580
make things add up correctly.
572
00:33:56,580 --> 00:34:02,080
The steady state for the
embedded chain is pi sub 0
573
00:34:02,080 --> 00:34:04,880
equals 1 minus rho over 2.
574
00:34:04,880 --> 00:34:07,970
How do I know that?
575
00:34:07,970 --> 00:34:10,550
You just have to use
algebra for that.
576
00:34:10,550 --> 00:34:11,800
But it's very easy.
577
00:34:24,850 --> 00:34:28,659
I'm going to have to get
three of these things.
578
00:34:28,659 --> 00:34:31,880
OK, any time you have a
birth-death chain, you can
579
00:34:31,880 --> 00:34:33,389
find the steady state
probabilities.
580
00:34:54,030 --> 00:34:56,880
The probability of going this
way is equal to the
581
00:34:56,880 --> 00:34:58,870
probability of going this way.
582
00:34:58,870 --> 00:35:02,195
If you add in the probability
of the steady state that
583
00:35:02,195 --> 00:35:05,440
you're concerned with, steady
state transition this way, the
584
00:35:05,440 --> 00:35:07,300
same as the probability
of a steady state
585
00:35:07,300 --> 00:35:08,670
transition this way.
586
00:35:08,670 --> 00:35:11,630
And you remember, the reason for
this is in a birth-death
587
00:35:11,630 --> 00:35:16,450
chain, the total number of
transitions from here to here
588
00:35:16,450 --> 00:35:19,820
has to be within 1 of the number
of transitions from
589
00:35:19,820 --> 00:35:21,650
here to there.
590
00:35:21,650 --> 00:35:25,670
So that if steady state
means anything--
591
00:35:25,670 --> 00:35:30,440
and if these of long-term sample
space probabilities
592
00:35:30,440 --> 00:35:33,380
with probability 1
mean anything--
593
00:35:33,380 --> 00:35:35,320
this has to be true.
594
00:35:35,320 --> 00:35:38,600
So when you do that, this
is what you get here.
595
00:35:38,600 --> 00:35:42,100
This is a strange 1
minus rho over 2.
596
00:35:42,100 --> 00:35:45,660
It's strange because of this
strange probability one here,
597
00:35:45,660 --> 00:35:49,180
and this strange probability mu
over lambda plus mu here.
598
00:35:49,180 --> 00:35:52,530
And otherwise, everything is
symmetric, so it looks the
599
00:35:52,530 --> 00:35:55,490
same as this one here.
600
00:35:55,490 --> 00:35:58,730
For this one, the steady state
for the sample time doesn't
601
00:35:58,730 --> 00:36:00,420
depend on delta.
602
00:36:00,420 --> 00:36:04,590
And it's pi sub i prime equals 1
minus rho times rho to the i
603
00:36:04,590 --> 00:36:06,820
where rho equals
lambda over mu.
604
00:36:06,820 --> 00:36:08,620
This is what we did before.
605
00:36:08,620 --> 00:36:13,450
And what we found is since
transitions this way have to
606
00:36:13,450 --> 00:36:18,690
equal transitions this way,
these self-transitions don't
607
00:36:18,690 --> 00:36:21,130
make any difference here.
608
00:36:21,130 --> 00:36:25,200
And you get the same answer
no matter what delta is.
609
00:36:25,200 --> 00:36:31,070
And therefore you have pretty
much a conviction, which can't
610
00:36:31,070 --> 00:36:34,770
totally rely on, that you can go
to the limit as delta goes
611
00:36:34,770 --> 00:36:38,800
to 0 and find out what is going
on in the actual Markov
612
00:36:38,800 --> 00:36:40,250
process itself.
613
00:36:40,250 --> 00:36:45,900
You'll be very surprised with
this result if this were not
614
00:36:45,900 --> 00:36:50,530
the result for steady state
probabilities in some sense
615
00:36:50,530 --> 00:36:52,860
for the Markov process.
616
00:36:52,860 --> 00:36:59,790
However, the embedded chain
probabilities and these
617
00:36:59,790 --> 00:37:03,650
probabilities down here
are not the same.
618
00:37:03,650 --> 00:37:07,730
What's the difference
between them?
619
00:37:07,730 --> 00:37:11,700
For the embedded chain, what
you're talking about is the
620
00:37:11,700 --> 00:37:15,210
ratio of transitions
that go from one
621
00:37:15,210 --> 00:37:17,850
state to another state.
622
00:37:17,850 --> 00:37:21,130
When you're dealing with the
process, what you're talking
623
00:37:21,130 --> 00:37:26,250
about is the probability that
you will be in one state.
624
00:37:26,250 --> 00:37:29,030
If when you get in one state
you stay there for a long
625
00:37:29,030 --> 00:37:36,770
time, because the rate of
transitions out of that state
626
00:37:36,770 --> 00:37:40,260
is very small, so you're going
to stay there for a long time.
627
00:37:40,260 --> 00:37:43,940
That enhances the probability
of being in that state.
628
00:37:43,940 --> 00:37:48,220
You see that right here,
because pi 0 is 1
629
00:37:48,220 --> 00:37:50,410
minus rho over 2.
630
00:37:50,410 --> 00:37:58,060
And pi 0 prime is 1 minus rho.
631
00:37:58,060 --> 00:37:59,246
It's bigger.
632
00:37:59,246 --> 00:38:02,270
And it's bigger because you're
going to stay there longer,
633
00:38:02,270 --> 00:38:04,830
because the rate of getting out
of there is not as big as
634
00:38:04,830 --> 00:38:07,210
it was before.
635
00:38:07,210 --> 00:38:09,810
So the steady state
probabilities in the embedded
636
00:38:09,810 --> 00:38:15,000
chain and the steady state
probabilities and the sample
637
00:38:15,000 --> 00:38:17,700
time approximation
are different.
638
00:38:17,700 --> 00:38:20,410
And the steady state
probabilities and the sample
639
00:38:20,410 --> 00:38:25,340
time approximation are the same,
when you go to the limit
640
00:38:25,340 --> 00:38:28,080
of infinitely fine
sample time.
641
00:38:28,080 --> 00:38:31,440
Now what we have to do is go
back and look at renewal
642
00:38:31,440 --> 00:38:35,340
theory and all those that and
actually convince ourselves
643
00:38:35,340 --> 00:38:36,590
that this works.
644
00:38:39,819 --> 00:38:43,300
OK, so here we have renewals
for Markov processes.
645
00:38:46,380 --> 00:38:48,100
And what have we done so far?
646
00:38:48,100 --> 00:38:50,400
We've been looking at
the Poisson process.
647
00:38:50,400 --> 00:38:53,150
We've been looking
at Markov chains.
648
00:38:53,150 --> 00:38:57,180
And we've been trying to refer
to this new kind of process.
649
00:38:57,180 --> 00:39:01,460
Now we bring in the last
actor, renewal theory.
650
00:39:01,460 --> 00:39:06,170
And as usual, Poisson processes
gives you the easy
651
00:39:06,170 --> 00:39:07,900
way to look at a problem.
652
00:39:07,900 --> 00:39:11,070
Markov chains gives you a way
to look at the problem, when
653
00:39:11,070 --> 00:39:13,680
you'd rather write equations
and think about it.
654
00:39:13,680 --> 00:39:16,130
And we renewal theory gives
you the way to look at the
655
00:39:16,130 --> 00:39:20,030
problem when you really are a
glutton for punishment, and
656
00:39:20,030 --> 00:39:23,200
you want to spend a lot of
time thinking about it.
657
00:39:23,200 --> 00:39:25,040
And you don't want to write any
equations, or you don't
658
00:39:25,040 --> 00:39:26,900
want to write many equation.
659
00:39:26,900 --> 00:39:31,240
OK, an irreducible Markov
process is a Markov process
660
00:39:31,240 --> 00:39:35,350
for which the embedded Markov
chain is irreducible.
661
00:39:35,350 --> 00:39:38,840
Remember that an irreducible a
Markov chain is one where all
662
00:39:38,840 --> 00:39:42,270
states are in the same class.
663
00:39:42,270 --> 00:39:47,970
We saw that irreducible Markov
chains when we had a countably
664
00:39:47,970 --> 00:39:51,122
infinite number of states, that
they could be transient,
665
00:39:51,122 --> 00:39:53,300
the state simply wanders
off with high
666
00:39:53,300 --> 00:39:55,300
probability, never to return.
667
00:39:55,300 --> 00:39:58,670
If you have an M/M/1 queue,
and the expect the service
668
00:39:58,670 --> 00:40:04,340
time is bigger than the
expected time between
669
00:40:04,340 --> 00:40:08,970
arrivals, then gradually
the queue builds up.
670
00:40:08,970 --> 00:40:12,480
The queue keeps getting longer
and longer as time goes on.
671
00:40:12,480 --> 00:40:14,470
There isn't any steady state.
672
00:40:14,470 --> 00:40:17,200
Looked at another way, the
steady state probabilities are
673
00:40:17,200 --> 00:40:21,510
always 0, if you want to
just calculate them.
674
00:40:21,510 --> 00:40:26,700
So we're going to see the
irreducible Markov processes
675
00:40:26,700 --> 00:40:30,670
can have even more bizarre
behavior than these Markov
676
00:40:30,670 --> 00:40:32,220
chains can.
677
00:40:32,220 --> 00:40:36,590
And part of that more bizarre
behavior is infinitely many
678
00:40:36,590 --> 00:40:40,790
transitions in a finite time.
679
00:40:40,790 --> 00:40:44,540
I mean, how do you talk about
steady state when you have an
680
00:40:44,540 --> 00:40:48,680
infinite number of transitions
and a finite time?
681
00:40:48,680 --> 00:40:50,950
I mean, essentially, the
Markov process is
682
00:40:50,950 --> 00:40:53,060
blowing up on you.
683
00:40:53,060 --> 00:40:55,900
Transitions get more
and more frequent.
684
00:40:55,900 --> 00:40:57,790
They go off to infinity.
685
00:40:57,790 --> 00:40:58,890
What do you do after that?
686
00:40:58,890 --> 00:41:00,650
I don't know.
687
00:41:00,650 --> 00:41:02,010
I can write these equations.
688
00:41:02,010 --> 00:41:03,860
I can solve these equations.
689
00:41:03,860 --> 00:41:05,840
But they don't mean anything.
690
00:41:05,840 --> 00:41:10,170
In other words sometimes,
talking about steady state, we
691
00:41:10,170 --> 00:41:13,120
usually write equations
for steady state.
692
00:41:13,120 --> 00:41:17,720
But as we saw with countable
state Markov chains, steady
693
00:41:17,720 --> 00:41:19,980
state doesn't always
exist there.
694
00:41:19,980 --> 00:41:23,520
There it evidenced itself with
steady state probabilities
695
00:41:23,520 --> 00:41:27,290
that were equal to 0, which
said that as time went on,
696
00:41:27,290 --> 00:41:30,420
things just got very diffused
or things wandered off to
697
00:41:30,420 --> 00:41:32,830
infinity or something
like that.
698
00:41:32,830 --> 00:41:37,170
Here it's this much worse thing,
where in fact you get
699
00:41:37,170 --> 00:41:40,070
an infinite number of
transitions very fast.
700
00:41:40,070 --> 00:41:45,000
And we'll see how that happens
a little later.
701
00:41:45,000 --> 00:41:48,770
You might have a transition
rate which goes down to 0.
702
00:41:48,770 --> 00:41:53,030
The process is chugging along
and gets slower, and slower,
703
00:41:53,030 --> 00:41:57,070
and slower, and pretty soon
nothing happens anymore.
704
00:41:57,070 --> 00:42:01,250
Well, always something happens
if you wait long enough, but
705
00:42:01,250 --> 00:42:04,000
as you wait longer and
longer, things happen
706
00:42:04,000 --> 00:42:07,520
more and more slowly.
707
00:42:07,520 --> 00:42:10,750
So we'll see all of these
things, and we'll see
708
00:42:10,750 --> 00:42:12,420
how this comes out.
709
00:42:12,420 --> 00:42:14,650
OK, let's review briefly
accountable
710
00:42:14,650 --> 00:42:17,060
state Markov chains--
711
00:42:17,060 --> 00:42:19,730
an irreducible, that means
everything can talk to
712
00:42:19,730 --> 00:42:24,550
everything else; is positive
recurrent if and only if the
713
00:42:24,550 --> 00:42:30,110
steady state equations, pi sub
j equals the sum of pi
714
00:42:30,110 --> 00:42:31,510
sub i, p sub ij.
715
00:42:31,510 --> 00:42:32,940
Remember what this is.
716
00:42:36,310 --> 00:42:40,120
If you're in steady state, the
probability of being in a
717
00:42:40,120 --> 00:42:42,730
state j is supposed
to be pi sub j.
718
00:42:42,730 --> 00:42:46,470
The probability of being in a
state is supposed to be equal
719
00:42:46,470 --> 00:42:49,840
then to the sum of the
probabilities of being in
720
00:42:49,840 --> 00:42:52,310
another state and going
to that state.
721
00:42:52,310 --> 00:42:54,100
That's the way it has to
be if you're going to
722
00:42:54,100 --> 00:42:55,610
have a steady state.
723
00:42:55,610 --> 00:42:57,220
So this is necessary.
724
00:42:57,220 --> 00:43:00,200
The pi sub j's have to be
greater than or equal to 0.
725
00:43:00,200 --> 00:43:03,410
And the sum of the pi sub
j's is equal to 1.
726
00:43:03,410 --> 00:43:04,710
It has a solution.
727
00:43:04,710 --> 00:43:08,970
If this has a solution, it's
unique, and if pi sub i is
728
00:43:08,970 --> 00:43:15,680
greater than 0 for all i, if
it's positive recurrent.
729
00:43:15,680 --> 00:43:17,730
We saw that if it wasn't
positive recurrent, other
730
00:43:17,730 --> 00:43:19,340
things could happen.
731
00:43:19,340 --> 00:43:23,950
Also the number of visits, n
sub ij of n, remember in a
732
00:43:23,950 --> 00:43:29,220
Markov chain, what we talked
about when we used renewal
733
00:43:29,220 --> 00:43:34,200
theory was the number of visits
over a particular
734
00:43:34,200 --> 00:43:39,010
number transitions
from i to j.
735
00:43:39,010 --> 00:43:45,390
n sub ij of n is the number
of times we hit j's
736
00:43:45,390 --> 00:43:46,640
in the first n trials.
737
00:43:51,350 --> 00:43:54,120
I always do this.
738
00:43:54,120 --> 00:43:59,815
Please take that n sub ij of n
and write 1 over n times n sub
739
00:43:59,815 --> 00:44:02,340
ij of n is equal to pi j.
740
00:44:02,340 --> 00:44:03,550
You all know that.
741
00:44:03,550 --> 00:44:06,140
I know it too.
742
00:44:06,140 --> 00:44:10,260
I don't know why it always gets
left off of my slides.
743
00:44:10,260 --> 00:44:14,190
Now, we guessed for a Markov
process the fraction of time
744
00:44:14,190 --> 00:44:19,337
in state j should be p sub j
equals pi sub j over a nu sub
745
00:44:19,337 --> 00:44:25,440
j divided by the sum over i
of pi sub i over nu sub i.
746
00:44:25,440 --> 00:44:27,590
Perhaps I should say I guessed
that because I
747
00:44:27,590 --> 00:44:30,190
already know it.
748
00:44:30,190 --> 00:44:35,270
I want to indicate to you why
if you didn't know anything
749
00:44:35,270 --> 00:44:38,510
and if you weren't suspicious
by this time, you would make
750
00:44:38,510 --> 00:44:41,450
that guess, OK?
751
00:44:41,450 --> 00:44:44,490
We had this embedded
Markov chain.
752
00:44:44,490 --> 00:44:47,480
Over a very long period
of time, the number of
753
00:44:47,480 --> 00:44:53,570
transitions into state i is
going to be the number of
754
00:44:53,570 --> 00:44:58,090
transitions into state i
divided by n is going
755
00:44:58,090 --> 00:44:59,460
to be pi sub i.
756
00:44:59,460 --> 00:45:02,940
That's what this equation here
says, or what it would say if
757
00:45:02,940 --> 00:45:05,830
I had written it correctly.
758
00:45:05,830 --> 00:45:09,720
Now, each time we get to pi
sub i, we're going to stay
759
00:45:09,720 --> 00:45:11,220
there for a while.
760
00:45:11,220 --> 00:45:16,270
The holding time in state
i is proportional to
761
00:45:16,270 --> 00:45:18,490
1 over nu sub i.
762
00:45:18,490 --> 00:45:22,230
The rate of the next transition
is nu sub i.
763
00:45:22,230 --> 00:45:26,272
So the expected holding time is
going to be 1 over nu sub
764
00:45:26,272 --> 00:45:31,150
i, which says that the fraction
of time that we're
765
00:45:31,150 --> 00:45:38,510
actually in state i should be
proportional to the number of
766
00:45:38,510 --> 00:45:43,250
times we go into state j times
the expected holding
767
00:45:43,250 --> 00:45:44,930
time in state j.
768
00:45:44,930 --> 00:45:49,350
Now when you write p sub j
equals pi sub j over nu sub j,
769
00:45:49,350 --> 00:45:51,740
you have a constant there
which is missing.
770
00:45:51,740 --> 00:45:55,790
Because what we're doing is
we're amortizing this over
771
00:45:55,790 --> 00:45:57,420
some long period of time.
772
00:45:57,420 --> 00:46:00,520
And we don't know what the
constant of amortization is.
773
00:46:00,520 --> 00:46:04,120
But these probability
should add up to 1.
774
00:46:04,120 --> 00:46:08,610
If life is at all fair to us,
the fraction of time that we
775
00:46:08,610 --> 00:46:13,030
spent in each state j should be
some number which adds up
776
00:46:13,030 --> 00:46:15,620
to 1 as we sum over j.
777
00:46:15,620 --> 00:46:18,230
So this is just a normalization
factor that you
778
00:46:18,230 --> 00:46:21,910
need to make the p sub
j's sum up to 1.
779
00:46:21,910 --> 00:46:25,800
Now what this means physically,
and why it appears
780
00:46:25,800 --> 00:46:28,630
here, is something we have to go
through some more analysis.
781
00:46:28,630 --> 00:46:30,880
But this is what we
would guess if we
782
00:46:30,880 --> 00:46:32,270
didn't know any better.
783
00:46:32,270 --> 00:46:34,860
And in fact, it's pretty
much true.
784
00:46:34,860 --> 00:46:36,960
It's not always true, but
it's pretty much true.
785
00:46:39,710 --> 00:46:42,200
So now let's use renewal
theory to actually see
786
00:46:42,200 --> 00:46:43,790
what's going on.
787
00:46:43,790 --> 00:46:47,440
And here's where we need a
little more notation even.
788
00:46:47,440 --> 00:46:54,140
Let n sub i of t be the number
of transitions between 0 and t
789
00:46:54,140 --> 00:46:58,060
for a Markov process starting
in state i.
790
00:46:58,060 --> 00:47:01,140
I can't talk about the number of
transitions if I don't say
791
00:47:01,140 --> 00:47:03,610
what state we start in, because
then I don't really
792
00:47:03,610 --> 00:47:06,120
have a random variable.
793
00:47:06,120 --> 00:47:10,110
I could say let's start and
steady state, and that seems
794
00:47:10,110 --> 00:47:12,570
very, very appealing.
795
00:47:12,570 --> 00:47:14,920
I've tried to do that many
times, because it would
796
00:47:14,920 --> 00:47:17,470
simplify all these theorems.
797
00:47:17,470 --> 00:47:21,330
And it just doesn't
work, believe me.
798
00:47:21,330 --> 00:47:24,840
So let's take the extra pain
of saying let's start
799
00:47:24,840 --> 00:47:26,200
in some state i.
800
00:47:26,200 --> 00:47:29,220
We don't know what it is, but
we'll just assume there is
801
00:47:29,220 --> 00:47:31,850
some state.
802
00:47:31,850 --> 00:47:36,420
And the theorem says that the
limit of M sub i of t is equal
803
00:47:36,420 --> 00:47:37,620
to infinity.
804
00:47:37,620 --> 00:47:39,950
Here I don't have a 1 over
t in front of it.
805
00:47:39,950 --> 00:47:42,470
I've written incorrectly.
806
00:47:42,470 --> 00:47:45,210
And this is a very technical
theorem.
807
00:47:45,210 --> 00:47:48,220
We proved the same kind of
technical theorem when we were
808
00:47:48,220 --> 00:47:50,860
talking about Markov chains,
if you remember.
809
00:47:50,860 --> 00:47:52,820
We were talking about
Markov chains.
810
00:47:52,820 --> 00:47:59,910
We said that in some sense, an
infinite number of transitions
811
00:47:59,910 --> 00:48:02,520
into each one of the states
had to occur.
812
00:48:02,520 --> 00:48:05,560
The same kind of proof
is the proof here.
813
00:48:05,560 --> 00:48:07,610
It had the same kind of proof
when we were talking about
814
00:48:07,610 --> 00:48:09,440
renewal theory.
815
00:48:09,440 --> 00:48:14,670
What is going on is given any
state, given the state the
816
00:48:14,670 --> 00:48:19,100
transition has to occur within
finite time, because there
817
00:48:19,100 --> 00:48:21,620
some exponential holding
time there.
818
00:48:21,620 --> 00:48:26,810
So the expected amount of time
for the next transition is 1
819
00:48:26,810 --> 00:48:28,260
over nu sub i.
820
00:48:28,260 --> 00:48:32,920
And that's finite for every
i in the chain.
821
00:48:32,920 --> 00:48:36,540
And therefore, as you go from
one state to another, as the
822
00:48:36,540 --> 00:48:41,430
frog those jumping from one lily
pad to another, and each
823
00:48:41,430 --> 00:48:44,700
lily pad that it jumps on
there's some expected time
824
00:48:44,700 --> 00:48:48,410
before it moves, and therefore
assuming that it keeps moving
825
00:48:48,410 --> 00:48:53,500
forever and doesn't die, which
is what we assume with these
826
00:48:53,500 --> 00:48:57,360
Markov chains, it will
eventually go through an
827
00:48:57,360 --> 00:49:00,220
infinite number of steps.
828
00:49:00,220 --> 00:49:02,730
And the proof of that
is in the text.
829
00:49:02,730 --> 00:49:06,190
But it's exactly the same proof
as you've seen several
830
00:49:06,190 --> 00:49:09,530
times before for renewal process
in countable state
831
00:49:09,530 --> 00:49:11,930
Markov chains.
832
00:49:11,930 --> 00:49:19,160
Next theorem is to say let M sub
ij of t be the number of
833
00:49:19,160 --> 00:49:23,930
transitions to j, starting
in state i.
834
00:49:23,930 --> 00:49:26,680
We can't get rid of the
starting state.
835
00:49:26,680 --> 00:49:28,960
Somehow we have to
keep it in there.
836
00:49:28,960 --> 00:49:33,010
We have some confidence that
it's not important, that it
837
00:49:33,010 --> 00:49:33,850
shouldn't be there.
838
00:49:33,850 --> 00:49:36,840
And we're going to see it
disappear very shortly.
839
00:49:36,840 --> 00:49:39,970
But we have to keep it there
for the time being.
840
00:49:39,970 --> 00:49:45,800
So if the embedded chain is
recurrent, then n sub ij of t
841
00:49:45,800 --> 00:49:49,276
is a delayed renewal process.
842
00:49:49,276 --> 00:49:51,180
And we sort of know that.
843
00:49:51,180 --> 00:49:53,890
Essentially, transitions
keep occurring.
844
00:49:53,890 --> 00:49:57,800
So renewals in the state
j must keep occurring.
845
00:49:57,800 --> 00:50:01,700
And therefore, any time you go
to state j, the amount of time
846
00:50:01,700 --> 00:50:07,080
that it takes until you get
there again, is finite.
847
00:50:07,080 --> 00:50:09,970
We're not selling it to expect
at time is finite.
848
00:50:09,970 --> 00:50:11,520
Expect time might be infinite.
849
00:50:11,520 --> 00:50:13,850
We'll see lots of cases
where it is.
850
00:50:13,850 --> 00:50:16,210
But you've got to get
there eventually.
851
00:50:16,210 --> 00:50:22,190
That's the same kind of thing we
saw for renewal theory when
852
00:50:22,190 --> 00:50:26,610
we had renewals and the things
that could happen eventually
853
00:50:26,610 --> 00:50:28,910
they did happen.
854
00:50:28,910 --> 00:50:32,310
I don't know whether any of
you are old enough to have
855
00:50:32,310 --> 00:50:34,620
heard about Murphy's Law.
856
00:50:34,620 --> 00:50:38,240
Murphy was an Irish
American to whom
857
00:50:38,240 --> 00:50:40,080
awful things kept happening.
858
00:50:40,080 --> 00:50:43,020
And Murphy's Law says that
if something awful
859
00:50:43,020 --> 00:50:45,160
can happen, it will.
860
00:50:45,160 --> 00:50:49,080
This says if this can happen,
eventually it will happen.
861
00:50:49,080 --> 00:50:50,740
It doesn't say it will
happen immediately.
862
00:50:50,740 --> 00:50:53,850
But it says it will
happen eventually.
863
00:50:53,850 --> 00:50:57,340
You can think of this as
Murphy's Law, if you want to,
864
00:50:57,340 --> 00:50:58,590
if you're familiar with that.
865
00:51:01,820 --> 00:51:04,710
So we want to talk about steady
state for irreducible
866
00:51:04,710 --> 00:51:06,720
Markov processes.
867
00:51:06,720 --> 00:51:12,300
Now, let p sub j of i be the
time average fraction of time
868
00:51:12,300 --> 00:51:15,760
in state j for the delayed
renewal process.
869
00:51:15,760 --> 00:51:19,910
Remember we talked about p sub
j in terms of these sample
870
00:51:19,910 --> 00:51:21,050
time Markov change.
871
00:51:21,050 --> 00:51:23,730
And we talked about them a
little bit in terms of
872
00:51:23,730 --> 00:51:27,570
imagining how long you would
stay in state j if you were in
873
00:51:27,570 --> 00:51:29,890
some kind of steady state.
874
00:51:29,890 --> 00:51:32,780
Here we want to talk
about p sub j of i.
875
00:51:35,340 --> 00:51:37,680
In terms of strong law of
large numbers kinds of
876
00:51:37,680 --> 00:51:41,010
results, we want to look at the
sample path average and
877
00:51:41,010 --> 00:51:44,890
see the convergence with
probability one.
878
00:51:44,890 --> 00:51:50,330
OK, so p sub j of i is a time
average fraction of time in
879
00:51:50,330 --> 00:51:54,470
state j for the delayed
renewal process.
880
00:51:54,470 --> 00:51:57,750
Remember we said that delay
renewal processes were really
881
00:51:57,750 --> 00:51:59,820
the same as renewal processes.
882
00:51:59,820 --> 00:52:02,850
You just had this first renewal,
which really didn't
883
00:52:02,850 --> 00:52:05,490
make any difference.
884
00:52:05,490 --> 00:52:09,400
And so p sub j of i is going
to be the limit as t
885
00:52:09,400 --> 00:52:14,870
approaches infinity of the
reward that we pick up forever
886
00:52:14,870 --> 00:52:16,690
of being in state j.
887
00:52:16,690 --> 00:52:19,900
You get one unit of reward
whenever you're in state j, 0
888
00:52:19,900 --> 00:52:21,850
units when you're
anywhere else.
889
00:52:21,850 --> 00:52:25,520
So this is the time average
fraction of time
890
00:52:25,520 --> 00:52:26,890
you're in state j.
891
00:52:26,890 --> 00:52:29,870
This is divided by t.
892
00:52:29,870 --> 00:52:31,940
And the assumption is
you start in time i.
893
00:52:31,940 --> 00:52:34,200
So that affects this
a little bit.
894
00:52:34,200 --> 00:52:38,840
The picture here says whenever
you go to state j, you're
895
00:52:38,840 --> 00:52:47,100
going to stay in state j for
some holding time U sub n.
896
00:52:47,100 --> 00:52:51,562
Then you go back to 0 reward
until the next time you went
897
00:52:51,562 --> 00:52:52,980
to state j.
898
00:52:52,980 --> 00:52:55,260
Then you jump up
to reward of 1.
899
00:52:55,260 --> 00:52:58,000
You stay there for your holding
time until you get
900
00:52:58,000 --> 00:53:02,600
into some other state, a
and that keeps going
901
00:53:02,600 --> 00:53:04,260
on forever and ever.
902
00:53:04,260 --> 00:53:07,090
What does the delayed renewal
reward theorem say?
903
00:53:16,600 --> 00:53:22,560
It says that the expected reward
over time is going to
904
00:53:22,560 --> 00:53:27,040
be the expect to reward in one
renewal divided by the
905
00:53:27,040 --> 00:53:29,450
expected length of
the renewal path.
906
00:53:29,450 --> 00:53:33,500
It says it's going to be
expected value of U sub n
907
00:53:33,500 --> 00:53:39,130
divided by the expected time
that you stay in state j.
908
00:53:39,130 --> 00:53:45,490
So it's 1 over nu sub j times
the expected time
909
00:53:45,490 --> 00:53:46,890
you're in a state j.
910
00:53:49,590 --> 00:53:52,980
That's a really neat result
that connects this steady
911
00:53:52,980 --> 00:53:54,660
state probability.
912
00:53:54,660 --> 00:53:59,720
Excuse my impolite computer.
913
00:53:59,720 --> 00:54:04,090
This relates to fraction of time
you're in state i to the
914
00:54:04,090 --> 00:54:06,570
expected delay in state j.
915
00:54:06,570 --> 00:54:09,970
It's one of those maddening
things where you say that's
916
00:54:09,970 --> 00:54:13,290
great, but I don't know how to
find either of those things.
917
00:54:13,290 --> 00:54:14,590
So we go on.
918
00:54:14,590 --> 00:54:17,420
We will find them.
919
00:54:17,420 --> 00:54:21,180
And what we will find
is W sub j.
920
00:54:21,180 --> 00:54:26,780
If we can find W sub j, we'll
also know p sub j.
921
00:54:26,780 --> 00:54:30,510
M sub ij of t is delayed
renewal process.
922
00:54:30,510 --> 00:54:34,590
The strong law for renewal says
the limit as t approaches
923
00:54:34,590 --> 00:54:41,950
infinity of Mi j of t over t is
1 over this waiting time.
924
00:54:41,950 --> 00:54:44,130
This is the number of
renewals you have
925
00:54:44,130 --> 00:54:46,220
as t goes to infinity.
926
00:54:46,220 --> 00:54:49,520
This is equal to 1 over the
expected length of that
927
00:54:49,520 --> 00:54:50,940
renewal period.
928
00:54:50,940 --> 00:54:51,840
Great.
929
00:54:51,840 --> 00:54:55,780
So we take the limit as
t goes to infinity.
930
00:54:55,780 --> 00:55:01,260
Of mij of t over mi of t.
931
00:55:01,260 --> 00:55:06,290
This is the number of
transitions overall up to time
932
00:55:06,290 --> 00:55:08,340
t starting in state i.
933
00:55:08,340 --> 00:55:10,960
This is the number of those
transitions which
934
00:55:10,960 --> 00:55:13,130
go into state j.
935
00:55:13,130 --> 00:55:15,550
How do I talk about that?
936
00:55:15,550 --> 00:55:23,410
Well, this quantity up here is
the number of transitions into
937
00:55:23,410 --> 00:55:30,520
state j over the number of
transition that take place.
938
00:55:30,520 --> 00:55:35,150
n sub ij of t is the number of
transitions out of total
939
00:55:35,150 --> 00:55:37,210
transitions.
940
00:55:37,210 --> 00:55:40,550
mi of t is the total number
of transitions.
941
00:55:40,550 --> 00:55:42,840
mi of t goes to infinity.
942
00:55:42,840 --> 00:55:49,480
So this limit here goes to the
limit of n sub ij of n over n.
943
00:55:49,480 --> 00:55:53,210
Remember, we even proved this
very carefully in class for
944
00:55:53,210 --> 00:55:56,170
the last application of it.
945
00:55:56,170 --> 00:55:59,350
This is something we've done
many times already in
946
00:55:59,350 --> 00:56:02,110
different situations.
947
00:56:02,110 --> 00:56:05,430
And this particular time we're
doing it, we won't go through
948
00:56:05,430 --> 00:56:06,810
any fuss about it.
949
00:56:06,810 --> 00:56:11,170
It's just a limit of n
sub ij of n over n.
950
00:56:11,170 --> 00:56:12,090
And what is that?
951
00:56:12,090 --> 00:56:16,330
That's the fraction, long term
fraction of transitions that
952
00:56:16,330 --> 00:56:18,120
go into state j.
953
00:56:18,120 --> 00:56:27,490
We know that for accountable
state Markov chain, which is
954
00:56:27,490 --> 00:56:30,480
recurrent, which is positive
recurrent, this is
955
00:56:30,480 --> 00:56:32,850
equal to pi sub j.
956
00:56:32,850 --> 00:56:35,720
So we have that as equal
to pi sub j.
957
00:56:35,720 --> 00:56:38,760
We then have one over w sub j,
that's what we're trying to
958
00:56:38,760 --> 00:56:44,140
find is equal to the limit of
mij, of t over t, which is the
959
00:56:44,140 --> 00:56:49,530
limit of mij of t over mi of
t times mi of t over t.
960
00:56:49,530 --> 00:56:52,200
We break this into a limit of
two terms, which we've done
961
00:56:52,200 --> 00:56:54,130
very carefully before.
962
00:56:54,130 --> 00:56:58,010
And this limit here
is pi sub j.
963
00:56:58,010 --> 00:57:02,710
This limit here is the limit
of m sub i of t.
964
00:57:02,710 --> 00:57:05,904
And we have already shown that
the limit of n sub i
965
00:57:05,904 --> 00:57:08,260
of t is equal to--
966
00:57:17,700 --> 00:57:20,320
Somewhere we showed that.
967
00:57:20,320 --> 00:57:21,570
Yeah.
968
00:57:25,460 --> 00:57:30,230
1 over w sub j is equal
to pj times new sub j.
969
00:57:30,230 --> 00:57:34,960
That's what we proved right
down here. p sub j of i is
970
00:57:34,960 --> 00:57:38,000
equal to 1 over new sub
j times w sub j.
971
00:57:40,750 --> 00:57:43,240
Except now we're just calling
it p sub j because we've
972
00:57:43,240 --> 00:57:44,170
already seen it.
973
00:57:44,170 --> 00:57:46,950
Doesn't depend on i at all.
974
00:57:46,950 --> 00:57:55,190
So one over w sub j is equal
to p sub j times new sub j.
975
00:57:55,190 --> 00:57:59,070
This says if we know what
w sub j is, we know
976
00:57:59,070 --> 00:58:00,880
what p sub j is.
977
00:58:00,880 --> 00:58:04,520
So it looks like we're not
making any progress.
978
00:58:04,520 --> 00:58:07,040
So what's going on?
979
00:58:07,040 --> 00:58:10,710
OK, well let's look at this
a little more carefully.
980
00:58:10,710 --> 00:58:14,770
This quantity here is a
function only of i.
981
00:58:19,350 --> 00:58:26,540
1 over w sub j, over p sub
j and new sub j is a
982
00:58:26,540 --> 00:58:27,850
function only of j.
983
00:58:27,850 --> 00:58:30,100
Everything else in this
equation is a
984
00:58:30,100 --> 00:58:32,530
function only of j.
985
00:58:32,530 --> 00:58:36,890
That says that this quantity
here is independent of i, and
986
00:58:36,890 --> 00:58:39,450
it's also independent of j.
987
00:58:39,450 --> 00:58:40,820
And what is it?
988
00:58:40,820 --> 00:58:44,310
It's the rate at which
transitions occur.
989
00:58:44,310 --> 00:58:46,250
Overall transitions.
990
00:58:46,250 --> 00:58:51,320
If you're in steady state, this
says that looking at any
991
00:58:51,320 --> 00:58:55,910
state j, the total number of
transitions that occur is
992
00:58:55,910 --> 00:59:00,310
equal to, well, I do it
on the next page.
993
00:59:00,310 --> 00:59:03,550
So let's goes there.
994
00:59:03,550 --> 00:59:07,400
It says that p sub j is equal
to 1 over new sub j.
995
00:59:07,400 --> 00:59:12,580
w sub j is equal to pi j over
new j times this limit here.
996
00:59:15,820 --> 00:59:17,070
OK.
997
00:59:20,060 --> 00:59:27,990
So in fact, we now have a way
of finding p sub j for all j
998
00:59:27,990 --> 00:59:30,870
if we can just find this
one number here.
999
00:59:30,870 --> 00:59:34,460
This is independent of i, so
this is just one number, which
1000
00:59:34,460 --> 00:59:37,920
we now know is something that
approaches some limit with
1001
00:59:37,920 --> 00:59:40,110
probability one.
1002
00:59:40,110 --> 00:59:44,860
So we only have one unknown
instead of this countably
1003
00:59:44,860 --> 00:59:46,110
infinite number of unknowns.
1004
00:59:50,410 --> 00:59:53,500
Seems like we haven't really
made any progress, because
1005
00:59:53,500 --> 00:59:56,550
before, what we did was to
normalize these p sub js.
1006
00:59:56,550 --> 01:00:00,800
We said they have to add up to
1, and let's normalize them.
1007
01:00:00,800 --> 01:00:03,850
And here we're doing
the same thing.
1008
01:00:03,850 --> 01:00:09,320
We're saying p sub j is equal
to pi sub j over new sub j
1009
01:00:09,320 --> 01:00:11,860
with this normalization
factor in.
1010
01:00:11,860 --> 01:00:15,970
And we're saying here that this
normalization factor has
1011
01:00:15,970 --> 01:00:22,370
to be equal to 1 over the
sum of pi k over new k.
1012
01:00:22,370 --> 01:00:30,290
In other words, if the p sub
j's add to 1, then the only
1013
01:00:30,290 --> 01:00:34,510
value this didn't have is
1 over the sum of pi k
1014
01:00:34,510 --> 01:00:35,760
times new sub k.
1015
01:00:38,020 --> 01:00:45,290
Unfortunately, there are
examples where the sum of pi
1016
01:00:45,290 --> 01:00:49,260
sub k over new sub k is
equal to infinity.
1017
01:00:49,260 --> 01:00:51,200
That's very awkward.
1018
01:00:51,200 --> 01:00:53,470
I'm going to give you an example
of that in just a
1019
01:00:53,470 --> 01:00:56,330
minute, and you'll see
what's going on.
1020
01:00:56,330 --> 01:01:00,900
But if pi sub k over new sub k
is equal to infinity, and the
1021
01:01:00,900 --> 01:01:05,810
theorem is true, it says that
the limit of mi of t over t is
1022
01:01:05,810 --> 01:01:08,275
equal to 1 over infinity,
which says it's zero.
1023
01:01:11,430 --> 01:01:16,550
So what this is telling us is
what we sort of visualize
1024
01:01:16,550 --> 01:01:21,170
before, but we couldn't
quite visualize it.
1025
01:01:21,170 --> 01:01:25,690
It was saying that either these
probabilities add up to
1026
01:01:25,690 --> 01:01:31,550
1, or if they don't add up to 1,
this quantity here doesn't
1027
01:01:31,550 --> 01:01:32,570
make any sense.
1028
01:01:32,570 --> 01:01:34,500
This is not approaching
a limit.
1029
01:01:34,500 --> 01:01:36,990
The only way this can approach,
well, this can
1030
01:01:36,990 --> 01:01:41,500
approach a limit where
the limit is 0.
1031
01:01:41,500 --> 01:01:43,870
Because this theorem holds
whether it approaches the
1032
01:01:43,870 --> 01:01:45,080
limit or not.
1033
01:01:45,080 --> 01:01:49,390
So it is possible for
this limit to the 0.
1034
01:01:49,390 --> 01:01:53,950
In this case, these p
sub js are all 0.
1035
01:01:53,950 --> 01:01:55,470
And we've seen this kind
of thing before.
1036
01:01:55,470 --> 01:01:58,380
We've seen that on a Markov
chain, all the steady state
1037
01:01:58,380 --> 01:02:02,380
probabilities can be equal to
zero, and that's a sign that
1038
01:02:02,380 --> 01:02:06,010
we're either in a transient
condition, or in a null
1039
01:02:06,010 --> 01:02:08,110
recurrent position.
1040
01:02:08,110 --> 01:02:13,970
Namely, the state just wonders
away, and over the long term,
1041
01:02:13,970 --> 01:02:17,420
each state has 0 probability.
1042
01:02:17,420 --> 01:02:20,180
And that looks like the same
kind of thing which is
1043
01:02:20,180 --> 01:02:23,250
happening here.
1044
01:02:23,250 --> 01:02:25,660
This looks trivial.
1045
01:02:25,660 --> 01:02:31,140
There's a fairly long proof
in the notes doing this.
1046
01:02:31,140 --> 01:02:35,710
The only way I can find to do
this is to truncate the chain,
1047
01:02:35,710 --> 01:02:39,780
and then go to the limit as
the number of states gets
1048
01:02:39,780 --> 01:02:41,210
larger and larger.
1049
01:02:41,210 --> 01:02:45,630
And when you do that, this
theorem becomes clear.
1050
01:02:45,630 --> 01:02:48,800
OK, let's look at an example
where the sum of pi k over new
1051
01:02:48,800 --> 01:02:53,190
k is equal to infinity, and
see what's going on.
1052
01:02:58,020 --> 01:03:01,790
Visualize something
like an mm1 queue.
1053
01:03:01,790 --> 01:03:06,000
We have arrivals, and
we have a server.
1054
01:03:06,000 --> 01:03:10,000
But as soon as the queue starts
building up, the server
1055
01:03:10,000 --> 01:03:12,300
starts to get very rattled.
1056
01:03:12,300 --> 01:03:15,010
And as the server gets more and
more rattled, it starts to
1057
01:03:15,010 --> 01:03:17,850
make more and more mistakes.
1058
01:03:17,850 --> 01:03:21,880
And as the queue builds up also,
customers come in and
1059
01:03:21,880 --> 01:03:25,110
look at the queue, and say I'll
come back tomorrow when
1060
01:03:25,110 --> 01:03:26,780
the queue isn't so long.
1061
01:03:26,780 --> 01:03:31,720
So we both have this situation
where as the queue is building
1062
01:03:31,720 --> 01:03:35,890
up, service is getting slower
and the arrival rate is
1063
01:03:35,890 --> 01:03:37,400
getting slower.
1064
01:03:37,400 --> 01:03:40,140
And we're assuming here to make
a nice example that the
1065
01:03:40,140 --> 01:03:44,040
two of them build up in
exactly the same way.
1066
01:03:44,040 --> 01:03:45,790
So that's what's
happening here.
1067
01:03:45,790 --> 01:03:50,295
The service rate when there's
one customer being served is 2
1068
01:03:50,295 --> 01:03:52,280
to the minus 1.
1069
01:03:52,280 --> 01:03:54,850
The service right rate when
there are two customers in the
1070
01:03:54,850 --> 01:03:57,680
system is 2 to the minus 2.
1071
01:03:57,680 --> 01:03:59,830
The service rate when there
are three customers in the
1072
01:03:59,830 --> 01:04:03,550
system is 2 to the minus 3.
1073
01:04:03,550 --> 01:04:09,340
For each of these states, we
still have these transition
1074
01:04:09,340 --> 01:04:11,810
probabilities for the
embedded chain.
1075
01:04:11,810 --> 01:04:15,350
And the embedded chain, the only
way to get from here to
1076
01:04:15,350 --> 01:04:18,350
here is with probability 1,
because that's the only
1077
01:04:18,350 --> 01:04:21,370
transition possible here.
1078
01:04:21,370 --> 01:04:25,250
We assume that from state 1,
you go to states 0 with
1079
01:04:25,250 --> 01:04:27,280
probability 0.6.
1080
01:04:27,280 --> 01:04:30,965
You go to state two with
probability 0.4.
1081
01:04:30,965 --> 01:04:36,330
With state 1, from state 2,
you go up with probability
1082
01:04:36,330 --> 01:04:40,530
0.4, you go down with
probability 0.6.
1083
01:04:40,530 --> 01:04:43,780
The embedded chain
looks great.
1084
01:04:43,780 --> 01:04:45,040
There's nothing wrong
with that.
1085
01:04:45,040 --> 01:04:51,510
That's a perfectly stable mm1
queue type of situation.
1086
01:04:51,510 --> 01:04:56,750
It's these damned holding
times which become very
1087
01:04:56,750 --> 01:04:59,480
disturbing.
1088
01:04:59,480 --> 01:05:04,600
Because if you look at pi sub
j, which is supposed to be 1
1089
01:05:04,600 --> 01:05:06,260
minus rho times rho to the j.
1090
01:05:06,260 --> 01:05:09,510
Rho is 2/3, it's
lambda over mu.
1091
01:05:09,510 --> 01:05:11,590
So it's lambda over lambda
plus mu over rho
1092
01:05:11,590 --> 01:05:14,124
plus lambda plus mu.
1093
01:05:14,124 --> 01:05:19,440
It's 0.4 divided by
0.6, which is 2/3.
1094
01:05:19,440 --> 01:05:24,230
If we look at pi sub j over new
sub j, it's equal to 2 to
1095
01:05:24,230 --> 01:05:29,080
the j times 1 minus rho,
times rho to the j.
1096
01:05:29,080 --> 01:05:34,810
It's 1 minus rho times
4/3 to the j.
1097
01:05:34,810 --> 01:05:40,440
This quantity gets bigger and
bigger as j increases.
1098
01:05:40,440 --> 01:05:43,550
So when you try to
sum pi i over new
1099
01:05:43,550 --> 01:05:46,910
sub j, you get infinity.
1100
01:05:46,910 --> 01:05:50,070
So what's going on?
1101
01:05:50,070 --> 01:05:52,670
None of the states here have
an infinite holding time
1102
01:05:52,670 --> 01:05:54,570
associated with them.
1103
01:05:54,570 --> 01:05:58,490
It's just that the expected
holding time
1104
01:05:58,490 --> 01:06:01,390
is going to be infinite.
1105
01:06:01,390 --> 01:06:04,480
The expected number of
transitions over a long period
1106
01:06:04,480 --> 01:06:14,050
of time, according to this
equation here, expected
1107
01:06:14,050 --> 01:06:18,040
transitions per unit
time is going to 0.
1108
01:06:18,040 --> 01:06:24,160
As time goes on, you keep
floating back to state 0, as
1109
01:06:24,160 --> 01:06:26,820
far as the embedded chain
is concerned.
1110
01:06:26,820 --> 01:06:30,680
But you're eventually going to
a steady state distribution,
1111
01:06:30,680 --> 01:06:33,650
which is laid out over
all the states.
1112
01:06:33,650 --> 01:06:40,310
That steady state distribution
looks very nice.
1113
01:06:40,310 --> 01:06:42,320
That's 1 minus rho times
rho to the j.
1114
01:06:42,320 --> 01:06:48,030
Rho is 2/3, so the probability
of being in state j is going
1115
01:06:48,030 --> 01:06:51,680
down exponentially
as j gets big.
1116
01:06:51,680 --> 01:06:54,570
But the time that you spend
there is going up
1117
01:06:54,570 --> 01:06:57,510
exponentially even faster.
1118
01:06:57,510 --> 01:07:01,740
And therefore, when we sum all
of these things up, the
1119
01:07:01,740 --> 01:07:07,330
overall expected rate is equal
to zero, because the sum of
1120
01:07:07,330 --> 01:07:12,220
the pi j over new j is
equal to infinity.
1121
01:07:12,220 --> 01:07:12,600
OK.
1122
01:07:12,600 --> 01:07:14,510
So this is one of the awful
things that are going to
1123
01:07:14,510 --> 01:07:17,590
happen with Markov processes.
1124
01:07:17,590 --> 01:07:19,620
We still have an
embedded chain.
1125
01:07:19,620 --> 01:07:21,960
The embedded chain
can be stable.
1126
01:07:21,960 --> 01:07:25,150
It can have a steady state, but
we've already found that
1127
01:07:25,150 --> 01:07:30,070
the fraction of time in a state
is not equal to the
1128
01:07:30,070 --> 01:07:33,390
fraction of transitions that go
into that state. pi sub j
1129
01:07:33,390 --> 01:07:36,560
is not in general equal
to p sub j.
1130
01:07:36,560 --> 01:07:40,740
And for this example here with
the rattled server and the
1131
01:07:40,740 --> 01:07:47,400
discouraged customers, the
amount of time that it takes,
1132
01:07:47,400 --> 01:07:51,285
the expected amount of time that
it takes for customers to
1133
01:07:51,285 --> 01:07:54,770
get served is going to zero.
1134
01:07:54,770 --> 01:07:59,280
Even though the queue
was saying stable.
1135
01:07:59,280 --> 01:08:01,380
Does mathematics lie?
1136
01:08:01,380 --> 01:08:01,930
I don't know.
1137
01:08:01,930 --> 01:08:04,110
I don't think so.
1138
01:08:04,110 --> 01:08:07,320
I've looked at this often enough
with great frustration,
1139
01:08:07,320 --> 01:08:10,230
but I believe it
at this point.
1140
01:08:10,230 --> 01:08:13,660
If you don't believe it, take
this chain here and truncate
1141
01:08:13,660 --> 01:08:17,840
it, and solve the problem for
the truncated chain, and then
1142
01:08:17,840 --> 01:08:20,220
look at what happens
as you start adding
1143
01:08:20,220 --> 01:08:22,430
states on one by one.
1144
01:08:22,430 --> 01:08:26,700
What happens as you start adding
states on one by one is
1145
01:08:26,700 --> 01:08:31,460
that the rate at which this
Markov process is serving
1146
01:08:31,460 --> 01:08:34,229
things is going to zero.
1147
01:08:34,229 --> 01:08:38,120
So the dilemma as the number
of states becomes infinite,
1148
01:08:38,120 --> 01:08:43,979
the rate at which things
happen is equal to 0.
1149
01:08:43,979 --> 01:08:45,880
It's not pleasant.
1150
01:08:45,880 --> 01:08:47,750
It's not intuitive.
1151
01:08:47,750 --> 01:08:50,210
But that's what it is.
1152
01:08:50,210 --> 01:08:51,460
And that can happen.
1153
01:08:56,310 --> 01:09:00,020
Again, let's go back to the
typical case of a positive
1154
01:09:00,020 --> 01:09:07,040
recurrent embedded chain, where
this funny sum here is
1155
01:09:07,040 --> 01:09:09,330
less than infinity.
1156
01:09:09,330 --> 01:09:12,710
If the sum here is less than
infinity, then you can
1157
01:09:12,710 --> 01:09:18,479
certainly express p sub j as
pi sub j over new sub j
1158
01:09:18,479 --> 01:09:21,890
divided by the sum over k of
p sub k over new sub k.
1159
01:09:21,890 --> 01:09:23,620
Why can I do that?
1160
01:09:23,620 --> 01:09:25,379
Because that's what
the formula says.
1161
01:09:28,750 --> 01:09:32,439
I don't like to live with
formulas, but sometimes things
1162
01:09:32,439 --> 01:09:36,520
get so dirty, the mathematics
play such awful tricks with
1163
01:09:36,520 --> 01:09:40,180
you that you have to live with
the formulas, and just try to
1164
01:09:40,180 --> 01:09:41,689
explain what they're doing.
1165
01:09:41,689 --> 01:09:44,529
p sub j is equal to this.
1166
01:09:44,529 --> 01:09:52,340
Limit of the service rate, if
this quantity is non infinite,
1167
01:09:52,340 --> 01:09:55,570
then things get churned
out of this queueing
1168
01:09:55,570 --> 01:09:58,820
system at some rate.
1169
01:09:58,820 --> 01:10:03,710
And the p sub js can
be solved for.
1170
01:10:03,710 --> 01:10:06,620
And this is the way
to solve for them.
1171
01:10:06,620 --> 01:10:08,790
OK, so that's pretty neat.
1172
01:10:08,790 --> 01:10:12,120
It says that if you can solve
the embedded chain, then you
1173
01:10:12,120 --> 01:10:14,810
have a nice formula for finding
the steady state
1174
01:10:14,810 --> 01:10:16,120
probabilities.
1175
01:10:16,120 --> 01:10:19,430
And you have a theorem which
says that so long as this
1176
01:10:19,430 --> 01:10:24,380
quantity is less than infinity
with probability one, the
1177
01:10:24,380 --> 01:10:27,420
fraction of time that you
stay in state j is
1178
01:10:27,420 --> 01:10:29,660
equal to this quantity.
1179
01:10:29,660 --> 01:10:31,710
Well, that's not good enough.
1180
01:10:31,710 --> 01:10:35,250
Because for the mm1 queue, we
saw that it was really a pain
1181
01:10:35,250 --> 01:10:39,270
in the neck to solve for the
steady state equations for the
1182
01:10:39,270 --> 01:10:41,400
embedded chain.
1183
01:10:41,400 --> 01:10:45,520
Things looked simpler for
the process itself.
1184
01:10:45,520 --> 01:10:51,010
So let's see if we can get those
equations back also.
1185
01:10:51,010 --> 01:10:55,710
What we would like to do is
to solve for the p sub j's
1186
01:10:55,710 --> 01:10:59,090
directly by using the steady
state embedded equation.
1187
01:10:59,090 --> 01:11:03,750
Embedded equations say that pi
sub j is equal to the sum over
1188
01:11:03,750 --> 01:11:07,330
i, pi sub i times p sub ij.
1189
01:11:07,330 --> 01:11:09,490
The probability of going into
a state is equal to the
1190
01:11:09,490 --> 01:11:11,000
probability of going
out of a state.
1191
01:11:15,970 --> 01:11:23,720
If I use this formula here, pi
sub j over new sub j divided
1192
01:11:23,720 --> 01:11:27,330
by some constant is
what p sub j is.
1193
01:11:27,330 --> 01:11:32,435
So pi sub j is equal to
p sub j times new sub
1194
01:11:32,435 --> 01:11:34,410
j times that constant.
1195
01:11:34,410 --> 01:11:41,290
Here we have the p sub
j times the new sub j
1196
01:11:41,290 --> 01:11:44,060
divided by that constant.
1197
01:11:44,060 --> 01:11:50,490
And that's equal to this sum
here over all i divided by the
1198
01:11:50,490 --> 01:11:51,280
same constant.
1199
01:11:51,280 --> 01:11:53,770
So the constant cancels out.
1200
01:11:53,770 --> 01:11:57,870
Namely, we left out that term
here, but that term is on this
1201
01:11:57,870 --> 01:11:59,940
side, and it's on this side.
1202
01:11:59,940 --> 01:12:03,530
So we have this equation here.
1203
01:12:03,530 --> 01:12:07,230
If you remember, I can't
remember, but you being
1204
01:12:07,230 --> 01:12:09,450
younger can perhaps remember.
1205
01:12:09,450 --> 01:12:12,810
But when we were dealing with
a sample time approximation,
1206
01:12:12,810 --> 01:12:16,600
if you leave the deltas out,
this is exactly the equation
1207
01:12:16,600 --> 01:12:18,440
that you got.
1208
01:12:18,440 --> 01:12:20,600
It's a nice equation.
1209
01:12:20,600 --> 01:12:25,910
It says that the rate at which
transitions occur out of state
1210
01:12:25,910 --> 01:12:29,300
i, the rate at which transitions
occur out of state
1211
01:12:29,300 --> 01:12:35,490
i, out of state j, excuse me,
is p sub j times new sub j.
1212
01:12:35,490 --> 01:12:40,330
Here's the holding time, and
there's also the probability
1213
01:12:40,330 --> 01:12:42,230
of being there.
1214
01:12:42,230 --> 01:12:43,380
Excuse me.
1215
01:12:43,380 --> 01:12:45,200
Let's be more clear
about that.
1216
01:12:45,200 --> 01:12:50,040
I'm not talking about given
you're in state j, the rate at
1217
01:12:50,040 --> 01:12:51,870
which you get out of state j.
1218
01:12:51,870 --> 01:12:56,700
I'm talking about the rate at
which you're in state j and
1219
01:12:56,700 --> 01:12:58,520
you're getting out of state j.
1220
01:12:58,520 --> 01:13:00,390
If you could make sense
out of that.
1221
01:13:00,390 --> 01:13:02,100
That's what this is.
1222
01:13:02,100 --> 01:13:06,090
This quantity here is the
overall rate at which you're
1223
01:13:06,090 --> 01:13:08,100
entering state j.
1224
01:13:08,100 --> 01:13:11,730
So these equations sort of
have the same intuitive
1225
01:13:11,730 --> 01:13:14,390
meaning as these equations
here do.
1226
01:13:19,450 --> 01:13:28,140
And then if you solve this
equation in the same way,
1227
01:13:28,140 --> 01:13:30,370
what's that doing?
1228
01:13:30,370 --> 01:13:31,610
Oh.
1229
01:13:31,610 --> 01:13:34,550
This gets you pi sub j in
terms of the p sub j's.
1230
01:13:34,550 --> 01:13:36,060
So there's a very nice symmetry
1231
01:13:36,060 --> 01:13:38,880
about this set of equations.
1232
01:13:38,880 --> 01:13:43,990
The p sub j's are found for the
pi sub j's in this way.
1233
01:13:43,990 --> 01:13:47,340
The pi sub j's are found from
the p sub j's by this same
1234
01:13:47,340 --> 01:13:49,010
sort of expression.
1235
01:13:49,010 --> 01:13:52,430
The theorem then says if the
embedded chain is positive
1236
01:13:52,430 --> 01:13:57,250
recurrent, and the sum of pi
i over new i is less than
1237
01:13:57,250 --> 01:14:01,740
infinity, then this equation
has a unique solution.
1238
01:14:01,740 --> 01:14:05,270
In other words, there is a
solution to the steady state
1239
01:14:05,270 --> 01:14:08,670
process equations.
1240
01:14:08,670 --> 01:14:16,130
And pi sub j and p sub j are
related by this, and by this.
1241
01:14:16,130 --> 01:14:19,740
If you know the pi sub j's, you
can find the p sub j's.
1242
01:14:19,740 --> 01:14:23,940
If you know the p sub j's, you
can find the pi sub j's.
1243
01:14:23,940 --> 01:14:28,180
There's a fudge factor, and the
sum of pi sub i over new
1244
01:14:28,180 --> 01:14:32,330
sub i is equal to sum of
p sub j times new sub j
1245
01:14:32,330 --> 01:14:34,300
to the minus 1.
1246
01:14:34,300 --> 01:14:37,660
This equation just falls out
of looking at this equation
1247
01:14:37,660 --> 01:14:39,670
and this equation.
1248
01:14:39,670 --> 01:14:42,210
I'm not going to do that here,
but if you just fiddle with
1249
01:14:42,210 --> 01:14:45,650
these equations a little bit,
that's what you find.
1250
01:14:49,554 --> 01:14:52,530
I think graduate students
love to push equations.
1251
01:14:52,530 --> 01:14:55,310
And if you push these equations,
you will rapidly
1252
01:14:55,310 --> 01:14:57,110
find that out.
1253
01:14:57,110 --> 01:14:59,680
So there's no point
to doing it here.
1254
01:14:59,680 --> 01:15:00,930
OK.
1255
01:15:02,430 --> 01:15:05,430
You can do the opposite
thing, also.
1256
01:15:05,430 --> 01:15:09,650
If the steady state process
equations are satisfied, and
1257
01:15:09,650 --> 01:15:13,450
the p sub j's are all greater
than zero, and the sum of the
1258
01:15:13,450 --> 01:15:15,910
p sub j's are equal to 1.
1259
01:15:15,910 --> 01:15:20,010
And if these equations are less
than infinity, this is
1260
01:15:20,010 --> 01:15:23,090
just by symmetry with
what we already did.
1261
01:15:23,090 --> 01:15:27,730
Then pi sub j has to be equal
to p sub j times new sub j
1262
01:15:27,730 --> 01:15:30,230
divided by this sum.
1263
01:15:30,230 --> 01:15:32,830
And this gives the steady
state equations for the
1264
01:15:32,830 --> 01:15:34,690
embedded chain.
1265
01:15:34,690 --> 01:15:37,230
And this shows that the embedded
chain has to be
1266
01:15:37,230 --> 01:15:39,980
positive recurrent, and says
that you have to be
1267
01:15:39,980 --> 01:15:41,380
able to go both ways.
1268
01:15:41,380 --> 01:15:44,600
So we already know that if you
can solve the steady state
1269
01:15:44,600 --> 01:15:48,310
equations for the embedded
chain, they have to be unique,
1270
01:15:48,310 --> 01:15:50,610
the probabilities all
have to be positive.
1271
01:15:50,610 --> 01:15:53,580
All those neat things for
accountable state and Markov
1272
01:15:53,580 --> 01:15:55,410
chains hold true.
1273
01:15:55,410 --> 01:15:59,830
This says that if you can solve
that virtually identical
1274
01:15:59,830 --> 01:16:04,590
set of equations for the
process, and you get a
1275
01:16:04,590 --> 01:16:13,210
solution, and the sum here is
finite, then in fact you can
1276
01:16:13,210 --> 01:16:15,310
go back the other way.
1277
01:16:15,310 --> 01:16:17,920
And going back the other way,
you know from what we know
1278
01:16:17,920 --> 01:16:21,530
about embedded chains that
there's a unique solution.
1279
01:16:21,530 --> 01:16:24,460
So there has to be a unique
solution both ways.
1280
01:16:24,460 --> 01:16:28,030
There has to be positive
recurrence both ways.
1281
01:16:28,030 --> 01:16:31,740
So we have a complete
story at this point.
1282
01:16:31,740 --> 01:16:34,160
I mean, you'll have to spend a
little more time putting it
1283
01:16:34,160 --> 01:16:37,580
together, but it really
is there.
1284
01:16:37,580 --> 01:16:42,630
If new sub j is bounded over
j, then the sum over j of p
1285
01:16:42,630 --> 01:16:46,590
sub j, new sub j is less
than infinity.
1286
01:16:46,590 --> 01:16:49,120
Also, the sample time
chain exists
1287
01:16:49,120 --> 01:16:50,750
because this is bounded.
1288
01:16:50,750 --> 01:16:54,230
It has the same steady
state solution as the
1289
01:16:54,230 --> 01:16:57,140
Markov process solution.
1290
01:16:57,140 --> 01:16:59,880
In other words, go back and look
at the solution for the
1291
01:16:59,880 --> 01:17:01,550
sample time chain.
1292
01:17:01,550 --> 01:17:05,270
Drop out the delta, and what
you will get is this set of
1293
01:17:05,270 --> 01:17:08,410
equations here.
1294
01:17:08,410 --> 01:17:11,880
If you have a birth death
process, it's a birth death
1295
01:17:11,880 --> 01:17:17,190
process for both the Markov
process, and also for the
1296
01:17:17,190 --> 01:17:18,190
embedded chain.
1297
01:17:18,190 --> 01:17:20,500
For the embedded chain,
you know that what you
1298
01:17:20,500 --> 01:17:24,040
have to do to get--
1299
01:17:24,040 --> 01:17:27,660
for a birth death chain, you
know an easy way to solve the
1300
01:17:27,660 --> 01:17:33,070
steady state equations for the
chain are to say transitions
1301
01:17:33,070 --> 01:17:35,610
this way equal transitions
this way.
1302
01:17:35,610 --> 01:17:38,910
It's the same for the process.
1303
01:17:38,910 --> 01:17:42,740
The amount of time that you
spend going this way is equal
1304
01:17:42,740 --> 01:17:46,360
to the average amount of time
you spent going this way.
1305
01:17:46,360 --> 01:17:51,900
So it says for a birth death
process, p sub i times q sub
1306
01:17:51,900 --> 01:17:53,800
i, i plus one.
1307
01:17:56,450 --> 01:18:02,870
That's an i there, is equal to
p sub i plus 1 times the
1308
01:18:02,870 --> 01:18:08,110
transition probability from
i plus 1 back to i.
1309
01:18:08,110 --> 01:18:12,190
So this symmetry exists
almost everywhere.
1310
01:18:12,190 --> 01:18:17,170
And then if the sum of p sub
j, new sub j is equal to
1311
01:18:17,170 --> 01:18:20,790
infinity, that's the bad case.
1312
01:18:20,790 --> 01:18:27,670
These equations say that
pi sub j is equal to 0
1313
01:18:27,670 --> 01:18:28,830
everywhere.
1314
01:18:28,830 --> 01:18:32,120
This is sort of the dual
of the situation we
1315
01:18:32,120 --> 01:18:33,480
already looked at.
1316
01:18:33,480 --> 01:18:38,070
In the case we already looked
at of the lazy or rattled
1317
01:18:38,070 --> 01:18:43,050
server and the discouraged
customers, eventually the rate
1318
01:18:43,050 --> 01:18:46,390
at which service occurred
went to 0.
1319
01:18:46,390 --> 01:18:52,230
This situation is a situation
where as far as the embedded
1320
01:18:52,230 --> 01:18:58,490
chain is concerned, it thinks
it's transient.
1321
01:18:58,490 --> 01:19:02,210
As far as the process is
concerned, the process thinks
1322
01:19:02,210 --> 01:19:03,710
it's just fine.
1323
01:19:03,710 --> 01:19:06,750
But then you look at the
process, and you find out that
1324
01:19:06,750 --> 01:19:10,600
what's happening is that an
infinite number of transitions
1325
01:19:10,600 --> 01:19:15,170
are taking place in
a finite time.
1326
01:19:15,170 --> 01:19:19,480
Markov process people call these
irregular processes.
1327
01:19:19,480 --> 01:19:22,306
Here's a picture of it
on the next slide.
1328
01:19:25,360 --> 01:19:29,250
Essentially, the embedded chain
for a hyperactive birth
1329
01:19:29,250 --> 01:19:30,480
death chain.
1330
01:19:30,480 --> 01:19:34,400
As you go to higher states, the
rate at which transitions
1331
01:19:34,400 --> 01:19:38,340
take place gets higher
and higher.
1332
01:19:38,340 --> 01:19:42,420
And for this particular example
here, you can solve
1333
01:19:42,420 --> 01:19:44,460
those process equations.
1334
01:19:44,460 --> 01:19:47,400
The process equations
look just fine.
1335
01:19:47,400 --> 01:19:51,110
This is where you have to be
careful with Markov processes.
1336
01:19:51,110 --> 01:19:55,360
Because you can solve these
Markov process equations, and
1337
01:19:55,360 --> 01:19:59,930
you get things that look fine,
when actually there isn't any
1338
01:19:59,930 --> 01:20:02,700
steady state behavior at all.
1339
01:20:02,700 --> 01:20:05,280
It's even worse than the
rate going to zero.
1340
01:20:05,280 --> 01:20:07,790
When the rate goes to infinity,
you can have an
1341
01:20:07,790 --> 01:20:12,560
infinite number of transitions
taking place in a finite time.
1342
01:20:12,560 --> 01:20:14,310
And then nothing happens.
1343
01:20:14,310 --> 01:20:16,800
Well, I don't know whether
something happens or not.
1344
01:20:16,800 --> 01:20:21,625
I can't visualize what happens
after the thing has exploded.
1345
01:20:24,640 --> 01:20:26,020
Except essentially, at
that point, there's
1346
01:20:26,020 --> 01:20:28,440
nothing nice going on.
1347
01:20:28,440 --> 01:20:32,920
And you have to say that even
though the process equations
1348
01:20:32,920 --> 01:20:36,910
have a steady state solution,
there is no steady state in
1349
01:20:36,910 --> 01:20:40,230
terms of over the long period of
time, this is the fraction
1350
01:20:40,230 --> 01:20:42,410
of time you spend in state j.
1351
01:20:42,410 --> 01:20:45,730
Because that's not the
way it's behaving.
1352
01:20:45,730 --> 01:20:46,980
OK.
1353
01:20:49,250 --> 01:20:51,450
I mean, you can see this
when you look at
1354
01:20:51,450 --> 01:20:53,490
the embedded chain.
1355
01:20:53,490 --> 01:20:56,350
The embedded chain
is transient.
1356
01:20:56,350 --> 01:20:58,590
You're moving up with
probability 0.6.
1357
01:20:58,590 --> 01:21:01,020
You're moving down with
probability 0.4.
1358
01:21:01,020 --> 01:21:03,990
So you keep moving up.
1359
01:21:03,990 --> 01:21:10,440
When you look at the process in
terms of the process with
1360
01:21:10,440 --> 01:21:16,960
the transition rates q sub ij,
the rates going up are always
1361
01:21:16,960 --> 01:21:18,550
less than the rates
going down.
1362
01:21:18,550 --> 01:21:25,520
This is because as you
move up in state,
1363
01:21:25,520 --> 01:21:28,130
you act so much faster.
1364
01:21:28,130 --> 01:21:33,720
The transition rates are higher
in higher states, and
1365
01:21:33,720 --> 01:21:37,750
therefore the transition rates
down are higher, and the
1366
01:21:37,750 --> 01:21:43,900
transition rate down from state
2 to state 1 is bigger
1367
01:21:43,900 --> 01:21:49,110
than the transition right
up from i to i plus 1.
1368
01:21:49,110 --> 01:21:54,000
And it looks stable, as far as
the process is concerned.
1369
01:21:54,000 --> 01:21:58,010
This is one example where you
can't look at the process and
1370
01:21:58,010 --> 01:22:01,190
find out anything from it
without also looking at the
1371
01:22:01,190 --> 01:22:04,570
embedded chain, and looking at
how many transitions you're
1372
01:22:04,570 --> 01:22:07,070
getting per unit time.
1373
01:22:07,070 --> 01:22:10,080
OK, so that's it.
1374
01:22:10,080 --> 01:22:11,330
Thank you.