1
00:00:00,000 --> 00:00:00,040
2
00:00:00,040 --> 00:00:02,460
The following content is
provided under a Creative
3
00:00:02,460 --> 00:00:03,870
Commons license.
4
00:00:03,870 --> 00:00:06,910
Your support will help MIT
OpenCourseWare continue to
5
00:00:06,910 --> 00:00:08,700
offer high-quality, educational
6
00:00:08,700 --> 00:00:10,560
resources for free.
7
00:00:10,560 --> 00:00:13,460
To make a donation or view
additional materials from
8
00:00:13,460 --> 00:00:19,290
hundreds of MIT courses, visit
MIT OpenCourseWare at
9
00:00:19,290 --> 00:00:21,764
ocw.mit.edu.
10
00:00:21,764 --> 00:00:22,590
PROFESSOR: All right.
11
00:00:22,590 --> 00:00:26,070
So today, we're going to start
by taking stock of what we
12
00:00:26,070 --> 00:00:27,670
discussed last time, review the
13
00:00:27,670 --> 00:00:29,810
definition of Markov chains.
14
00:00:29,810 --> 00:00:32,400
And then most of the lecture,
we're going to concentrate on
15
00:00:32,400 --> 00:00:34,520
their steady-state behavior.
16
00:00:34,520 --> 00:00:38,860
Meaning, we're going to look at
what does a Markov chain do
17
00:00:38,860 --> 00:00:40,820
if it has run for a long time.
18
00:00:40,820 --> 00:00:42,860
What can we say about
the probabilities of
19
00:00:42,860 --> 00:00:44,610
the different states?
20
00:00:44,610 --> 00:00:47,390
So what I would like to repeat
is a statement I made last
21
00:00:47,390 --> 00:00:49,730
time that Markov chains
is a very, very
22
00:00:49,730 --> 00:00:51,910
useful class of models.
23
00:00:51,910 --> 00:00:56,820
Pretty much anything in
the real world can be
24
00:00:56,820 --> 00:01:01,380
approximately modeled by a
Markov chain provided that you
25
00:01:01,380 --> 00:01:05,019
set your states in
the proper way.
26
00:01:05,019 --> 00:01:06,890
So we're going to see
some examples.
27
00:01:06,890 --> 00:01:09,310
You're going to see more
examples in the problems
28
00:01:09,310 --> 00:01:11,140
you're going to do in homework
and recitation.
29
00:01:11,140 --> 00:01:13,880
30
00:01:13,880 --> 00:01:15,810
On the other hand, we're
not going to go
31
00:01:15,810 --> 00:01:17,830
too deep into examples.
32
00:01:17,830 --> 00:01:20,310
Rather, we're going to develop
the general methodology.
33
00:01:20,310 --> 00:02:11,440
34
00:02:11,440 --> 00:02:12,690
OK.
35
00:02:12,690 --> 00:02:19,310
36
00:02:19,310 --> 00:02:19,730
All right.
37
00:02:19,730 --> 00:02:22,280
Markov models can be
pretty general.
38
00:02:22,280 --> 00:02:24,670
They can run in continuous
or discrete time.
39
00:02:24,670 --> 00:02:27,540
They can have continuous or
discrete state spaces.
40
00:02:27,540 --> 00:02:30,280
In this class, we're going to
stick just to the case where
41
00:02:30,280 --> 00:02:33,880
the state space is discrete and
time is discrete because
42
00:02:33,880 --> 00:02:35,570
this is the simplest case.
43
00:02:35,570 --> 00:02:38,760
And also, it's the one where
you build your intuition
44
00:02:38,760 --> 00:02:40,830
before going to more
general cases
45
00:02:40,830 --> 00:02:42,840
perhaps in other classes.
46
00:02:42,840 --> 00:02:47,810
So the state is discrete
and finite.
47
00:02:47,810 --> 00:02:50,130
There's a finite number
of states.
48
00:02:50,130 --> 00:02:53,540
At any point in time, the
process is sitting on one of
49
00:02:53,540 --> 00:02:54,580
those states.
50
00:02:54,580 --> 00:03:00,520
Time is discrete, so at each
unit of time, somebody
51
00:03:00,520 --> 00:03:03,000
whistles and then
the state jumps.
52
00:03:03,000 --> 00:03:06,110
And when it jumps, it can either
land in the same place,
53
00:03:06,110 --> 00:03:08,180
or it can land somewhere else.
54
00:03:08,180 --> 00:03:10,740
And the evolution of the
process is described by
55
00:03:10,740 --> 00:03:13,130
transition probabilities.
56
00:03:13,130 --> 00:03:16,740
Pij is the probability that the
next state is j given that
57
00:03:16,740 --> 00:03:18,630
the current state is i.
58
00:03:18,630 --> 00:03:21,640
And the most important property
that the Markov chain
59
00:03:21,640 --> 00:03:24,690
has, the definition of a
Markov chain or Markov
60
00:03:24,690 --> 00:03:28,880
process, is that this
probability, Pij, is the same
61
00:03:28,880 --> 00:03:31,490
every time that you
land at state i --
62
00:03:31,490 --> 00:03:36,320
no matter how you got there
and also no matter
63
00:03:36,320 --> 00:03:38,000
what time it is.
64
00:03:38,000 --> 00:03:41,650
So the model we have is time
homogeneous, which basically
65
00:03:41,650 --> 00:03:44,630
means that those transition
probabilities are the same at
66
00:03:44,630 --> 00:03:45,720
every time.
67
00:03:45,720 --> 00:03:49,320
So the model is time invariant
in that sense.
68
00:03:49,320 --> 00:03:52,670
So we're interested in what the
chain or the process is
69
00:03:52,670 --> 00:03:54,780
going to do in the longer run.
70
00:03:54,780 --> 00:03:57,060
So we're interested, let's say,
in the probability that
71
00:03:57,060 --> 00:04:00,760
starting at a certain state, n
times steps later, we find
72
00:04:00,760 --> 00:04:03,760
ourselves at some particular
state j.
73
00:04:03,760 --> 00:04:06,350
Fortunately, we can calculate
those probabilities
74
00:04:06,350 --> 00:04:07,660
recursively.
75
00:04:07,660 --> 00:04:12,990
Of course, at the first time 1,
the probability of being 1
76
00:04:12,990 --> 00:04:16,950
time later at state j given that
we are right now at state
77
00:04:16,950 --> 00:04:20,910
i, by definition, this is just
the transition probabilities.
78
00:04:20,910 --> 00:04:26,000
So by knowing these, we can
start a recursion that tells
79
00:04:26,000 --> 00:04:27,730
us the transition probabilities
80
00:04:27,730 --> 00:04:31,430
for more than n steps.
81
00:04:31,430 --> 00:04:32,865
This recursion, it's
a formula.
82
00:04:32,865 --> 00:04:34,000
It's always true.
83
00:04:34,000 --> 00:04:36,860
You can copy it or
memorize it.
84
00:04:36,860 --> 00:04:40,460
But there is a big idea behind
that formula that you should
85
00:04:40,460 --> 00:04:41,480
keep in mind.
86
00:04:41,480 --> 00:04:43,750
And basically, the divide
and conquer idea.
87
00:04:43,750 --> 00:04:47,530
It's an application of the
total probability law.
88
00:04:47,530 --> 00:04:49,080
So let's fix i.
89
00:04:49,080 --> 00:04:52,360
The probability that you find
yourself at state j, you break
90
00:04:52,360 --> 00:04:55,770
it up into the probabilities of
the different ways that you
91
00:04:55,770 --> 00:04:57,850
can get to state j.
92
00:04:57,850 --> 00:04:59,240
What are those different ways?
93
00:04:59,240 --> 00:05:02,460
The different ways are the
different states k at which
94
00:05:02,460 --> 00:05:05,220
you might find yourself
the previous time.
95
00:05:05,220 --> 00:05:08,400
So with some probability, with
this probability, you find
96
00:05:08,400 --> 00:05:11,070
yourself at state k
the previous time.
97
00:05:11,070 --> 00:05:13,350
And then with probability
Pkj, you make a
98
00:05:13,350 --> 00:05:15,030
transition to state j.
99
00:05:15,030 --> 00:05:19,000
So this is a possible scenario
that takes you to state j
100
00:05:19,000 --> 00:05:21,090
after n transitions.
101
00:05:21,090 --> 00:05:25,350
And by summing over all the k's,
then we have considered
102
00:05:25,350 --> 00:05:28,430
all the possible scenarios.
103
00:05:28,430 --> 00:05:31,590
Now, before we move to the more
serious stuff, let's do a
104
00:05:31,590 --> 00:05:38,420
little bit of warm up to get
a handle on how we use
105
00:05:38,420 --> 00:05:42,170
transition probabilities to
calculate more general
106
00:05:42,170 --> 00:05:45,250
probabilities, then talk about
some structural properties of
107
00:05:45,250 --> 00:05:47,980
Markov chains, and then
eventually get to the main
108
00:05:47,980 --> 00:05:51,310
business of today, which is
a steady-state behavior.
109
00:05:51,310 --> 00:05:56,380
So somebody gives you this
chain, and our convention is
110
00:05:56,380 --> 00:06:00,340
that those arcs that are not
shown here corresponds to 0
111
00:06:00,340 --> 00:06:01,510
probabilities.
112
00:06:01,510 --> 00:06:05,760
And each one of the arcs that's
shown has a non-zero
113
00:06:05,760 --> 00:06:09,060
probability, and somebody
gives it to us.
114
00:06:09,060 --> 00:06:11,650
Suppose that the chain
starts at state 1.
115
00:06:11,650 --> 00:06:15,720
We want to calculate the
probability that it follows
116
00:06:15,720 --> 00:06:16,830
this particular path.
117
00:06:16,830 --> 00:06:20,620
That is, it goes to 2,
then to 6, then to 7.
118
00:06:20,620 --> 00:06:22,920
How do we calculate the
probability of a particular
119
00:06:22,920 --> 00:06:24,280
trajectory?
120
00:06:24,280 --> 00:06:26,700
Well, this is the
probability--
121
00:06:26,700 --> 00:06:30,980
so it's the probability of the
trajectory from 1 that you go
122
00:06:30,980 --> 00:06:34,480
to 2, then to 6, then to 7.
123
00:06:34,480 --> 00:06:38,020
So the probability of this
trajectory is we use the
124
00:06:38,020 --> 00:06:39,270
multiplication rule.
125
00:06:39,270 --> 00:06:42,140
The probability of several
things happening is the
126
00:06:42,140 --> 00:06:44,700
probability that the first
thing happens, which is a
127
00:06:44,700 --> 00:06:47,140
transition from 1 to 2.
128
00:06:47,140 --> 00:06:53,190
And then given that we are at
state 2, we multiply with a
129
00:06:53,190 --> 00:06:57,120
conditional probability that
the next event happens.
130
00:06:57,120 --> 00:07:02,370
That is, that X2 is equal to 6
given that right now, we are
131
00:07:02,370 --> 00:07:03,570
at state 1.
132
00:07:03,570 --> 00:07:07,760
And that conditional probability
is just P26.
133
00:07:07,760 --> 00:07:09,840
And notice that this conditional
probability
134
00:07:09,840 --> 00:07:13,380
applies no matter how
we got to state 2.
135
00:07:13,380 --> 00:07:15,340
This is the Markov assumption.
136
00:07:15,340 --> 00:07:17,970
So we don't care about the
fact that we came in in a
137
00:07:17,970 --> 00:07:19,120
particular way.
138
00:07:19,120 --> 00:07:22,370
Given that we came in here, this
probability P26, that the
139
00:07:22,370 --> 00:07:24,640
next transition takes us to 6.
140
00:07:24,640 --> 00:07:28,620
And then given that all that
stuff happened, so given that
141
00:07:28,620 --> 00:07:32,060
right now, we are at state 6,
we need to multiply with a
142
00:07:32,060 --> 00:07:34,680
conditional probability that the
next transition takes us
143
00:07:34,680 --> 00:07:36,020
to state 7.
144
00:07:36,020 --> 00:07:39,920
And this is just the P67.
145
00:07:39,920 --> 00:07:44,140
So to find the probability
of following a specific
146
00:07:44,140 --> 00:07:49,280
trajectory, you just multiply
the transition probabilities
147
00:07:49,280 --> 00:07:50,885
along the particular
trajectory.
148
00:07:50,885 --> 00:07:54,490
149
00:07:54,490 --> 00:07:58,150
Now, if you want to calculate
something else, such as for
150
00:07:58,150 --> 00:08:02,100
example, the probability that
4 time steps later, I find
151
00:08:02,100 --> 00:08:06,250
myself at state 7 given that
they started, let's say, at
152
00:08:06,250 --> 00:08:07,680
this state.
153
00:08:07,680 --> 00:08:09,920
How do you calculate
this probability?
154
00:08:09,920 --> 00:08:15,350
One way is to use the recursion
for the Rijs that we
155
00:08:15,350 --> 00:08:17,990
know that it is always valid.
156
00:08:17,990 --> 00:08:20,970
But for short and simple
examples, and with a small
157
00:08:20,970 --> 00:08:24,890
time horizon, perhaps you can do
this in a brute force way.
158
00:08:24,890 --> 00:08:27,100
What would be the
brute force way?
159
00:08:27,100 --> 00:08:30,410
This is the event that 4 time
steps later, I find
160
00:08:30,410 --> 00:08:32,210
myself at state 7.
161
00:08:32,210 --> 00:08:36,350
This event can happen
in various ways.
162
00:08:36,350 --> 00:08:41,070
So we can take stock of all the
different ways, and write
163
00:08:41,070 --> 00:08:42,760
down their probabilities.
164
00:08:42,760 --> 00:08:44,800
So starting from 2.
165
00:08:44,800 --> 00:08:49,870
One possibility is to follow
this trajectory, 1 transition,
166
00:08:49,870 --> 00:08:53,790
2 transitions, 3 transitions,
4 transitions.
167
00:08:53,790 --> 00:08:55,810
And that takes me to state 7.
168
00:08:55,810 --> 00:08:57,760
What's the probability
of this trajectory?
169
00:08:57,760 --> 00:09:05,160
It's P26 times P67 times
P76 and then times P67.
170
00:09:05,160 --> 00:09:08,000
So this is a probability of a
particular trajectory that
171
00:09:08,000 --> 00:09:11,040
takes you to state 7
after 4 time steps.
172
00:09:11,040 --> 00:09:14,790
But there's other trajectories
as well.
173
00:09:14,790 --> 00:09:16,300
What could be it?
174
00:09:16,300 --> 00:09:23,470
I might start from state 2, go
to state 6, stay at state 6,
175
00:09:23,470 --> 00:09:26,450
stay at state 6 once more.
176
00:09:26,450 --> 00:09:31,670
And then from state
6, go to state 7.
177
00:09:31,670 --> 00:09:36,800
And so there must be one more.
178
00:09:36,800 --> 00:09:38,470
What's the other one?
179
00:09:38,470 --> 00:09:45,620
I guess I could go 1, 2, 6, 7.
180
00:09:45,620 --> 00:09:45,690
OK.
181
00:09:45,690 --> 00:09:48,012
That's the other trajectory.
182
00:09:48,012 --> 00:10:01,850
Plus P21 times P12 times
P26 and times P67.
183
00:10:01,850 --> 00:10:05,580
So the transition probability,
the overall probability of
184
00:10:05,580 --> 00:10:09,490
finding ourselves at state 7,
is broken down as the sum of
185
00:10:09,490 --> 00:10:12,580
the probabilities of all the
different ways that I can get
186
00:10:12,580 --> 00:10:16,000
to state 7 in exactly 4 steps.
187
00:10:16,000 --> 00:10:19,460
So we could always do that
without knowing much about
188
00:10:19,460 --> 00:10:21,750
Markov chains or the general
formula for the
189
00:10:21,750 --> 00:10:24,490
Rij's that we had.
190
00:10:24,490 --> 00:10:26,370
What's the trouble with
this procedure?
191
00:10:26,370 --> 00:10:29,190
The trouble with this procedure
is that the number
192
00:10:29,190 --> 00:10:34,450
of possible trajectories becomes
quite large if this
193
00:10:34,450 --> 00:10:37,550
index is a little bigger.
194
00:10:37,550 --> 00:10:41,510
If this 4 was 100, and you
ask how many different
195
00:10:41,510 --> 00:10:45,240
trajectories of length 100 are
there to take me from here to
196
00:10:45,240 --> 00:10:48,170
there, that number of
trajectories would be huge.
197
00:10:48,170 --> 00:10:51,120
It grows exponentially with
the time horizon.
198
00:10:51,120 --> 00:10:55,110
And this kind of calculation
would be impossible.
199
00:10:55,110 --> 00:10:58,310
The basic equation, the
recursion that have for the
200
00:10:58,310 --> 00:11:01,650
Rij's is basically a clever
way of organizing this
201
00:11:01,650 --> 00:11:04,870
computation so that the amount
of computation that you do is
202
00:11:04,870 --> 00:11:06,960
not exponential in
the time horizon.
203
00:11:06,960 --> 00:11:11,100
Rather, it's sort of linear
with the time horizon.
204
00:11:11,100 --> 00:11:14,590
For each time step you need in
the time horizon, you just
205
00:11:14,590 --> 00:11:17,260
keep repeating the same
iteration over and over.
206
00:11:17,260 --> 00:11:20,460
207
00:11:20,460 --> 00:11:20,565
OK.
208
00:11:20,565 --> 00:11:24,940
Now, the other thing that we
discussed last time, briefly,
209
00:11:24,940 --> 00:11:28,530
was a classification of the
different states of the Markov
210
00:11:28,530 --> 00:11:31,510
chain in two different types.
211
00:11:31,510 --> 00:11:37,020
A Markov chain, in general, has
states that are recurrent,
212
00:11:37,020 --> 00:11:39,670
which means that from a
recurrent state, I can go
213
00:11:39,670 --> 00:11:41,000
somewhere else.
214
00:11:41,000 --> 00:11:44,910
But from that somewhere else,
there's always some way of
215
00:11:44,910 --> 00:11:46,040
coming back.
216
00:11:46,040 --> 00:11:49,910
So if you have a chain of this
form, no matter where you go,
217
00:11:49,910 --> 00:11:52,500
no matter where you start,
you can always come
218
00:11:52,500 --> 00:11:54,540
back where you started.
219
00:11:54,540 --> 00:11:56,800
States of this kind are
called recurrent.
220
00:11:56,800 --> 00:12:00,320
On the other hand, if you have
a few states all this kind, a
221
00:12:00,320 --> 00:12:04,560
transition of this type, then
these states are transient in
222
00:12:04,560 --> 00:12:07,560
the sense that from those
states, it's possible to go
223
00:12:07,560 --> 00:12:11,520
somewhere else from which place
there's no way to come
224
00:12:11,520 --> 00:12:13,580
back where you started.
225
00:12:13,580 --> 00:12:18,370
The general structure of a
Markov chain is basically a
226
00:12:18,370 --> 00:12:20,660
collection of transient
states.
227
00:12:20,660 --> 00:12:23,960
You're certain that you are
going to leave the transient
228
00:12:23,960 --> 00:12:27,370
states eventually.
229
00:12:27,370 --> 00:12:30,300
And after you leave the
transient states, you enter
230
00:12:30,300 --> 00:12:33,570
into a class of states in
which you are trapped.
231
00:12:33,570 --> 00:12:35,890
You are trapped if you
get inside here.
232
00:12:35,890 --> 00:12:39,040
You are trapped if you
get inside there.
233
00:12:39,040 --> 00:12:41,270
This is a recurrent
class of states.
234
00:12:41,270 --> 00:12:43,210
From any state, you can
get to any other
235
00:12:43,210 --> 00:12:44,380
state within this class.
236
00:12:44,380 --> 00:12:46,100
That's another recurrent
class.
237
00:12:46,100 --> 00:12:49,380
From any state inside here,
you can get anywhere else
238
00:12:49,380 --> 00:12:50,680
inside that class.
239
00:12:50,680 --> 00:12:52,930
But these 2 classes, you
do not communicate.
240
00:12:52,930 --> 00:12:56,310
If you start here, there's
no way to get there.
241
00:12:56,310 --> 00:12:59,820
If you have 2 recurrent classes,
then it's clear that
242
00:12:59,820 --> 00:13:02,310
the initial conditions
of your Markov chain
243
00:13:02,310 --> 00:13:04,130
matter in the long run.
244
00:13:04,130 --> 00:13:07,450
If you start here, you will be
stuck inside here for the long
245
00:13:07,450 --> 00:13:09,140
run and similarly about here.
246
00:13:09,140 --> 00:13:11,680
So the initial conditions
do make a difference.
247
00:13:11,680 --> 00:13:14,720
On the other hand, if this class
was not here and you
248
00:13:14,720 --> 00:13:17,300
only had that class, what would
happen to the chain?
249
00:13:17,300 --> 00:13:18,480
Let's say you start here.
250
00:13:18,480 --> 00:13:19,490
You move around.
251
00:13:19,490 --> 00:13:21,730
At some point, you make
that transition.
252
00:13:21,730 --> 00:13:23,260
You get stuck in here.
253
00:13:23,260 --> 00:13:26,410
And inside here, you keep
circulating, because of the
254
00:13:26,410 --> 00:13:30,210
randomness, you keep visiting
all states over and over.
255
00:13:30,210 --> 00:13:35,800
And hopefully or possibly, in
the long run, it doesn't
256
00:13:35,800 --> 00:13:39,610
matter exactly what time it is
or where you started, but the
257
00:13:39,610 --> 00:13:43,860
probability of being at that
particular state is the same
258
00:13:43,860 --> 00:13:46,310
no matter what the initial
condition was.
259
00:13:46,310 --> 00:13:48,820
So with a single recurrent
class, we hope that the
260
00:13:48,820 --> 00:13:50,750
initial conditions
do not matter.
261
00:13:50,750 --> 00:13:55,780
With 2 or more recurrent
classes, initial conditions
262
00:13:55,780 --> 00:13:58,660
will definitely matter.
263
00:13:58,660 --> 00:14:03,620
So how many recurrent classes we
have is something that has
264
00:14:03,620 --> 00:14:06,790
to do with the long-term
behavior of the chain and the
265
00:14:06,790 --> 00:14:09,790
extent to which initial
conditions matter.
266
00:14:09,790 --> 00:14:16,000
Another way that initial
conditions may matter is if a
267
00:14:16,000 --> 00:14:19,360
chain has a periodic
structure.
268
00:14:19,360 --> 00:14:21,990
There are many ways of
defining periodicity.
269
00:14:21,990 --> 00:14:25,240
The one that I find sort of the
most intuitive and with
270
00:14:25,240 --> 00:14:27,410
the least amount
of mathematical
271
00:14:27,410 --> 00:14:29,670
symbols is the following.
272
00:14:29,670 --> 00:14:34,630
The state space of a chain is
said to be periodic if you can
273
00:14:34,630 --> 00:14:39,510
lump the states into a number
of clusters called
274
00:14:39,510 --> 00:14:42,550
d clusters or groups.
275
00:14:42,550 --> 00:14:45,980
And the transition diagram has
the property that from a
276
00:14:45,980 --> 00:14:48,860
cluster, you always
make a transition
277
00:14:48,860 --> 00:14:50,870
into the next cluster.
278
00:14:50,870 --> 00:14:52,860
So here d is equal to 2.
279
00:14:52,860 --> 00:14:55,570
We have two subsets of
the state space.
280
00:14:55,570 --> 00:14:58,130
Whenever we're here, next
time we'll be there.
281
00:14:58,130 --> 00:15:01,080
Whenever we're here, next
time we will be there.
282
00:15:01,080 --> 00:15:03,830
So this chain has a periodic
structure.
283
00:15:03,830 --> 00:15:05,880
There may be still
some randomness.
284
00:15:05,880 --> 00:15:10,270
When I jump from here to here,
the state to which I jump may
285
00:15:10,270 --> 00:15:14,490
be random, but I'm sure that I'm
going to be inside here.
286
00:15:14,490 --> 00:15:17,610
And then next time, I will be
sure that I'm inside here.
287
00:15:17,610 --> 00:15:20,310
This would be a structure of a
diagram in which we have a
288
00:15:20,310 --> 00:15:21,410
period of 3.
289
00:15:21,410 --> 00:15:25,540
If you start in this lump, you
know that the next time, you
290
00:15:25,540 --> 00:15:27,480
would be in a state
inside here.
291
00:15:27,480 --> 00:15:30,830
Next time, you'll be in a state
inside here, and so on.
292
00:15:30,830 --> 00:15:35,220
So these chains certainly have
a periodic structure.
293
00:15:35,220 --> 00:15:37,860
And that periodicity
gets maintained.
294
00:15:37,860 --> 00:15:42,500
If I start, let's say, at this
lump, at even times,
295
00:15:42,500 --> 00:15:44,500
I'm sure I'm here.
296
00:15:44,500 --> 00:15:47,660
At odd times, I'm
sure I am here.
297
00:15:47,660 --> 00:15:51,600
So the exact time does matter
in determining the
298
00:15:51,600 --> 00:15:54,660
probabilities of the
different states.
299
00:15:54,660 --> 00:15:57,140
And in particular, the
probability of being at the
300
00:15:57,140 --> 00:16:00,500
particular state cannot convert
to a state value.
301
00:16:00,500 --> 00:16:03,410
The probability of being at the
state inside here is going
302
00:16:03,410 --> 00:16:06,300
to be 0 for all times.
303
00:16:06,300 --> 00:16:08,290
In general, it's going
to be some positive
304
00:16:08,290 --> 00:16:10,160
number for even times.
305
00:16:10,160 --> 00:16:13,810
So it goes 0 positive, zero,
positive, 0 positive.
306
00:16:13,810 --> 00:16:15,370
Doesn't settle to anything.
307
00:16:15,370 --> 00:16:19,860
So when we have periodicity,
we do not expect the states
308
00:16:19,860 --> 00:16:22,600
probabilities to converge to
something, but rather, we
309
00:16:22,600 --> 00:16:24,580
expect them to oscillate.
310
00:16:24,580 --> 00:16:26,830
Now, how can we tell whether
a Markov chain
311
00:16:26,830 --> 00:16:29,920
is periodic or not?
312
00:16:29,920 --> 00:16:33,900
There are systematic ways of
doing it, but usually with the
313
00:16:33,900 --> 00:16:36,560
types of examples we see in this
class, we just eyeball
314
00:16:36,560 --> 00:16:39,890
the chain, and we tell whether
it's periodic or not.
315
00:16:39,890 --> 00:16:45,240
So is this chain down here, is
it the periodic one or not?
316
00:16:45,240 --> 00:16:48,870
How many people think
it's periodic?
317
00:16:48,870 --> 00:16:50,680
No one.
318
00:16:50,680 --> 00:16:51,200
One.
319
00:16:51,200 --> 00:16:54,070
How many people think
it's not periodic?
320
00:16:54,070 --> 00:16:54,560
OK.
321
00:16:54,560 --> 00:16:56,270
Not periodic?
322
00:16:56,270 --> 00:16:57,570
Let's see.
323
00:16:57,570 --> 00:16:59,490
Let me do some drawing here.
324
00:16:59,490 --> 00:17:03,230
325
00:17:03,230 --> 00:17:04,119
OK.
326
00:17:04,119 --> 00:17:05,369
Is it periodic?
327
00:17:05,369 --> 00:17:07,856
328
00:17:07,856 --> 00:17:09,140
It is.
329
00:17:09,140 --> 00:17:14,180
From a red state, you can only
get to a white state.
330
00:17:14,180 --> 00:17:17,849
And from a white state, you can
only get to a red state.
331
00:17:17,849 --> 00:17:20,660
So this chain, even though it's
not apparent from the
332
00:17:20,660 --> 00:17:24,589
picture, actually has
this structure.
333
00:17:24,589 --> 00:17:28,600
We can group the states into red
states and white states.
334
00:17:28,600 --> 00:17:32,810
And from reds, we always go to
a white, and from a white, we
335
00:17:32,810 --> 00:17:34,500
always go to a red.
336
00:17:34,500 --> 00:17:36,540
So this tells you
that sometimes
337
00:17:36,540 --> 00:17:38,680
eyeballing is not as easy.
338
00:17:38,680 --> 00:17:40,810
If you have lots and lots of
states, you might have some
339
00:17:40,810 --> 00:17:43,280
trouble doing this exercise.
340
00:17:43,280 --> 00:17:47,230
On the other hand, something
very useful to know.
341
00:17:47,230 --> 00:17:50,270
Sometimes it's extremely
easy to tell that the
342
00:17:50,270 --> 00:17:52,360
chain is not periodic.
343
00:17:52,360 --> 00:17:53,770
What's that case?
344
00:17:53,770 --> 00:17:58,600
Suppose that your chain has a
self-transition somewhere.
345
00:17:58,600 --> 00:18:01,670
Then automatically,
you know that your
346
00:18:01,670 --> 00:18:04,660
chain is not periodic.
347
00:18:04,660 --> 00:18:08,560
So remember, the definition of
periodicity requires that if
348
00:18:08,560 --> 00:18:12,230
you are in a certain group of
states, next time, you will be
349
00:18:12,230 --> 00:18:14,380
in a different group.
350
00:18:14,380 --> 00:18:16,450
But if you have
self-transitions, that
351
00:18:16,450 --> 00:18:17,600
property is not true.
352
00:18:17,600 --> 00:18:20,790
If you have a possible
self-transition, it's possible
353
00:18:20,790 --> 00:18:24,530
that you stay inside your own
group for the next time step.
354
00:18:24,530 --> 00:18:29,560
So whenever you have a
self-transition, this implies
355
00:18:29,560 --> 00:18:31,440
that the chain is
not periodic.
356
00:18:31,440 --> 00:18:34,280
357
00:18:34,280 --> 00:18:39,240
And usually that's the simplest
and easy way that we
358
00:18:39,240 --> 00:18:44,210
can tell most of the time that
the chain is not periodic.
359
00:18:44,210 --> 00:18:49,110
So now, we come to the big topic
of today, the central
360
00:18:49,110 --> 00:18:53,080
topic, which is the question
about what does the chain do
361
00:18:53,080 --> 00:18:55,550
in the long run.
362
00:18:55,550 --> 00:19:00,060
The question we are asking and
which we motivated last time
363
00:19:00,060 --> 00:19:02,510
by looking at an example.
364
00:19:02,510 --> 00:19:05,790
It's something that did happen
in our example of last time.
365
00:19:05,790 --> 00:19:08,000
So we're asking whether
this happens for
366
00:19:08,000 --> 00:19:09,440
every Markov chain.
367
00:19:09,440 --> 00:19:12,400
We're asking the question
whether the probability of
368
00:19:12,400 --> 00:19:18,250
being at state j at some
time n settles to a
369
00:19:18,250 --> 00:19:20,350
steady-state value.
370
00:19:20,350 --> 00:19:22,900
Let's call it pi sub j.
371
00:19:22,900 --> 00:19:26,960
That these were asking whether
this quantity has a limit as n
372
00:19:26,960 --> 00:19:29,400
goes to infinity, so that
we can talk about the
373
00:19:29,400 --> 00:19:32,300
steady-state probability
of state j.
374
00:19:32,300 --> 00:19:36,060
And furthermore, we asked
whether the steady-state
375
00:19:36,060 --> 00:19:38,900
probability of that state
does not depend
376
00:19:38,900 --> 00:19:40,800
on the initial state.
377
00:19:40,800 --> 00:19:44,110
In other words, after the chain
runs for a long, long
378
00:19:44,110 --> 00:19:48,620
time, it doesn't matter exactly
what time it is, and
379
00:19:48,620 --> 00:19:51,990
it doesn't matter where the
chain started from.
380
00:19:51,990 --> 00:19:54,860
You can tell me the probability
that the state is
381
00:19:54,860 --> 00:19:58,700
a particular j is approximately
the steady-state
382
00:19:58,700 --> 00:20:00,450
probability pi sub j.
383
00:20:00,450 --> 00:20:03,470
It doesn't matter exactly what
time it is as long as you tell
384
00:20:03,470 --> 00:20:06,900
me that a lot of time
has elapsed so
385
00:20:06,900 --> 00:20:09,520
that n is a big number.
386
00:20:09,520 --> 00:20:11,150
So this is the question.
387
00:20:11,150 --> 00:20:14,210
We have seen examples, and we
understand that this is not
388
00:20:14,210 --> 00:20:16,600
going to be the case always.
389
00:20:16,600 --> 00:20:19,880
For example, as I just
discussed, if we have 2
390
00:20:19,880 --> 00:20:23,850
recurrent classes, where
we start does matter.
391
00:20:23,850 --> 00:20:28,060
The probability pi(j) of being
in that state j is going to be
392
00:20:28,060 --> 00:20:32,650
0 if we start here, but it would
be something positive if
393
00:20:32,650 --> 00:20:34,710
we were to start in that lump.
394
00:20:34,710 --> 00:20:37,650
So the initial state does matter
if we have multiple
395
00:20:37,650 --> 00:20:39,690
recurrent classes.
396
00:20:39,690 --> 00:20:45,590
But if we have only a single
class of recurrent states from
397
00:20:45,590 --> 00:20:48,780
each one of which you can get
to any other one, then we
398
00:20:48,780 --> 00:20:49,980
don't have that problem.
399
00:20:49,980 --> 00:20:53,010
Then we expect initial
conditions to be forgotten.
400
00:20:53,010 --> 00:20:55,100
So that's one condition
that we need.
401
00:20:55,100 --> 00:20:58,960
402
00:20:58,960 --> 00:21:01,670
And then the other condition
that we need is that the chain
403
00:21:01,670 --> 00:21:02,930
is not periodic.
404
00:21:02,930 --> 00:21:07,580
If the chain is periodic, then
these Rij's do not converge.
405
00:21:07,580 --> 00:21:09,510
They keep oscillating.
406
00:21:09,510 --> 00:21:13,190
If we do not have periodicity,
then there is hope that we
407
00:21:13,190 --> 00:21:16,070
will get the convergence
that we need.
408
00:21:16,070 --> 00:21:19,210
It turns out this is the big
theory of Markov chains-- the
409
00:21:19,210 --> 00:21:20,900
steady-state convergence
theorem.
410
00:21:20,900 --> 00:21:26,470
It turns out that yes, the
rijs do converge to a
411
00:21:26,470 --> 00:21:29,510
steady-state limit, which
we call a steady-state
412
00:21:29,510 --> 00:21:35,180
probability as long as these two
conditions are satisfied.
413
00:21:35,180 --> 00:21:37,500
We're not going to prove
this theorem.
414
00:21:37,500 --> 00:21:41,790
If you're really interested, the
end of chapter exercises
415
00:21:41,790 --> 00:21:45,670
basically walk you through a
proof of this result, but it's
416
00:21:45,670 --> 00:21:49,470
probably a little too much for
doing it in this class.
417
00:21:49,470 --> 00:21:52,140
What is the intuitive idea
behind this theorem?
418
00:21:52,140 --> 00:21:52,860
Let's see.
419
00:21:52,860 --> 00:21:56,830
Let's think intuitively
as to why the initial
420
00:21:56,830 --> 00:21:59,010
state doesn't matter.
421
00:21:59,010 --> 00:22:02,870
Think of two copies of the chain
that starts at different
422
00:22:02,870 --> 00:22:06,430
initial states, and the
state moves randomly.
423
00:22:06,430 --> 00:22:09,390
As the state moves randomly
starting from the two initial
424
00:22:09,390 --> 00:22:11,900
states a random trajectory.
425
00:22:11,900 --> 00:22:15,610
as long as you have a single
recurrent class at some point,
426
00:22:15,610 --> 00:22:19,170
and you don't have periodicity
at some point, those states,
427
00:22:19,170 --> 00:22:22,610
those two trajectories,
are going to collide.
428
00:22:22,610 --> 00:22:25,650
Just because there's enough
randomness there.
429
00:22:25,650 --> 00:22:28,830
Even though we started from
different places, the state is
430
00:22:28,830 --> 00:22:30,490
going to be the same.
431
00:22:30,490 --> 00:22:33,540
After the state becomes the
same, then the future of these
432
00:22:33,540 --> 00:22:37,100
trajectories, probabilistically,
is the same
433
00:22:37,100 --> 00:22:39,800
because they both started
at the same state.
434
00:22:39,800 --> 00:22:42,540
So this means that the
initial conditions
435
00:22:42,540 --> 00:22:45,330
stopped having any influence.
436
00:22:45,330 --> 00:22:50,040
That's sort of the high-level
idea of why the initial state
437
00:22:50,040 --> 00:22:50,820
gets forgotten.
438
00:22:50,820 --> 00:22:53,780
Even if you started at different
initial states, at
439
00:22:53,780 --> 00:22:57,030
some time, you may find yourself
to be in the same
440
00:22:57,030 --> 00:22:59,120
state as the other trajectory.
441
00:22:59,120 --> 00:23:05,370
And once that happens, your
initial conditions cannot have
442
00:23:05,370 --> 00:23:08,210
any effect into the future.
443
00:23:08,210 --> 00:23:09,230
All right.
444
00:23:09,230 --> 00:23:15,650
So let's see how we might
calculate those steady-state
445
00:23:15,650 --> 00:23:16,870
probabilities.
446
00:23:16,870 --> 00:23:19,850
The way we calculate the
steady-state probabilities is
447
00:23:19,850 --> 00:23:24,230
by taking this recursion, which
is always true for the
448
00:23:24,230 --> 00:23:27,150
end-step transition
probabilities, and take the
449
00:23:27,150 --> 00:23:29,400
limit of both sides.
450
00:23:29,400 --> 00:23:32,970
The limit of this side is the
steady-state probability of
451
00:23:32,970 --> 00:23:36,010
state j, which is pi sub j.
452
00:23:36,010 --> 00:23:38,140
The limit of this
side, we put the
453
00:23:38,140 --> 00:23:40,120
limit inside the summation.
454
00:23:40,120 --> 00:23:44,330
Now, as n goes to infinity,
n - also goes to infinity.
455
00:23:44,330 --> 00:23:48,530
So this Rik is going to be the
steady-state probability of
456
00:23:48,530 --> 00:23:51,150
state k starting from state i.
457
00:23:51,150 --> 00:23:53,170
Now where we started
doesn't matter.
458
00:23:53,170 --> 00:23:54,620
So this is just the
steady-state
459
00:23:54,620 --> 00:23:56,290
probability of state k.
460
00:23:56,290 --> 00:24:00,000
So this term converges to that
one, and this gives us an
461
00:24:00,000 --> 00:24:03,010
equation that's satisfied
by the steady-state
462
00:24:03,010 --> 00:24:04,030
probabilities.
463
00:24:04,030 --> 00:24:06,150
Actually, it's not
one equation.
464
00:24:06,150 --> 00:24:10,580
We get one equation for
each one of the j's.
465
00:24:10,580 --> 00:24:13,220
So if we have 10 possible
states, we're going to get the
466
00:24:13,220 --> 00:24:15,580
system of 10 linear equations.
467
00:24:15,580 --> 00:24:18,840
In the unknowns, pi(1)
up to pi(10).
468
00:24:18,840 --> 00:24:19,210
OK.
469
00:24:19,210 --> 00:24:20,800
10 unknowns, 10 equations.
470
00:24:20,800 --> 00:24:23,150
You might think that
we are in business.
471
00:24:23,150 --> 00:24:27,646
But actually, this system of
equations is singular.
472
00:24:27,646 --> 00:24:30,180
0 is a possible solution
of this system.
473
00:24:30,180 --> 00:24:32,900
If you plug pi equal to
zero everywhere, the
474
00:24:32,900 --> 00:24:33,900
equations are satisfied.
475
00:24:33,900 --> 00:24:37,550
It does not have a unique
solution, so maybe we need one
476
00:24:37,550 --> 00:24:40,580
more condition to get the
uniquely solvable system of
477
00:24:40,580 --> 00:24:41,900
linear equations.
478
00:24:41,900 --> 00:24:44,350
It turns out that this
system of equations
479
00:24:44,350 --> 00:24:46,150
has a unique solution.
480
00:24:46,150 --> 00:24:48,980
If you impose an additional
condition, which is pretty
481
00:24:48,980 --> 00:24:52,160
natural, the pi(j)'s are the
probabilities of the different
482
00:24:52,160 --> 00:24:54,960
states, so they should
add to 1.
483
00:24:54,960 --> 00:24:58,410
So you want this one equation
to the mix.
484
00:24:58,410 --> 00:25:05,490
And once you do that, then this
system of equations is
485
00:25:05,490 --> 00:25:07,340
going to have a unique
solution.
486
00:25:07,340 --> 00:25:09,550
And so we can find the
steady-state probabilities of
487
00:25:09,550 --> 00:25:12,980
the Markov chain by just
solving these linear
488
00:25:12,980 --> 00:25:16,130
equations, which is numerically
straightforward.
489
00:25:16,130 --> 00:25:18,790
Now, these equations are
quite important.
490
00:25:18,790 --> 00:25:23,240
I mean, they're the central
point in the Markov chain.
491
00:25:23,240 --> 00:25:24,220
They have a name.
492
00:25:24,220 --> 00:25:27,030
They're called the balance
equations.
493
00:25:27,030 --> 00:25:31,260
And it's worth interpreting
them in a
494
00:25:31,260 --> 00:25:33,450
somewhat different way.
495
00:25:33,450 --> 00:25:37,030
So intuitively, one can
sometimes think of
496
00:25:37,030 --> 00:25:39,290
probabilities as frequencies.
497
00:25:39,290 --> 00:25:45,780
For example, if I toss an
unbiased coin, probability 1/2
498
00:25:45,780 --> 00:25:49,600
of heads, you could also say
that if I keep flipping that
499
00:25:49,600 --> 00:25:52,510
coin, in the long run,
1/2 of the time, I'm
500
00:25:52,510 --> 00:25:54,500
going to see heads.
501
00:25:54,500 --> 00:25:58,910
Similarly, let's try an
interpretation of this pi(j),
502
00:25:58,910 --> 00:26:02,500
the steady-state probability,
the long-term probability of
503
00:26:02,500 --> 00:26:04,980
finding myself at state j.
504
00:26:04,980 --> 00:26:08,440
Let's try to interpret it as
the frequency with which I
505
00:26:08,440 --> 00:26:12,620
find myself at state j if
I run a very, very long
506
00:26:12,620 --> 00:26:14,940
trajectory over that
Markov chain.
507
00:26:14,940 --> 00:26:18,400
So the trajectory moves
around, visits states.
508
00:26:18,400 --> 00:26:22,470
It visits the different states
with different frequencies.
509
00:26:22,470 --> 00:26:27,380
And let's think of the
probability that you are at a
510
00:26:27,380 --> 00:26:32,270
certain state as being sort of
the same as the frequency of
511
00:26:32,270 --> 00:26:34,420
visiting that state.
512
00:26:34,420 --> 00:26:37,290
This turns out to be a
correct statement.
513
00:26:37,290 --> 00:26:41,040
If you were more rigorous, you
would have to prove it.
514
00:26:41,040 --> 00:26:44,390
But it's an interpretation which
is valid and which gives
515
00:26:44,390 --> 00:26:48,560
us a lot of intuition about what
these equation is saying.
516
00:26:48,560 --> 00:26:50,170
So let's think as follows.
517
00:26:50,170 --> 00:26:54,240
Let's focus on a particular
state j, and think of
518
00:26:54,240 --> 00:27:00,660
transitions into the state j
versus transitions out of the
519
00:27:00,660 --> 00:27:05,410
state j, or transitions into
j versus transitions
520
00:27:05,410 --> 00:27:07,080
starting from j.
521
00:27:07,080 --> 00:27:10,010
So transition starting
from that includes a
522
00:27:10,010 --> 00:27:11,260
self-transition.
523
00:27:11,260 --> 00:27:14,980
524
00:27:14,980 --> 00:27:15,110
Ok.
525
00:27:15,110 --> 00:27:18,300
So how often do we get a
transition, if we interpret
526
00:27:18,300 --> 00:27:21,230
the pi(j)'s as frequencies,
how often do we get a
527
00:27:21,230 --> 00:27:23,110
transition into j?
528
00:27:23,110 --> 00:27:25,870
Here's how we think about it.
529
00:27:25,870 --> 00:27:31,110
A fraction pi(1) of the time,
we're going to be at state 1.
530
00:27:31,110 --> 00:27:35,070
Whenever we are at state 1,
there's going to be a
531
00:27:35,070 --> 00:27:40,550
probability, P1j, that we make
a transition of this kind.
532
00:27:40,550 --> 00:27:44,730
So out of the times that we're
at state 1, there's a
533
00:27:44,730 --> 00:27:50,141
frequency, P1j with which the
next transition is into j.
534
00:27:50,141 --> 00:27:53,230
535
00:27:53,230 --> 00:27:57,870
So out of the overall number of
transitions that happen at
536
00:27:57,870 --> 00:28:01,820
the trajectory, what fraction
of those transitions is
537
00:28:01,820 --> 00:28:03,700
exactly of that kind?
538
00:28:03,700 --> 00:28:06,710
That fraction of transitions is
the fraction of time that
539
00:28:06,710 --> 00:28:11,940
you find yourself at 1 times the
fraction with which out of
540
00:28:11,940 --> 00:28:15,700
one you happen to visit
next state j.
541
00:28:15,700 --> 00:28:19,300
So we interpreted this number
as the frequency of
542
00:28:19,300 --> 00:28:21,610
transitions of this kind.
543
00:28:21,610 --> 00:28:24,780
At any given time, our chain
can do transitions of
544
00:28:24,780 --> 00:28:28,470
different kinds, transitions of
the general form from some
545
00:28:28,470 --> 00:28:30,670
k, I go to some l.
546
00:28:30,670 --> 00:28:33,600
So we try to do some
accounting.
547
00:28:33,600 --> 00:28:37,740
How often does a transition of
each particular kind happen?
548
00:28:37,740 --> 00:28:40,950
And this is the frequency with
which transitions of that
549
00:28:40,950 --> 00:28:42,970
particular kind happens.
550
00:28:42,970 --> 00:28:44,610
Now, what's the total
frequency of
551
00:28:44,610 --> 00:28:46,870
transitions into state j?
552
00:28:46,870 --> 00:28:49,880
Transitions into state j can
happen by having a transition
553
00:28:49,880 --> 00:28:54,510
from 1 to j, from 2 to j,
or from state m to j.
554
00:28:54,510 --> 00:28:58,590
So to find the total frequency
with which we would observe
555
00:28:58,590 --> 00:29:03,960
transitions into j is going
to be this particular sum.
556
00:29:03,960 --> 00:29:09,090
Now, you are at state j if and
only if the last transition
557
00:29:09,090 --> 00:29:11,440
was into state j.
558
00:29:11,440 --> 00:29:16,230
So the frequency with which you
are at j is the frequency
559
00:29:16,230 --> 00:29:20,100
with which transitions
into j happen.
560
00:29:20,100 --> 00:29:24,020
So this equation expresses
exactly that statement.
561
00:29:24,020 --> 00:29:27,740
The probability of being at
state j is the sum of the
562
00:29:27,740 --> 00:29:32,440
probabilities that the last
transition was into state j.
563
00:29:32,440 --> 00:29:35,120
Or in terms of frequencies, the
frequency with which you
564
00:29:35,120 --> 00:29:39,170
find yourself at state j is the
sum of the frequencies of
565
00:29:39,170 --> 00:29:42,580
all the possible transition
types that take you
566
00:29:42,580 --> 00:29:45,360
inside state j.
567
00:29:45,360 --> 00:29:47,770
So that's a useful intuition
to have, and we're going to
568
00:29:47,770 --> 00:29:52,360
see an example a little later
that it gives us short cuts
569
00:29:52,360 --> 00:29:55,240
into analyzing Markov chains.
570
00:29:55,240 --> 00:29:58,270
But before we move,
let's revisit the
571
00:29:58,270 --> 00:30:01,450
example from last time.
572
00:30:01,450 --> 00:30:03,560
And let us write down
the balance
573
00:30:03,560 --> 00:30:06,090
equations for this example.
574
00:30:06,090 --> 00:30:09,380
So the steady-state probability
that I find myself
575
00:30:09,380 --> 00:30:16,370
at state 1 is the probability
that the previous time I was
576
00:30:16,370 --> 00:30:21,550
at state 1 and I made
a self-transition--
577
00:30:21,550 --> 00:30:25,540
So the probability that I was
here last time and I made a
578
00:30:25,540 --> 00:30:28,290
transition of this kind, plus
the probability that the last
579
00:30:28,290 --> 00:30:32,220
time I was here and I made a
transition of that kind.
580
00:30:32,220 --> 00:30:36,100
So plus pi(2) times 0.2.
581
00:30:36,100 --> 00:30:43,060
And similarly, for the other
states, the steady-state
582
00:30:43,060 --> 00:30:46,250
probably that I find myself at
state 2 is the probability
583
00:30:46,250 --> 00:30:50,650
that last time I was at state 1
and I made a transition into
584
00:30:50,650 --> 00:30:53,780
state 2, plus the probability
that the last time I was at
585
00:30:53,780 --> 00:30:57,750
state 2 and I made the
transition into state 1.
586
00:30:57,750 --> 00:30:59,910
Now, these are two
equations and two
587
00:30:59,910 --> 00:31:02,040
unknowns, pi(1) and pi(2).
588
00:31:02,040 --> 00:31:06,150
But you notice that both of
these equations tell you the
589
00:31:06,150 --> 00:31:07,170
same thing.
590
00:31:07,170 --> 00:31:12,226
They tell you that 0.5pi(1)
equals 0.2pi(2).
591
00:31:12,226 --> 00:31:17,770
592
00:31:17,770 --> 00:31:21,820
Either of these equations tell
you exactly this if you move
593
00:31:21,820 --> 00:31:22,910
terms around.
594
00:31:22,910 --> 00:31:25,450
So these two equations are
not really two equations.
595
00:31:25,450 --> 00:31:27,010
It's just one equation.
596
00:31:27,010 --> 00:31:30,520
They are linearly dependent
equations, and in order to
597
00:31:30,520 --> 00:31:33,320
solve the problem, we need the
additional condition that
598
00:31:33,320 --> 00:31:36,300
pi(1) + pi(2) is equal to 1.
599
00:31:36,300 --> 00:31:38,420
Now, we have our system
of two equations,
600
00:31:38,420 --> 00:31:39,880
which you can solve.
601
00:31:39,880 --> 00:31:45,030
And once you solve it, you find
that pi(1) is 2/7 and
602
00:31:45,030 --> 00:31:48,610
pi(2) is 5/7.
603
00:31:48,610 --> 00:31:52,880
So these are the steady state
probabilities of the two
604
00:31:52,880 --> 00:31:54,130
different states.
605
00:31:54,130 --> 00:31:56,600
606
00:31:56,600 --> 00:32:01,770
If we start this chain, at some
state, let's say state 1,
607
00:32:01,770 --> 00:32:07,050
and we let it run for a long,
long time, the chain settles
608
00:32:07,050 --> 00:32:08,580
into steady state.
609
00:32:08,580 --> 00:32:09,440
What does that mean?
610
00:32:09,440 --> 00:32:12,470
It does not mean that
the state itself
611
00:32:12,470 --> 00:32:13,960
enters steady state.
612
00:32:13,960 --> 00:32:17,390
The state will keep jumping
around forever and ever.
613
00:32:17,390 --> 00:32:21,040
It will keep visiting both
states once in a while.
614
00:32:21,040 --> 00:32:23,250
So the jumping never ceases.
615
00:32:23,250 --> 00:32:25,790
The thing that gets into
steady state is the
616
00:32:25,790 --> 00:32:30,180
probability of finding
yourself at state 1.
617
00:32:30,180 --> 00:32:34,020
So the probability that you find
yourself at state 1 at
618
00:32:34,020 --> 00:32:37,640
time one trillion is
approximately 2/7.
619
00:32:37,640 --> 00:32:40,990
The probability you find
yourself at state 1 at time
620
00:32:40,990 --> 00:32:45,590
two trillions is again,
approximately 2/7.
621
00:32:45,590 --> 00:32:48,640
So the probability of being in
that state settles into a
622
00:32:48,640 --> 00:32:52,270
steady value.
623
00:32:52,270 --> 00:32:55,630
That's what the steady-state
convergence means.
624
00:32:55,630 --> 00:32:58,370
It's convergence of
probabilities, not convergence
625
00:32:58,370 --> 00:33:00,680
of the process itself.
626
00:33:00,680 --> 00:33:04,750
And again, the two main things
that are happening in this
627
00:33:04,750 --> 00:33:07,940
example, and more generally,
when we have a single class
628
00:33:07,940 --> 00:33:10,650
and no periodicity, is
that the initial
629
00:33:10,650 --> 00:33:12,600
state does not matter.
630
00:33:12,600 --> 00:33:15,980
There's enough randomness here
so that no matter where you
631
00:33:15,980 --> 00:33:19,720
start, the randomness kind of
washes out any memory of where
632
00:33:19,720 --> 00:33:20,780
you started.
633
00:33:20,780 --> 00:33:23,270
And also in this example,
clearly, we do not have
634
00:33:23,270 --> 00:33:27,500
periodicity because
we have self arcs.
635
00:33:27,500 --> 00:33:30,960
And this, in particular, implies
that the exact time
636
00:33:30,960 --> 00:33:32,210
does not matter.
637
00:33:32,210 --> 00:33:35,740
638
00:33:35,740 --> 00:33:41,510
So now, we're going to spend
the rest of our time by
639
00:33:41,510 --> 00:33:45,330
looking into a special class
of chains that's a little
640
00:33:45,330 --> 00:33:47,450
easier to deal with,
but still, it's
641
00:33:47,450 --> 00:33:49,436
an important class.
642
00:33:49,436 --> 00:33:52,010
So what's the moral from here?
643
00:33:52,010 --> 00:33:55,680
This was a simple example with
two states, and we could find
644
00:33:55,680 --> 00:33:59,440
the steady-state probabilities
by solving a simple system of
645
00:33:59,440 --> 00:34:01,320
two-by-two equations.
646
00:34:01,320 --> 00:34:05,690
If you have a chain with 100
states, it's no problem for a
647
00:34:05,690 --> 00:34:09,120
computer to solve a system
of 100-by-100 equations.
648
00:34:09,120 --> 00:34:12,800
But you can certainly not do it
by hand, and usually, you
649
00:34:12,800 --> 00:34:15,560
cannot get any closed-form
formulas, so you do not
650
00:34:15,560 --> 00:34:17,969
necessarily get a
lot of insight.
651
00:34:17,969 --> 00:34:21,330
So one looks for special
structures or models that
652
00:34:21,330 --> 00:34:25,610
maybe give you a little more
insight or maybe lead you to
653
00:34:25,610 --> 00:34:27,690
closed-form formulas.
654
00:34:27,690 --> 00:34:32,130
And an interesting subclass of
Markov chains in which all of
655
00:34:32,130 --> 00:34:35,080
these nice things do happen,
is the class
656
00:34:35,080 --> 00:34:39,080
of birth/death processes.
657
00:34:39,080 --> 00:34:41,500
So what's a birth/death
process?
658
00:34:41,500 --> 00:34:44,510
It's a Markov chain who's
diagram looks
659
00:34:44,510 --> 00:34:46,810
basically like this.
660
00:34:46,810 --> 00:34:52,020
So the states of the Markov
chain start from 0 and go up
661
00:34:52,020 --> 00:34:54,460
to some finite integer m.
662
00:34:54,460 --> 00:34:57,400
What's special about this chain
is that if you are at a
663
00:34:57,400 --> 00:35:02,710
certain state, next time you can
either go up by 1, you can
664
00:35:02,710 --> 00:35:06,640
go down by 1, or you
can stay in place.
665
00:35:06,640 --> 00:35:09,820
So it's like keeping track
of some population
666
00:35:09,820 --> 00:35:11,310
at any given time.
667
00:35:11,310 --> 00:35:13,820
One person gets born,
or one person
668
00:35:13,820 --> 00:35:15,680
dies, or nothing happens.
669
00:35:15,680 --> 00:35:19,150
Again, we're not accounting
for twins here.
670
00:35:19,150 --> 00:35:24,430
So we're given this structure,
and we are given the
671
00:35:24,430 --> 00:35:27,660
transition probabilities, the
probabilities associated with
672
00:35:27,660 --> 00:35:29,630
transitions of the
different types.
673
00:35:29,630 --> 00:35:32,880
So we use P's for the upward
transitions, Q's for the
674
00:35:32,880 --> 00:35:34,550
downward transitions.
675
00:35:34,550 --> 00:35:37,830
An example of a chain of this
kind was the supermarket
676
00:35:37,830 --> 00:35:40,410
counter model that we
discussed last time.
677
00:35:40,410 --> 00:35:45,080
That is, a customer arrives,
so this increments
678
00:35:45,080 --> 00:35:46,310
the state by 1.
679
00:35:46,310 --> 00:35:49,670
Or a customer finishes service,
in which case, the
680
00:35:49,670 --> 00:35:53,020
state gets decremented by 1,
or nothing happens in which
681
00:35:53,020 --> 00:35:55,500
you stay in place, and so on.
682
00:35:55,500 --> 00:35:59,880
In the supermarket model, these
P's inside here were all
683
00:35:59,880 --> 00:36:03,780
taken to be equal because we
assume that the arrival rate
684
00:36:03,780 --> 00:36:07,400
was sort of constant
at each time slot.
685
00:36:07,400 --> 00:36:10,960
But you can generalize a little
bit by assuming that
686
00:36:10,960 --> 00:36:15,330
these transition probabilities
P1 here, P2 there, and so on
687
00:36:15,330 --> 00:36:18,190
may be different from
state to state.
688
00:36:18,190 --> 00:36:21,750
So in general, from state
i, there's going to be a
689
00:36:21,750 --> 00:36:24,400
transition probability
Pi that the next
690
00:36:24,400 --> 00:36:26,210
transition is upwards.
691
00:36:26,210 --> 00:36:29,820
And there's going to be a
probability Qi that the next
692
00:36:29,820 --> 00:36:31,910
transition is downwards.
693
00:36:31,910 --> 00:36:35,310
And so from that state, the
probability that the next
694
00:36:35,310 --> 00:36:37,640
transition is downwards is
going to be Q_(i+1).
695
00:36:37,640 --> 00:36:40,930
696
00:36:40,930 --> 00:36:43,650
So this is the structure
of our chain.
697
00:36:43,650 --> 00:36:47,580
As I said, it's a crude model
of what happens at the
698
00:36:47,580 --> 00:36:53,820
supermarket counter but it's
also a good model for lots of
699
00:36:53,820 --> 00:36:55,370
types of service systems.
700
00:36:55,370 --> 00:36:59,400
Again, you have a server
somewhere that has a buffer.
701
00:36:59,400 --> 00:37:00,970
Jobs come into the buffer.
702
00:37:00,970 --> 00:37:02,460
So the buffer builds up.
703
00:37:02,460 --> 00:37:06,880
The server processes jobs, so
the buffer keeps going down.
704
00:37:06,880 --> 00:37:10,470
And the state of the chain would
be the number of jobs
705
00:37:10,470 --> 00:37:12,570
that you have inside
your buffer.
706
00:37:12,570 --> 00:37:18,690
Or you could be thinking about
active phone calls out of a
707
00:37:18,690 --> 00:37:19,680
certain city.
708
00:37:19,680 --> 00:37:22,520
Each time that the phone call
is placed, the number of
709
00:37:22,520 --> 00:37:24,090
active phone calls
goes up by 1.
710
00:37:24,090 --> 00:37:28,360
Each time that the phone call
stops happening, is
711
00:37:28,360 --> 00:37:31,790
terminated, then the count
goes down by 1.
712
00:37:31,790 --> 00:37:34,710
So it's for processes of this
kind that a model with this
713
00:37:34,710 --> 00:37:36,890
structure is going to show up.
714
00:37:36,890 --> 00:37:39,240
And they do show up in
many, many models.
715
00:37:39,240 --> 00:37:43,730
Or you can think about the
number of people in a certain
716
00:37:43,730 --> 00:37:45,690
population that have
a disease.
717
00:37:45,690 --> 00:37:51,010
So 1 more person gets the
flu, the count goes up.
718
00:37:51,010 --> 00:37:55,320
1 more person gets healed,
the count goes down.
719
00:37:55,320 --> 00:37:58,350
And these probabilities in such
an epidemic model would
720
00:37:58,350 --> 00:38:02,270
certainly depend on
the current state.
721
00:38:02,270 --> 00:38:06,280
If lots of people already have
the flu, the probability that
722
00:38:06,280 --> 00:38:10,010
another person catches it
would be pretty high.
723
00:38:10,010 --> 00:38:13,800
Whereas, if no one has the flu,
then the probability that
724
00:38:13,800 --> 00:38:16,820
you get a transition where
someone catches the flu, that
725
00:38:16,820 --> 00:38:18,960
probability would
be pretty small.
726
00:38:18,960 --> 00:38:26,990
So the transition rates, the
incidence of new people who
727
00:38:26,990 --> 00:38:30,010
have the disease definitely
depends on how many people
728
00:38:30,010 --> 00:38:31,560
already have the disease.
729
00:38:31,560 --> 00:38:34,970
And that motivates cases where
those P's, the upward
730
00:38:34,970 --> 00:38:39,220
transition probabilities,
depend on the
731
00:38:39,220 --> 00:38:42,400
state of the chain.
732
00:38:42,400 --> 00:38:44,040
So how do we study this chain?
733
00:38:44,040 --> 00:38:49,220
You can sit down and write the
system of n linear equations
734
00:38:49,220 --> 00:38:50,550
in the pi's.
735
00:38:50,550 --> 00:38:52,920
And this way, find the
steady-state probabilities of
736
00:38:52,920 --> 00:38:53,790
this chain.
737
00:38:53,790 --> 00:38:55,850
But this is a little harder.
738
00:38:55,850 --> 00:38:59,310
It's more work than one
actually needs to do.
739
00:38:59,310 --> 00:39:03,200
There's a very clever shortcut
that applies
740
00:39:03,200 --> 00:39:05,160
to birth/death processes.
741
00:39:05,160 --> 00:39:08,410
And it's based on the frequency
interpretation that
742
00:39:08,410 --> 00:39:10,000
we discussed a little
while ago.
743
00:39:10,000 --> 00:39:14,070
744
00:39:14,070 --> 00:39:17,390
Let's put a line somewhere in
the middle of this chain, and
745
00:39:17,390 --> 00:39:21,710
focus on the relation between
this part and that part in
746
00:39:21,710 --> 00:39:22,750
more detail.
747
00:39:22,750 --> 00:39:25,460
So think of the chain continuing
in this direction,
748
00:39:25,460 --> 00:39:26,390
that direction.
749
00:39:26,390 --> 00:39:30,020
But let's just focus on 2
adjacent states, and look at
750
00:39:30,020 --> 00:39:32,270
this particular cut.
751
00:39:32,270 --> 00:39:34,270
What is the chain going to do?
752
00:39:34,270 --> 00:39:35,550
Let's say it starts here.
753
00:39:35,550 --> 00:39:37,250
It's going to move around.
754
00:39:37,250 --> 00:39:40,280
At some point, it makes a
transition to the other side.
755
00:39:40,280 --> 00:39:42,860
And that's a transition
from i to i+1.
756
00:39:42,860 --> 00:39:45,470
It stays on the other
side for some time.
757
00:39:45,470 --> 00:39:48,490
It gets here, and eventually,
it's going to make a
758
00:39:48,490 --> 00:39:50,340
transition to this side.
759
00:39:50,340 --> 00:39:53,110
Then it keeps moving
and so on.
760
00:39:53,110 --> 00:39:57,680
Now, there's a certain balance
that must be obeyed here.
761
00:39:57,680 --> 00:40:01,300
The number of upward transitions
through this line
762
00:40:01,300 --> 00:40:04,220
cannot be very different from
the number of downward
763
00:40:04,220 --> 00:40:06,080
transitions.
764
00:40:06,080 --> 00:40:09,530
Because we cross this
way, then next time,
765
00:40:09,530 --> 00:40:10,630
we'll cross that way.
766
00:40:10,630 --> 00:40:12,580
Then next time, we'll
cross this way.
767
00:40:12,580 --> 00:40:13,970
We'll cross that way.
768
00:40:13,970 --> 00:40:18,940
So the frequency with which
transitions of this kind occur
769
00:40:18,940 --> 00:40:21,500
has to be the same as the
long-term frequency that
770
00:40:21,500 --> 00:40:24,430
transitions of that
kind occur.
771
00:40:24,430 --> 00:40:28,510
You cannot go up 100 times and
go down only 50 times.
772
00:40:28,510 --> 00:40:31,740
If you have gone up 100 times,
it means that you have gone
773
00:40:31,740 --> 00:40:36,960
down 99, or 100, or 101,
but nothing much more
774
00:40:36,960 --> 00:40:38,570
different than that.
775
00:40:38,570 --> 00:40:41,440
So the frequency with
which transitions of
776
00:40:41,440 --> 00:40:43,670
this kind get observed.
777
00:40:43,670 --> 00:40:47,610
That is, out of a large number
of transitions, what fraction
778
00:40:47,610 --> 00:40:49,890
of transitions are
of these kind?
779
00:40:49,890 --> 00:40:52,160
That fraction has to be the
same as the fraction of
780
00:40:52,160 --> 00:40:54,870
transitions that happened
to be of that kind.
781
00:40:54,870 --> 00:40:56,620
What are these fractions?
782
00:40:56,620 --> 00:40:58,480
We discussed that before.
783
00:40:58,480 --> 00:41:04,840
The fraction of times at which
transitions of this kind are
784
00:41:04,840 --> 00:41:08,340
observed is the fraction of time
that we happen to be at
785
00:41:08,340 --> 00:41:09,350
that state.
786
00:41:09,350 --> 00:41:11,790
And out of the times that we
are in that state, the
787
00:41:11,790 --> 00:41:15,010
fraction of transitions that
happen to be upward
788
00:41:15,010 --> 00:41:16,040
transitions.
789
00:41:16,040 --> 00:41:20,820
So this is the frequency with
which transitions of this kind
790
00:41:20,820 --> 00:41:22,430
are observed.
791
00:41:22,430 --> 00:41:25,350
And with the same argument,
this is the frequency with
792
00:41:25,350 --> 00:41:28,670
which transitions of that
kind are observed.
793
00:41:28,670 --> 00:41:31,120
Since these two frequencies
are the same, these two
794
00:41:31,120 --> 00:41:34,390
numbers must be the same, and
we get an equation that
795
00:41:34,390 --> 00:41:38,350
relates the Pi to P_(i+1).
796
00:41:38,350 --> 00:41:43,040
This has a nice form because
it gives us a recursion.
797
00:41:43,040 --> 00:41:45,760
If we knew pi(i), we could then
798
00:41:45,760 --> 00:41:48,860
immediately calculate pi(i+1).
799
00:41:48,860 --> 00:41:51,860
So it's a system of equations
that's very
800
00:41:51,860 --> 00:41:54,860
easy to solve almost.
801
00:41:54,860 --> 00:41:57,320
But how do we get started?
802
00:41:57,320 --> 00:42:01,850
If I knew pi(0), I could find
by pi(1) and then use this
803
00:42:01,850 --> 00:42:05,330
recursion to find pi(2),
pi(3), and so on.
804
00:42:05,330 --> 00:42:06,970
But we don't know pi(0).
805
00:42:06,970 --> 00:42:09,920
It's one more unknown.
806
00:42:09,920 --> 00:42:14,290
It's an unknown, and we need
to actually use the extra
807
00:42:14,290 --> 00:42:20,000
normalization condition that
the sum of the pi's is 1.
808
00:42:20,000 --> 00:42:23,810
And after we use that
normalization condition, then
809
00:42:23,810 --> 00:42:26,580
we can find all of the pi's.
810
00:42:26,580 --> 00:42:32,550
811
00:42:32,550 --> 00:42:38,450
So you basically fix pi(0) as a
symbol, solve this equation
812
00:42:38,450 --> 00:42:41,580
symbolically, and
everything gets
813
00:42:41,580 --> 00:42:43,830
expressed in terms of pi(0).
814
00:42:43,830 --> 00:42:46,510
And then use that normalization
condition to
815
00:42:46,510 --> 00:42:48,700
find pi(0), and you're done.
816
00:42:48,700 --> 00:42:51,570
Let's illustrate the details
of this procedure on a
817
00:42:51,570 --> 00:42:53,690
particular special case.
818
00:42:53,690 --> 00:42:57,360
So in our special case, we're
going to simplify things now
819
00:42:57,360 --> 00:43:01,760
by assuming that all those
upward P's are the same, and
820
00:43:01,760 --> 00:43:05,780
all of those downward
Q's are the same.
821
00:43:05,780 --> 00:43:08,860
So at each point in time, if
you're sitting somewhere in
822
00:43:08,860 --> 00:43:13,060
the middle, you have probability
P of moving up and
823
00:43:13,060 --> 00:43:16,710
probability Q of moving down.
824
00:43:16,710 --> 00:43:23,460
This rho, the ratio of P/Q is
frequency of going up versus
825
00:43:23,460 --> 00:43:25,670
frequency of going down.
826
00:43:25,670 --> 00:43:29,070
If it's a service system, you
can think of it as a measure
827
00:43:29,070 --> 00:43:32,140
of how loaded the system is.
828
00:43:32,140 --> 00:43:39,100
If P is equal to Q, it's means
that if you're at this state,
829
00:43:39,100 --> 00:43:42,700
you're equally likely to move
left or right, so the system
830
00:43:42,700 --> 00:43:44,820
is kind of balanced.
831
00:43:44,820 --> 00:43:46,980
The state doesn't have a
tendency to move in this
832
00:43:46,980 --> 00:43:49,450
direction or in that
direction.
833
00:43:49,450 --> 00:43:53,860
If rho is bigger than 1 so that
P is bigger than Q, it
834
00:43:53,860 --> 00:43:56,730
means that whenever I'm at some
state in the middle, I'm
835
00:43:56,730 --> 00:44:00,830
more likely to move right rather
than move left, which
836
00:44:00,830 --> 00:44:04,230
means that my state, of course
it's random, but it has a
837
00:44:04,230 --> 00:44:07,170
tendency to move in
that direction.
838
00:44:07,170 --> 00:44:10,280
And if you think of this as a
number of customers in queue,
839
00:44:10,280 --> 00:44:13,750
it means your system has the
tendency to become loaded and
840
00:44:13,750 --> 00:44:15,420
to build up a queue.
841
00:44:15,420 --> 00:44:19,790
So rho being bigger than 1
corresponds to a heavy load,
842
00:44:19,790 --> 00:44:21,570
where queues build up.
843
00:44:21,570 --> 00:44:25,470
Rho less than 1 corresponds to
the system where queues have
844
00:44:25,470 --> 00:44:27,310
the tendency to drain down.
845
00:44:27,310 --> 00:44:30,880
846
00:44:30,880 --> 00:44:32,680
Now, let's write down
the equations.
847
00:44:32,680 --> 00:44:40,540
We have this recursion P_(i+1)
is Pi times Pi over Qi.
848
00:44:40,540 --> 00:44:44,040
In our case here, the P's and
the Q's do not depend on the
849
00:44:44,040 --> 00:44:47,190
particular index, so we
get this relation.
850
00:44:47,190 --> 00:44:51,830
And this P over Q is just
the load factor rho.
851
00:44:51,830 --> 00:44:54,790
Once you look at this equation,
clearly you realize
852
00:44:54,790 --> 00:44:58,440
that by pi(1) is rho
times pi(0).
853
00:44:58,440 --> 00:45:02,286
pi(2) is going to be --
854
00:45:02,286 --> 00:45:04,480
So we'll do it in detail.
855
00:45:04,480 --> 00:45:08,130
So pi(1) is pi(0) times rho.
856
00:45:08,130 --> 00:45:15,580
pi(2) is pi(1) times rho,
which is pi(0) times
857
00:45:15,580 --> 00:45:18,350
rho-squared.
858
00:45:18,350 --> 00:45:21,340
And then you continue doing
this calculation.
859
00:45:21,340 --> 00:45:25,530
And you find that you can
express every pi(i) in terms
860
00:45:25,530 --> 00:45:31,490
of pi(0) and you get this
factor of rho^i.
861
00:45:31,490 --> 00:45:34,840
And then you use the last
equation that we have -- that
862
00:45:34,840 --> 00:45:38,110
the sum of the probabilities
has to be equal to 1.
863
00:45:38,110 --> 00:45:41,670
And that equation is going to
tell us that the sum over all
864
00:45:41,670 --> 00:45:50,890
i's from 0 to m of pi(0) rho
to the i is equal to 1.
865
00:45:50,890 --> 00:45:58,730
And therefore, pi(0) is 1 over
(the sum over the rho to the i
866
00:45:58,730 --> 00:46:03,100
for i going from 0 to m).
867
00:46:03,100 --> 00:46:09,870
So now we found pi(0), and by
plugging in this expression,
868
00:46:09,870 --> 00:46:12,680
we have the steady-state
probabilities of all of the
869
00:46:12,680 --> 00:46:14,950
different states.
870
00:46:14,950 --> 00:46:18,990
Let's look at some special
cases of this.
871
00:46:18,990 --> 00:46:24,390
Suppose that rho
is equal to 1.
872
00:46:24,390 --> 00:46:30,410
If rho is equal to 1, then
pi(i) is equal to pi(0).
873
00:46:30,410 --> 00:46:33,250
It means that all
the steady-state
874
00:46:33,250 --> 00:46:35,840
probabilities are equal.
875
00:46:35,840 --> 00:46:39,380
It's means that every
state is equally
876
00:46:39,380 --> 00:46:42,750
likely in the long run.
877
00:46:42,750 --> 00:46:44,790
So this is an example.
878
00:46:44,790 --> 00:46:48,730
It's called a symmetric
random walk.
879
00:46:48,730 --> 00:46:53,260
It's a very popular model for
modeling people who are drunk.
880
00:46:53,260 --> 00:46:56,730
So you start at a state
at any point in time.
881
00:46:56,730 --> 00:47:00,140
Either you stay in place, or you
have an equal probability
882
00:47:00,140 --> 00:47:02,910
of going left or going right.
883
00:47:02,910 --> 00:47:06,220
There's no bias in
either direction.
884
00:47:06,220 --> 00:47:10,710
You might think that in such a
process, you will tend to kind
885
00:47:10,710 --> 00:47:14,750
of get stuck near one end
or the other end.
886
00:47:14,750 --> 00:47:17,320
Well, it's not really clear
what to expect.
887
00:47:17,320 --> 00:47:21,260
It turns out that in such a
model, in the long run, the
888
00:47:21,260 --> 00:47:24,610
drunk person is equally
likely to be at any
889
00:47:24,610 --> 00:47:26,430
one of those states.
890
00:47:26,430 --> 00:47:31,300
The steady-state probability is
the same for all i's if rho
891
00:47:31,300 --> 00:47:33,880
is equal to 1.
892
00:47:33,880 --> 00:47:39,570
And so if you show up at a
random time, and you ask where
893
00:47:39,570 --> 00:47:43,670
is my state, you will be told
it's equally likely to be at
894
00:47:43,670 --> 00:47:46,600
any one of those places.
895
00:47:46,600 --> 00:47:48,370
So let's make that note.
896
00:47:48,370 --> 00:47:51,980
If rho equal to 1, implies
that all the
897
00:47:51,980 --> 00:47:53,660
pi(i)'s are 1/(M+1) --
898
00:47:53,660 --> 00:47:57,040
899
00:47:57,040 --> 00:48:01,320
M+1 because that's how many
states we have in our model.
900
00:48:01,320 --> 00:48:04,210
Now, let's look at
a different case.
901
00:48:04,210 --> 00:48:08,630
Suppose that M is
a huge number.
902
00:48:08,630 --> 00:48:13,600
So essentially, our supermarket
has a very large
903
00:48:13,600 --> 00:48:19,600
space, a lot of space to
store their customers.
904
00:48:19,600 --> 00:48:24,130
But suppose that the system
is on the stable side.
905
00:48:24,130 --> 00:48:27,900
P is less than Q, which means
that there's a tendency for
906
00:48:27,900 --> 00:48:31,420
customers to be served faster
than they arrive.
907
00:48:31,420 --> 00:48:35,500
The drift in this chain, it
tends to be in that direction.
908
00:48:35,500 --> 00:48:41,920
So when rho is less than 1,
which is this case, and when M
909
00:48:41,920 --> 00:48:45,810
is going to infinity, this
infinite sum is the sum of a
910
00:48:45,810 --> 00:48:47,690
geometric series.
911
00:48:47,690 --> 00:48:50,480
And you recognize it
(hopefully) --
912
00:48:50,480 --> 00:48:53,480
913
00:48:53,480 --> 00:48:56,520
this series is going
to 1/(1-rho).
914
00:48:56,520 --> 00:49:00,280
And because it's in the
denominator, pi(0) ends up
915
00:49:00,280 --> 00:49:02,970
being 1-rho.
916
00:49:02,970 --> 00:49:06,230
So by taking the limit as M
goes to infinity, in this
917
00:49:06,230 --> 00:49:09,460
case, and when rho is less than
1 so that this series is
918
00:49:09,460 --> 00:49:12,220
convergent, we get
this formula.
919
00:49:12,220 --> 00:49:15,780
So we get the closed-form
formula for the pi(i)'s.
920
00:49:15,780 --> 00:49:19,840
In particular, pi(i) is (1-
rho)(rho to the i).
921
00:49:19,840 --> 00:49:21,150
to
922
00:49:21,150 --> 00:49:24,520
So these pi(i)'s are essentially
a probability
923
00:49:24,520 --> 00:49:26,200
distribution.
924
00:49:26,200 --> 00:49:32,100
They tell us if we show up at
time 1 billion and we ask,
925
00:49:32,100 --> 00:49:33,900
where is my state?
926
00:49:33,900 --> 00:49:37,500
You will be told that
the state is 0.
927
00:49:37,500 --> 00:49:41,040
Your system is empty with
probability 1-rho, minus or
928
00:49:41,040 --> 00:49:44,410
there's one customer in the
system, and that happens with
929
00:49:44,410 --> 00:49:46,460
probability (rho
- 1) times rho.
930
00:49:46,460 --> 00:49:49,950
And it keeps going
down this way.
931
00:49:49,950 --> 00:49:53,920
And it's pretty much a geometric
distribution except
932
00:49:53,920 --> 00:49:58,130
that it has shifted so that it
starts at 0 whereas the usual
933
00:49:58,130 --> 00:50:00,890
geometric distribution
starts at 1.
934
00:50:00,890 --> 00:50:04,670
So this is a mini introduction
into queuing theory.
935
00:50:04,670 --> 00:50:08,730
This is the first and simplest
model that one encounters when
936
00:50:08,730 --> 00:50:10,850
you start studying
queuing theory.
937
00:50:10,850 --> 00:50:13,360
This is clearly a model of a
queueing phenomenon such as
938
00:50:13,360 --> 00:50:16,450
the supermarket counter with
the P's corresponding to
939
00:50:16,450 --> 00:50:19,280
arrivals, the Q's corresponding
to departures.
940
00:50:19,280 --> 00:50:22,690
And this particular queuing
system when M is very, very
941
00:50:22,690 --> 00:50:26,670
large and rho is less than 1,
has a very simple and nice
942
00:50:26,670 --> 00:50:28,880
solution in closed form.
943
00:50:28,880 --> 00:50:31,650
And that's why it's
very much liked.
944
00:50:31,650 --> 00:50:33,610
And let me just take
two seconds to
945
00:50:33,610 --> 00:50:38,030
draw one last picture.
946
00:50:38,030 --> 00:50:40,300
So this is the probability
of the different i's.
947
00:50:40,300 --> 00:50:41,420
It gives you a PMF.
948
00:50:41,420 --> 00:50:43,890
This PMF has an expected
value.
949
00:50:43,890 --> 00:50:47,090
And the expectation, the
expected number of customers
950
00:50:47,090 --> 00:50:50,560
in the system, is given
by this formula.
951
00:50:50,560 --> 00:50:54,230
And this formula, which is
interesting to anyone who
952
00:50:54,230 --> 00:50:57,070
tries to analyze a system
of this kind,
953
00:50:57,070 --> 00:50:58,430
tells you the following.
954
00:50:58,430 --> 00:51:03,970
That as long as a rho is less
than 1, then the expected
955
00:51:03,970 --> 00:51:07,270
number of customers in
the system is finite.
956
00:51:07,270 --> 00:51:09,970
But if rho becomes very
close to 1 --
957
00:51:09,970 --> 00:51:13,790
So if your load factor is
something like .99, you expect
958
00:51:13,790 --> 00:51:17,630
to have a large number of
customers in the system at any
959
00:51:17,630 --> 00:51:19,040
given time.
960
00:51:19,040 --> 00:51:20,440
OK.
961
00:51:20,440 --> 00:51:22,420
All right.
962
00:51:22,420 --> 00:51:23,270
Have a good weekend.
963
00:51:23,270 --> 00:51:25,390
We'll continue next time.