1
00:00:00,880 --> 00:00:03,330
So let us first review
the steady state
2
00:00:03,330 --> 00:00:05,670
behavior of Markov chains.
3
00:00:05,670 --> 00:00:07,940
Consider again the
following example.
4
00:00:07,940 --> 00:00:11,430
This chain has some recurrent
states, some transient states,
5
00:00:11,430 --> 00:00:13,660
and a single recurrent class.
6
00:00:13,660 --> 00:00:18,640
So for example,
state 9 is recurrent.
7
00:00:18,640 --> 00:00:20,090
State 3 is recurrent.
8
00:00:20,090 --> 00:00:21,990
State 5 is recurrent.
9
00:00:21,990 --> 00:00:23,390
And why are they recurrent?
10
00:00:23,390 --> 00:00:27,780
Because whenever you are in
9, no matter where you go,
11
00:00:27,780 --> 00:00:30,460
there's always a
way to come back.
12
00:00:30,460 --> 00:00:35,620
You can go to 3 and come
back, go 5 and come back.
13
00:00:35,620 --> 00:00:37,555
And actually, this
is a recurrent class.
14
00:00:41,220 --> 00:00:44,030
And this is a recurrent
class, because all
15
00:00:44,030 --> 00:00:47,670
these recurrent states
communicate between each other.
16
00:00:47,670 --> 00:00:50,160
What about the other states--
well they are not recurrent.
17
00:00:50,160 --> 00:00:53,660
So for example, state 1--
and once you are here,
18
00:00:53,660 --> 00:00:55,600
there is a possibility
that you go there,
19
00:00:55,600 --> 00:00:56,810
and you will never come back.
20
00:00:56,810 --> 00:00:57,934
So it can not be recurrent.
21
00:00:57,934 --> 00:00:59,360
So it's transient.
22
00:00:59,360 --> 00:01:00,000
What about 4?
23
00:01:00,000 --> 00:01:03,050
4 also has the possibility
at one point to go there.
24
00:01:03,050 --> 00:01:05,060
And then from there, it
will never come back.
25
00:01:05,060 --> 00:01:07,160
So 4 is also transient.
26
00:01:07,160 --> 00:01:10,310
As for 2, no matter
where it goes, well,
27
00:01:10,310 --> 00:01:13,030
it's going to reach or
touch a transient state.
28
00:01:13,030 --> 00:01:14,960
So by definition, it
will be also transient.
29
00:01:14,960 --> 00:01:17,320
So they have three
transient states
30
00:01:17,320 --> 00:01:19,680
and three recurrent states.
31
00:01:19,680 --> 00:01:23,610
Also, this recurrent
class is not periodic.
32
00:01:23,610 --> 00:01:26,780
So it's aperiodic.
33
00:01:26,780 --> 00:01:29,480
And why is it not periodic?
34
00:01:29,480 --> 00:01:32,940
Well, here, there is
a simple way to tell.
35
00:01:32,940 --> 00:01:36,610
We have the existence of a
self-transition probability.
36
00:01:36,610 --> 00:01:40,490
And that's enough to show that
this recurrent class is not
37
00:01:40,490 --> 00:01:41,810
periodic.
38
00:01:41,810 --> 00:01:44,690
So this is one of the
nicest possible Markov
39
00:01:44,690 --> 00:01:49,900
chains in the sense that they
have the following property--
40
00:01:49,900 --> 00:01:51,700
the probability that
you find yourself
41
00:01:51,700 --> 00:01:55,050
at some particular state j.
42
00:01:55,050 --> 00:01:57,580
At time n, when n
is very large, it
43
00:01:57,580 --> 00:01:59,310
converges to a
steady-state value
44
00:01:59,310 --> 00:02:03,510
that we denote by pi of j.
45
00:02:03,510 --> 00:02:06,410
There are two aspects
to this property.
46
00:02:06,410 --> 00:02:10,320
First, the limit exists, so
the probability of state j
47
00:02:10,320 --> 00:02:11,830
does not fluctuate.
48
00:02:11,830 --> 00:02:15,380
It settles to something
in the long run.
49
00:02:15,380 --> 00:02:17,110
And furthermore,
that probability
50
00:02:17,110 --> 00:02:21,170
is not affected by i.
51
00:02:21,170 --> 00:02:24,500
Now, if we don't know where
the chain started, and we
52
00:02:24,500 --> 00:02:28,010
want to know the unconditional
probability of being in state j
53
00:02:28,010 --> 00:02:35,120
in the long run,
when n is large,
54
00:02:35,120 --> 00:02:38,180
then either we are given
an initial distribution
55
00:02:38,180 --> 00:02:41,310
over the states,
or we can choose
56
00:02:41,310 --> 00:02:42,610
any initial distribution.
57
00:02:42,610 --> 00:02:45,430
For example, we can assume that
all initial states are equally
58
00:02:45,430 --> 00:02:49,650
likely-- or any other
type of distributions.
59
00:02:49,650 --> 00:02:52,660
And then you can condition
over all the initial states,
60
00:02:52,660 --> 00:02:54,590
use the total
probability theorem,
61
00:02:54,590 --> 00:02:57,970
and you're going to get the same
answer, pi of j, in the limit.
62
00:02:57,970 --> 00:02:59,870
Let's see how to do that.
63
00:02:59,870 --> 00:03:02,892
So this is the
summation of all i.
64
00:03:02,892 --> 00:03:05,890
So you condition
on that state i.
65
00:03:05,890 --> 00:03:11,340
So it's Rij of n times
the initial probability
66
00:03:11,340 --> 00:03:15,040
distribution of your choosing.
67
00:03:15,040 --> 00:03:17,079
So this is the total
probability theorem.
68
00:03:17,079 --> 00:03:20,360
Now, in the limits,
when n goes to infinity,
69
00:03:20,360 --> 00:03:25,050
this goes to pi of
j, independent of i.
70
00:03:25,050 --> 00:03:27,290
So you can take this
expression, the limit,
71
00:03:27,290 --> 00:03:30,030
and take it out
of the summation.
72
00:03:30,030 --> 00:03:33,730
And then you have the summation
of-- probability of x0
73
00:03:33,730 --> 00:03:35,470
equals 1.
74
00:03:35,470 --> 00:03:38,000
These are probabilities,
so they sum to 1.
75
00:03:41,470 --> 00:03:47,850
So in the end, you have
that converges to pi of j.
76
00:03:47,850 --> 00:03:53,460
So the conditional probability,
given the initial state,
77
00:03:53,460 --> 00:03:58,000
is in the limit, the same as
the unconditional probability
78
00:03:58,000 --> 00:03:59,820
when n is large.
79
00:03:59,820 --> 00:04:06,030
And in that sense, it tells
us that x of n and x of 0
80
00:04:06,030 --> 00:04:08,570
are approximately independent.
81
00:04:08,570 --> 00:04:13,180
They become independent in the
limit as n goes to infinity.
82
00:04:13,180 --> 00:04:16,620
So if the Markov chain
has run for a long time,
83
00:04:16,620 --> 00:04:20,610
and you are asked the question,
"Where is the chain now,"
84
00:04:20,610 --> 00:04:23,060
then your answer should
be, I don't know.
85
00:04:23,060 --> 00:04:24,090
It's random.
86
00:04:24,090 --> 00:04:27,670
But it's going to
be in a particular j
87
00:04:27,670 --> 00:04:31,590
with that particular
probability, pi of j.
88
00:04:31,590 --> 00:04:33,700
So in that sense, the
steady-state probabilities
89
00:04:33,700 --> 00:04:36,420
are valuable information.
90
00:04:36,420 --> 00:04:38,730
So how do we compute them?
91
00:04:38,730 --> 00:04:43,290
Well, for transient states,
like these, they are 0.
92
00:04:43,290 --> 00:04:45,520
So pi of 1 is 0.
93
00:04:45,520 --> 00:04:47,320
Pi of 2 is 0.
94
00:04:47,320 --> 00:04:50,820
And pi of 4 is 0.
95
00:04:50,820 --> 00:04:52,050
And why is that?
96
00:04:52,050 --> 00:04:55,850
Well, if your initial state
were one of the states here,
97
00:04:55,850 --> 00:04:59,159
the probability of being in
here is 0, no matter what.
98
00:04:59,159 --> 00:05:01,750
But even if you
started here initially,
99
00:05:01,750 --> 00:05:04,180
in one of these states,
you might, for a while,
100
00:05:04,180 --> 00:05:06,960
fluctuate and turn
around like that.
101
00:05:06,960 --> 00:05:10,000
But eventually, after a
finite amount of time,
102
00:05:10,000 --> 00:05:14,170
you will go to that class
and never come back to 1.
103
00:05:14,170 --> 00:05:16,020
So in the long run,
the probability
104
00:05:16,020 --> 00:05:20,110
of finding yourself
in state 1 will be 0.
105
00:05:20,110 --> 00:05:23,100
And this is the
same for 2 and 4.
106
00:05:23,100 --> 00:05:26,050
Now, how do we calculate these?
107
00:05:26,050 --> 00:05:29,400
Well, for these states
in the recurrent class,
108
00:05:29,400 --> 00:05:33,210
we compute them by solving a
linear system of equations,
109
00:05:33,210 --> 00:05:36,990
which are called the
balance equation-- these--
110
00:05:36,990 --> 00:05:39,450
together with an
extra condition.
111
00:05:39,450 --> 00:05:42,700
The normalization
equation here has
112
00:05:42,700 --> 00:05:45,560
to be satisfied, because
these are probabilities,
113
00:05:45,560 --> 00:05:48,420
and they have to sum up to 1.
114
00:05:48,420 --> 00:05:52,800
And we have seen that the
system of m plus 1 equation
115
00:05:52,800 --> 00:05:56,120
provides a unique solution
to this kind of system
116
00:05:56,120 --> 00:05:57,510
for the recurrent class.
117
00:05:57,510 --> 00:06:00,330
So you would apply that
to that recurrent class.
118
00:06:00,330 --> 00:06:03,520
And in that example,
you have three states,
119
00:06:03,520 --> 00:06:07,440
so you would choose m
equals 3 for that example.
120
00:06:07,440 --> 00:06:10,930
And you would solve the
system to get the pi j.
121
00:06:10,930 --> 00:06:15,720
Now, what if we had
multiple recurrent classes?
122
00:06:15,720 --> 00:06:17,520
Consider this chain.
123
00:06:17,520 --> 00:06:20,200
It is an expanded version
of the previous one
124
00:06:20,200 --> 00:06:21,910
with additional states.
125
00:06:21,910 --> 00:06:25,550
Some of these are recurrent,
and one is transient.
126
00:06:25,550 --> 00:06:29,640
But now we have two
recurrent classes.
127
00:06:29,640 --> 00:06:34,650
And that was our 1
class, so class 1.
128
00:06:34,650 --> 00:06:42,180
And now we have a second
recurrent class, class 2.
129
00:06:42,180 --> 00:06:44,370
So what happens in
the long run, when
130
00:06:44,370 --> 00:06:46,320
you have situations like that?
131
00:06:46,320 --> 00:06:50,030
Well, in the long run,
if you start from here,
132
00:06:50,030 --> 00:06:52,780
you're going stay here.
133
00:06:52,780 --> 00:06:55,620
And in some sense, the study
of that recurrent class
134
00:06:55,620 --> 00:06:58,150
is the same as the study
of that recurrent class.
135
00:06:58,150 --> 00:07:00,950
And in order to find the
steady-state probabilities
136
00:07:00,950 --> 00:07:04,360
of these states, assuming that
you started in one of these,
137
00:07:04,360 --> 00:07:06,380
will be exactly
the same as before.
138
00:07:06,380 --> 00:07:13,030
So you will use the same
system, with m equals 3 here.
139
00:07:13,030 --> 00:07:15,670
Now, if you had
started here instead,
140
00:07:15,670 --> 00:07:17,830
again, this is a
recurrent class,
141
00:07:17,830 --> 00:07:20,670
and you have m
equals 2 states here.
142
00:07:20,670 --> 00:07:23,690
And in order to find what is
the steady-state probabilities
143
00:07:23,690 --> 00:07:27,960
of these two states, you could
use the same kind of result
144
00:07:27,960 --> 00:07:32,130
here, but you apply it with
m equals 2 in isolation.
145
00:07:32,130 --> 00:07:35,090
So you just concentrate on that.
146
00:07:35,090 --> 00:07:38,560
If, on the other hand, your
Markov chain started from here,
147
00:07:38,560 --> 00:07:40,880
for example, for that
specific example,
148
00:07:40,880 --> 00:07:43,630
you're guaranteed that the next
transition you'll end up here.
149
00:07:43,630 --> 00:07:46,040
And then you can do the
same thing as before.
150
00:07:46,040 --> 00:07:49,850
We still know that the
steady-state probability of 8
151
00:07:49,850 --> 00:07:52,650
will be 0 and 0 and 0 and 0.
152
00:07:52,650 --> 00:07:55,750
Now, what would happen
if you started from here,
153
00:07:55,750 --> 00:07:57,590
from one of these states?
154
00:07:57,590 --> 00:08:00,280
Well again, for a
while, you might
155
00:08:00,280 --> 00:08:02,570
travel throughout
this system here.
156
00:08:02,570 --> 00:08:05,230
But eventually, you're going
to move away from that.
157
00:08:05,230 --> 00:08:08,100
And you will either go
through a transition going
158
00:08:08,100 --> 00:08:12,180
into that recurrent
class via this transition
159
00:08:12,180 --> 00:08:13,460
or via this transition.
160
00:08:13,460 --> 00:08:16,910
And once you're in there,
essentially, the chain
161
00:08:16,910 --> 00:08:18,260
will remain there.
162
00:08:18,260 --> 00:08:20,580
And so you do the same
calculation as before.
163
00:08:20,580 --> 00:08:24,850
And if, on the other hand, you
transition away from that class
164
00:08:24,850 --> 00:08:28,270
and arrived in this
recurrent class,
165
00:08:28,270 --> 00:08:31,800
then you would apply the
result that you had here.
166
00:08:31,800 --> 00:08:36,740
So in some sense, conditional
on the fact that you left
167
00:08:36,740 --> 00:08:38,480
the states and you
arrived there--
168
00:08:38,480 --> 00:08:42,179
in that conditional world, you
can isolate yourself and really
169
00:08:42,179 --> 00:08:44,840
solve the problem for
that class-- and the same
170
00:08:44,840 --> 00:08:46,190
from that class.
171
00:08:46,190 --> 00:08:50,580
Now, of course, this raises the
question, if I start from here,
172
00:08:50,580 --> 00:08:55,390
how do I know whether I'm
going to get here or here?
173
00:08:55,390 --> 00:08:57,090
Well, we don't know.
174
00:08:57,090 --> 00:08:58,430
It's random.
175
00:08:58,430 --> 00:09:01,160
So we will be interested in
calculating the probabilities
176
00:09:01,160 --> 00:09:05,840
that eventually you end
up here versus here.
177
00:09:05,840 --> 00:09:07,380
And this is
something that we are
178
00:09:07,380 --> 00:09:11,158
going to do towards the
end of today's lecture.