1
00:00:00,000 --> 00:00:00,040
2
00:00:00,040 --> 00:00:02,460
The following content is
provided under a Creative
3
00:00:02,460 --> 00:00:03,870
Commons license.
4
00:00:03,870 --> 00:00:06,910
Your support will help MIT
OpenCourseWare continue to
5
00:00:06,910 --> 00:00:10,560
offer high quality educational
resources for free.
6
00:00:10,560 --> 00:00:13,460
To make a donation or view
additional materials from
7
00:00:13,460 --> 00:00:19,290
hundreds of MIT courses, visit
MIT OpenCourseWare at
8
00:00:19,290 --> 00:00:20,540
ocw.mit.edu.
9
00:00:20,540 --> 00:00:22,340
10
00:00:22,340 --> 00:00:25,020
PROFESSOR: So we're going to
start now with a new chapter.
11
00:00:25,020 --> 00:00:28,720
We're going to talk about
Markov processes.
12
00:00:28,720 --> 00:00:32,390
The good news is that this is
a subject that is a lot more
13
00:00:32,390 --> 00:00:37,040
intuitive and simple in many
ways than, let's say, the
14
00:00:37,040 --> 00:00:38,410
Poisson processes.
15
00:00:38,410 --> 00:00:40,590
So hopefully this will
be enjoyable.
16
00:00:40,590 --> 00:00:42,680
So Markov processes
is, a general
17
00:00:42,680 --> 00:00:46,420
class of random processes.
18
00:00:46,420 --> 00:00:49,610
In some sense, it's more
elaborate than the Bernoulli
19
00:00:49,610 --> 00:00:52,780
and Poisson processes, because
now we're going to have
20
00:00:52,780 --> 00:00:55,820
dependencies between difference
times, instead of
21
00:00:55,820 --> 00:00:57,960
having memoryless processes.
22
00:00:57,960 --> 00:01:00,100
So the basic idea is
the following.
23
00:01:00,100 --> 00:01:05,060
In physics, for example, you
write down equations for how a
24
00:01:05,060 --> 00:01:08,240
system evolves that has
the general form.
25
00:01:08,240 --> 00:01:11,780
The new state of a system one
second later is some function
26
00:01:11,780 --> 00:01:14,970
of old state.
27
00:01:14,970 --> 00:01:20,020
So Newton's equations and all
that in physics allow you to
28
00:01:20,020 --> 00:01:21,490
write equations of this kind.
29
00:01:21,490 --> 00:01:26,260
And so if that a particle is
moving at a certain velocity
30
00:01:26,260 --> 00:01:28,790
and it's at some location, you
can predict when it's going to
31
00:01:28,790 --> 00:01:30,510
be a little later.
32
00:01:30,510 --> 00:01:34,760
Markov processes have the same
flavor, except that there's
33
00:01:34,760 --> 00:01:38,810
also some randomness thrown
inside the equation.
34
00:01:38,810 --> 00:01:42,720
So that's what Markov process
essentially is.
35
00:01:42,720 --> 00:01:47,820
It describes the evolution of
the system, or some variables,
36
00:01:47,820 --> 00:01:51,450
but in the presence of some
noise so that the motion
37
00:01:51,450 --> 00:01:55,110
itself is a bit random.
38
00:01:55,110 --> 00:01:58,880
So this is a pretty
general framework.
39
00:01:58,880 --> 00:02:02,850
So pretty much any useful or
interesting random process
40
00:02:02,850 --> 00:02:06,280
that you can think about, you
can always described it as a
41
00:02:06,280 --> 00:02:09,050
Markov process if you
define properly the
42
00:02:09,050 --> 00:02:10,720
notion of the state.
43
00:02:10,720 --> 00:02:13,960
So what we're going to do is
we're going to introduce the
44
00:02:13,960 --> 00:02:17,630
class of Markov processes by,
example, by talking about the
45
00:02:17,630 --> 00:02:19,900
checkout counter in
a supermarket.
46
00:02:19,900 --> 00:02:22,650
Then we're going to abstract
from our example so that we
47
00:02:22,650 --> 00:02:25,580
get a more general definition.
48
00:02:25,580 --> 00:02:28,410
And then we're going to do a
few things, such as how to
49
00:02:28,410 --> 00:02:32,250
predict what's going to happen
n time steps later, if we
50
00:02:32,250 --> 00:02:34,210
start at the particular state.
51
00:02:34,210 --> 00:02:36,460
And then talk a little bit
about some structural
52
00:02:36,460 --> 00:02:40,640
properties of Markov processes
or Markov chains.
53
00:02:40,640 --> 00:02:42,050
So here's our example.
54
00:02:42,050 --> 00:02:44,770
55
00:02:44,770 --> 00:02:49,670
You go to the checkout counter
at the supermarket, and you
56
00:02:49,670 --> 00:02:54,460
stand there and watch the
customers who come.
57
00:02:54,460 --> 00:02:59,090
So customers come, they get in
queue, and customers get
58
00:02:59,090 --> 00:03:01,250
served one at a time.
59
00:03:01,250 --> 00:03:03,730
So the discussion is going to
be in terms of supermarket
60
00:03:03,730 --> 00:03:06,950
checkout counters, but the
same story applies to any
61
00:03:06,950 --> 00:03:08,420
service system.
62
00:03:08,420 --> 00:03:12,470
You may have a server, jobs
arrive to that server, they
63
00:03:12,470 --> 00:03:15,580
get put into the queue, and
the server processes those
64
00:03:15,580 --> 00:03:17,750
jobs one at a time.
65
00:03:17,750 --> 00:03:20,480
Now to make a probabilistic
model, we need to make some
66
00:03:20,480 --> 00:03:23,240
assumption about the customer
arrivals and the customer
67
00:03:23,240 --> 00:03:24,290
departures.
68
00:03:24,290 --> 00:03:26,150
And we want to keep things
as simple as
69
00:03:26,150 --> 00:03:28,160
possible to get started.
70
00:03:28,160 --> 00:03:31,220
So let's assume that customers
arrive according to a
71
00:03:31,220 --> 00:03:34,230
Bernoulli process with
some parameter b.
72
00:03:34,230 --> 00:03:37,380
So essentially, that's the same
as the assumption that
73
00:03:37,380 --> 00:03:40,590
the time between consecutive
customer arrivals is a
74
00:03:40,590 --> 00:03:45,280
geometric random variable
with parameter b.
75
00:03:45,280 --> 00:03:49,740
Another way of thinking about
the arrival process--
76
00:03:49,740 --> 00:03:54,260
that's not how it happens, but
it's helpful, mathematically,
77
00:03:54,260 --> 00:03:58,330
is to think of someone who's
flipping a coin with bias
78
00:03:58,330 --> 00:03:59,510
equal to b.
79
00:03:59,510 --> 00:04:02,750
And whenever the coin
lands heads,
80
00:04:02,750 --> 00:04:05,050
then a customer arrives.
81
00:04:05,050 --> 00:04:08,960
So it's as if there's a coin
flip being done by nature that
82
00:04:08,960 --> 00:04:11,580
decides the arrivals
of the customers.
83
00:04:11,580 --> 00:04:16,110
So we know that coin flipping
to determine the customer
84
00:04:16,110 --> 00:04:19,550
arrivals is the same as having
geometric inter-arrival times.
85
00:04:19,550 --> 00:04:23,200
We know that from our study
of the Bernoulli process.
86
00:04:23,200 --> 00:04:23,630
OK.
87
00:04:23,630 --> 00:04:27,790
And now how about the customer
service times.
88
00:04:27,790 --> 00:04:29,610
We're going to assume that--
89
00:04:29,610 --> 00:04:30,660
OK.
90
00:04:30,660 --> 00:04:34,120
If there is no customer in
queue, no one being served,
91
00:04:34,120 --> 00:04:37,040
then of course, no
one is going to
92
00:04:37,040 --> 00:04:38,530
depart from the queue.
93
00:04:38,530 --> 00:04:42,270
But if there a customer in
queue, then that customer
94
00:04:42,270 --> 00:04:45,200
starts being served, and is
going to be served for a
95
00:04:45,200 --> 00:04:46,870
random amount of time.
96
00:04:46,870 --> 00:04:50,820
And we make the assumption that
the time it takes for the
97
00:04:50,820 --> 00:04:54,470
clerk to serve the customer has
a geometric distribution
98
00:04:54,470 --> 00:04:57,270
with some known parameter q.
99
00:04:57,270 --> 00:05:00,630
So the time it takes to serve a
customer is random, because
100
00:05:00,630 --> 00:05:04,060
it's random how many items they
got in their cart, and
101
00:05:04,060 --> 00:05:06,990
how many coupons they have
to unload and so on.
102
00:05:06,990 --> 00:05:09,730
So it's random.
103
00:05:09,730 --> 00:05:13,230
In the real world, it has some
probability distribution.
104
00:05:13,230 --> 00:05:16,240
Let's not care exactly about
what it would be in the real
105
00:05:16,240 --> 00:05:18,830
world, but as a modeling
approximation or just to get
106
00:05:18,830 --> 00:05:22,200
started, let's pretend that
customer service time are well
107
00:05:22,200 --> 00:05:25,150
described by a geometric
distribution,
108
00:05:25,150 --> 00:05:27,500
with a parameter q.
109
00:05:27,500 --> 00:05:31,010
An equivalent way of thinking
about the customer service,
110
00:05:31,010 --> 00:05:32,640
mathematically, would
be, again, in
111
00:05:32,640 --> 00:05:34,350
terms of coin flipping.
112
00:05:34,350 --> 00:05:38,490
That is, the clerk has a coin
with a bias, and at each time
113
00:05:38,490 --> 00:05:40,620
slot the clerk flips the coin.
114
00:05:40,620 --> 00:05:43,470
With probability q,
service is over.
115
00:05:43,470 --> 00:05:49,100
With probability 1-q, you
continue the service process.
116
00:05:49,100 --> 00:05:52,370
An assumption that we're going
to make is that the coin flips
117
00:05:52,370 --> 00:05:56,220
that happen here to determine
the arrivals, they're all
118
00:05:56,220 --> 00:05:57,860
independent of each other.
119
00:05:57,860 --> 00:06:02,610
The coin flips that determine
the end of service are also
120
00:06:02,610 --> 00:06:03,930
independent from each other.
121
00:06:03,930 --> 00:06:07,160
But also the coin flips involved
here are independent
122
00:06:07,160 --> 00:06:09,740
from the coin flips that
happened there.
123
00:06:09,740 --> 00:06:15,190
So how arrivals happen is
independent with what happens
124
00:06:15,190 --> 00:06:17,490
at the service process.
125
00:06:17,490 --> 00:06:18,330
OK.
126
00:06:18,330 --> 00:06:21,240
So suppose now you
want to answer a
127
00:06:21,240 --> 00:06:23,200
question such as the following.
128
00:06:23,200 --> 00:06:24,830
The time is 7:00 PM.
129
00:06:24,830 --> 00:06:28,640
What's the probability that the
customer will be departing
130
00:06:28,640 --> 00:06:30,990
at this particular time?
131
00:06:30,990 --> 00:06:33,870
Well, you say, it depends.
132
00:06:33,870 --> 00:06:37,740
If the queue is empty at that
time, then you're certain that
133
00:06:37,740 --> 00:06:40,200
you're not going to have
a customer departure.
134
00:06:40,200 --> 00:06:44,970
But if the queue is not empty,
then there is probability q
135
00:06:44,970 --> 00:06:47,880
that a departure will
happen at that time.
136
00:06:47,880 --> 00:06:52,230
So the answer to a question like
this has something to do
137
00:06:52,230 --> 00:06:54,460
with the state of the
system at that time.
138
00:06:54,460 --> 00:06:56,770
It depends what the queue is.
139
00:06:56,770 --> 00:07:02,770
And if I ask you, will the
queue be empty at 7:10?
140
00:07:02,770 --> 00:07:06,870
Well, the answer to that
question depends on whether at
141
00:07:06,870 --> 00:07:10,630
7 o'clock whether the queue
was huge or not.
142
00:07:10,630 --> 00:07:14,770
So knowing something about the
state of the queue right now
143
00:07:14,770 --> 00:07:17,480
gives me relevant information
about what may
144
00:07:17,480 --> 00:07:19,790
happen in the future.
145
00:07:19,790 --> 00:07:22,770
So what is the state
of the system?
146
00:07:22,770 --> 00:07:26,080
Therefore we're brought to
start using this term.
147
00:07:26,080 --> 00:07:28,720
So the state basically
corresponds to
148
00:07:28,720 --> 00:07:31,980
anything that's relevant.
149
00:07:31,980 --> 00:07:34,900
Anything that's happening
right now that's kind of
150
00:07:34,900 --> 00:07:38,120
relevant to what may happen
in the future.
151
00:07:38,120 --> 00:07:41,360
Knowing the size of the queue
right now, is useful
152
00:07:41,360 --> 00:07:45,700
information for me to make
predictions about what may
153
00:07:45,700 --> 00:07:49,100
happen 2 minutes
later from now.
154
00:07:49,100 --> 00:07:52,510
So in this particular example,
a reasonable choice for the
155
00:07:52,510 --> 00:07:56,410
state is to just count
how many customers
156
00:07:56,410 --> 00:07:58,950
we have in the queue.
157
00:07:58,950 --> 00:08:02,330
And let's assume that our
supermarket building is not
158
00:08:02,330 --> 00:08:05,230
too big, so it can only
hold 10 people.
159
00:08:05,230 --> 00:08:07,480
So we're going to limit
the states.
160
00:08:07,480 --> 00:08:11,230
Instead of going from 0 to
infinity, we're going to
161
00:08:11,230 --> 00:08:13,580
truncate our model at ten.
162
00:08:13,580 --> 00:08:18,240
So we have 11 possible states,
corresponding to 0 customers
163
00:08:18,240 --> 00:08:22,620
in queue, 1 customer in queue,
2 customers, and so on, all
164
00:08:22,620 --> 00:08:24,190
the way up to 10.
165
00:08:24,190 --> 00:08:27,600
So these are the different
possible states of the system,
166
00:08:27,600 --> 00:08:33,340
assuming that the store cannot
handle more than 10 customers.
167
00:08:33,340 --> 00:08:37,360
So this is the first step, to
write down the set of possible
168
00:08:37,360 --> 00:08:38,950
states for our system.
169
00:08:38,950 --> 00:08:41,820
Then the next thing to do is
to start describing the
170
00:08:41,820 --> 00:08:45,570
possible transitions
between the states.
171
00:08:45,570 --> 00:08:48,750
At any given time step,
what are the
172
00:08:48,750 --> 00:08:50,030
things that can happen?
173
00:08:50,030 --> 00:08:53,180
We can have a customer
arrival, which
174
00:08:53,180 --> 00:08:55,830
moves the state 1 higher.
175
00:08:55,830 --> 00:08:58,560
We can have a customer
departure, which moves the
176
00:08:58,560 --> 00:09:00,320
state 1 lower.
177
00:09:00,320 --> 00:09:03,080
There's a possibility that
nothing happens, in which case
178
00:09:03,080 --> 00:09:04,710
the state stays the same.
179
00:09:04,710 --> 00:09:06,470
And there's also the possibility
of having
180
00:09:06,470 --> 00:09:10,800
simultaneously an arrival and a
departure, in which case the
181
00:09:10,800 --> 00:09:12,700
state again stays the same.
182
00:09:12,700 --> 00:09:16,290
So let's write some
representative probabilities.
183
00:09:16,290 --> 00:09:19,630
If we have 2 customers, the
probability that during this
184
00:09:19,630 --> 00:09:22,800
step we go down, this is the
probability that we have a
185
00:09:22,800 --> 00:09:26,940
service completion, but to
no customer arrival.
186
00:09:26,940 --> 00:09:30,060
So this is the probability
associated with this
187
00:09:30,060 --> 00:09:31,730
transition.
188
00:09:31,730 --> 00:09:37,280
The other possibility is that
there's a customer arrival,
189
00:09:37,280 --> 00:09:40,910
which happens with probability
p, and we do not have a
190
00:09:40,910 --> 00:09:45,120
customer departure, and so the
probability of that particular
191
00:09:45,120 --> 00:09:47,690
transition is this number.
192
00:09:47,690 --> 00:09:50,960
And then finally, the
probability that we stay in
193
00:09:50,960 --> 00:09:55,530
the same state, this can happen
in 2 possible ways.
194
00:09:55,530 --> 00:10:00,360
One way is that we have an
arrival and a departure
195
00:10:00,360 --> 00:10:01,930
simultaneously.
196
00:10:01,930 --> 00:10:05,690
And the other possibility is
that we have no arrival and no
197
00:10:05,690 --> 00:10:09,670
departure, so that the
state stays the same.
198
00:10:09,670 --> 00:10:11,870
So these transition
probabilities would be the
199
00:10:11,870 --> 00:10:15,420
same starting from any other
states, state 3, or
200
00:10:15,420 --> 00:10:17,050
state 9, and so on.
201
00:10:17,050 --> 00:10:20,010
Transition probabilities become
a little different at
202
00:10:20,010 --> 00:10:23,750
the borders, at the boundaries
of this diagram, because if
203
00:10:23,750 --> 00:10:27,350
you're in a state 0, then you
cannot have any customer
204
00:10:27,350 --> 00:10:28,130
departures.
205
00:10:28,130 --> 00:10:31,940
There's no one to be served, but
there is a probability p
206
00:10:31,940 --> 00:10:36,040
that the customer arrives, in
which case the number of
207
00:10:36,040 --> 00:10:38,110
customers in the system
goes to 1.
208
00:10:38,110 --> 00:10:41,150
Then probability 1-p,
nothing happens.
209
00:10:41,150 --> 00:10:46,020
Similarly with departures, if
the system is full, there's no
210
00:10:46,020 --> 00:10:47,780
room for another arrival.
211
00:10:47,780 --> 00:10:50,300
But we may have a departure that
happens with probability
212
00:10:50,300 --> 00:10:55,250
q, and nothing happens
with probability 1-q.
213
00:10:55,250 --> 00:11:00,260
So this is the full transition
diagram annotated with
214
00:11:00,260 --> 00:11:02,150
transition probabilities.
215
00:11:02,150 --> 00:11:05,970
And this is a complete
description of a discrete
216
00:11:05,970 --> 00:11:10,000
time, finite state
Markov chain.
217
00:11:10,000 --> 00:11:13,010
So this is a complete
probabilistic model.
218
00:11:13,010 --> 00:11:15,520
Once you have all of these
pieces of information, you can
219
00:11:15,520 --> 00:11:18,370
start calculating things, and
trying to predict what's going
220
00:11:18,370 --> 00:11:20,140
to happen in the future.
221
00:11:20,140 --> 00:11:24,460
Now let us abstract from this
example and come up with a
222
00:11:24,460 --> 00:11:27,530
more general definition.
223
00:11:27,530 --> 00:11:37,010
So we have this concept of the
state which describes the
224
00:11:37,010 --> 00:11:40,560
current situation in the system
that we're looking at.
225
00:11:40,560 --> 00:11:44,440
The current state is random, so
we're going to think of it
226
00:11:44,440 --> 00:11:50,570
as a random variable Xn is the
state, and transitions after
227
00:11:50,570 --> 00:11:52,560
the system started operating.
228
00:11:52,560 --> 00:11:56,450
So the system starts operating
at some initial state X0, and
229
00:11:56,450 --> 00:12:00,190
after n transitions, it
moves to state Xn.
230
00:12:00,190 --> 00:12:03,020
Now we have a set of
possible states.
231
00:12:03,020 --> 00:12:06,930
State 1 state 2, state
3, and in general,
232
00:12:06,930 --> 00:12:10,680
state i and state j.
233
00:12:10,680 --> 00:12:13,870
To keep things simple, we
assume that the set of
234
00:12:13,870 --> 00:12:16,700
possible states is
a finite set.
235
00:12:16,700 --> 00:12:19,350
As you can imagine, we can
have systems in which the
236
00:12:19,350 --> 00:12:21,160
state space is going
to be infinite.
237
00:12:21,160 --> 00:12:23,240
It could be discrete,
or continuous.
238
00:12:23,240 --> 00:12:25,870
But all that is more difficult
and more complicated.
239
00:12:25,870 --> 00:12:29,110
It makes sense to start from the
simplest possible setting
240
00:12:29,110 --> 00:12:33,770
where we just deal with the
finite state space.
241
00:12:33,770 --> 00:12:39,430
And time is discrete, so we can
think of this state in the
242
00:12:39,430 --> 00:12:42,660
beginning, after 1 transition,
2 transitions, and so on.
243
00:12:42,660 --> 00:12:46,600
So we're in discrete time and we
have finite in many states.
244
00:12:46,600 --> 00:12:49,900
So the system starts somewhere,
and at every time
245
00:12:49,900 --> 00:12:54,506
step, the state is,
let's say, here.
246
00:12:54,506 --> 00:12:59,850
A whistle blows, and the state
jumps to a random next state.
247
00:12:59,850 --> 00:13:05,120
So it may move here, or it may
move there, or it may move
248
00:13:05,120 --> 00:13:08,510
here, or it might stay
in the place.
249
00:13:08,510 --> 00:13:11,400
So one possible transition is
the transition before you
250
00:13:11,400 --> 00:13:13,880
jump, and just land
in the same place
251
00:13:13,880 --> 00:13:15,760
where you started from.
252
00:13:15,760 --> 00:13:19,410
Now we want to describe the
statistics of these
253
00:13:19,410 --> 00:13:20,490
transitions.
254
00:13:20,490 --> 00:13:23,760
If I am at that state, how
likely is it to that, next
255
00:13:23,760 --> 00:13:26,885
time, I'm going to find
myself at that state?
256
00:13:26,885 --> 00:13:30,390
Well, we describe the statistics
of this transition
257
00:13:30,390 --> 00:13:35,730
by writing down a transition
probability, the transition
258
00:13:35,730 --> 00:13:41,420
probability of going from
state 3 to state 1.
259
00:13:41,420 --> 00:13:44,180
So this transition probability
is to be thought of as a
260
00:13:44,180 --> 00:13:45,960
conditional probability.
261
00:13:45,960 --> 00:13:49,620
Given that right now I am
at state i what is the
262
00:13:49,620 --> 00:13:55,650
probability that next time
I find myself at state j?
263
00:13:55,650 --> 00:14:00,100
So given that right now I am
at state 3, P31 is the
264
00:14:00,100 --> 00:14:02,090
probability that the next
time I'm going to find
265
00:14:02,090 --> 00:14:04,740
myself at state 1.
266
00:14:04,740 --> 00:14:09,340
Similarly here, we would have
a probability P3i, which is
267
00:14:09,340 --> 00:14:12,710
the probability that given that
right now I'm at state 3,
268
00:14:12,710 --> 00:14:17,680
next time I'm going to find
myself at state i.
269
00:14:17,680 --> 00:14:21,390
Now one can write such
conditional probabilities down
270
00:14:21,390 --> 00:14:25,110
in principle, but we
need to make--
271
00:14:25,110 --> 00:14:29,040
so you might think of this as a
definition here, but we need
272
00:14:29,040 --> 00:14:34,050
to make one additional big
assumption, and this is the
273
00:14:34,050 --> 00:14:36,360
assumption that to
make a process
274
00:14:36,360 --> 00:14:38,540
to be a Markov process.
275
00:14:38,540 --> 00:14:41,210
This is the so-called
Markov property, and
276
00:14:41,210 --> 00:14:43,770
here's what it says.
277
00:14:43,770 --> 00:14:46,760
Let me describe it first
in words here.
278
00:14:46,760 --> 00:14:52,230
Every time that I find myself
at state 3, the probability
279
00:14:52,230 --> 00:14:56,380
that next time I'm going to find
myself at state 1 is this
280
00:14:56,380 --> 00:15:00,890
particular number, no matter
how I got there.
281
00:15:00,890 --> 00:15:04,870
That is, this transition
probability is not affected by
282
00:15:04,870 --> 00:15:06,560
the past of the process.
283
00:15:06,560 --> 00:15:11,930
It doesn't care about what
path I used to find
284
00:15:11,930 --> 00:15:14,150
myself at state 3.
285
00:15:14,150 --> 00:15:17,060
Mathematically, it means
the following.
286
00:15:17,060 --> 00:15:19,700
You have this transition
probability that from state i
287
00:15:19,700 --> 00:15:21,480
jump to state j.
288
00:15:21,480 --> 00:15:24,530
Suppose that I gave you some
additional information, that I
289
00:15:24,530 --> 00:15:27,780
told you everything else that
happened in the past of the
290
00:15:27,780 --> 00:15:30,410
process, everything that
happened, how did you
291
00:15:30,410 --> 00:15:32,570
get to state i?
292
00:15:32,570 --> 00:15:35,550
The assumption we're making is
that this information about
293
00:15:35,550 --> 00:15:39,940
the past has no bearing in
making predictions about the
294
00:15:39,940 --> 00:15:44,890
future, as long as you know
where you are right now.
295
00:15:44,890 --> 00:15:49,300
So if I tell you, right now, you
are at state i, and by the
296
00:15:49,300 --> 00:15:53,010
way, you got there by following
a particular path,
297
00:15:53,010 --> 00:15:56,940
you can ignore the extra
information of the particular
298
00:15:56,940 --> 00:15:58,410
path that you followed.
299
00:15:58,410 --> 00:16:01,610
You only take into account
where you are right now.
300
00:16:01,610 --> 00:16:05,800
So every time you find yourself
at that state, no
301
00:16:05,800 --> 00:16:10,050
matter how you got there, you
will find yourself next time
302
00:16:10,050 --> 00:16:12,980
at state 1 with probability
P31.
303
00:16:12,980 --> 00:16:17,740
So the past has no bearing into
the future, as long as
304
00:16:17,740 --> 00:16:21,600
you know where you are
sitting right now.
305
00:16:21,600 --> 00:16:27,390
For this property to happen, you
need to choose your state
306
00:16:27,390 --> 00:16:29,650
carefully in the right way.
307
00:16:29,650 --> 00:16:32,950
In that sense, the states
needs to include any
308
00:16:32,950 --> 00:16:36,310
information that's relevant
about the
309
00:16:36,310 --> 00:16:38,280
future of the system.
310
00:16:38,280 --> 00:16:41,580
Anything that's not in the state
is not going to play a
311
00:16:41,580 --> 00:16:45,340
role, but the state needs to
have all the information
312
00:16:45,340 --> 00:16:48,580
that's relevant in determining
what kind of transitions are
313
00:16:48,580 --> 00:16:50,080
going to happen next.
314
00:16:50,080 --> 00:16:54,690
So to take an example, before
you go to Markov process, just
315
00:16:54,690 --> 00:16:57,660
from the deterministic world,
if you have a ball that's
316
00:16:57,660 --> 00:17:01,630
flying up in the air, and you
want to make predictions about
317
00:17:01,630 --> 00:17:02,730
the future.
318
00:17:02,730 --> 00:17:06,369
If I tell you that the state of
the ball is the position of
319
00:17:06,369 --> 00:17:11,579
the ball at the particular time,
is that enough for you
320
00:17:11,579 --> 00:17:15,240
to make predictions where the
ball is going to go next?
321
00:17:15,240 --> 00:17:15,700
No.
322
00:17:15,700 --> 00:17:19,460
You need to know both the
position and the velocity.
323
00:17:19,460 --> 00:17:21,710
If you know position and
velocity, you can make
324
00:17:21,710 --> 00:17:23,490
predictions about the future.
325
00:17:23,490 --> 00:17:27,520
So the state of a ball that's
flying is position together
326
00:17:27,520 --> 00:17:29,450
with velocity.
327
00:17:29,450 --> 00:17:32,430
If you were to just take
position, that would not be
328
00:17:32,430 --> 00:17:36,290
enough information, because if
I tell you current position,
329
00:17:36,290 --> 00:17:39,640
and then I tell you past
position, you could use the
330
00:17:39,640 --> 00:17:42,120
information from the past
position to complete the
331
00:17:42,120 --> 00:17:43,930
trajectory and to make
the prediction.
332
00:17:43,930 --> 00:17:47,870
So information from the past
is useful if you don't know
333
00:17:47,870 --> 00:17:48,580
the velocity.
334
00:17:48,580 --> 00:17:53,650
But if both position and
velocity, you don't care how
335
00:17:53,650 --> 00:17:56,220
you got there, or what
time you started.
336
00:17:56,220 --> 00:17:58,660
From position and velocity, you
can make predictions about
337
00:17:58,660 --> 00:17:59,800
the future.
338
00:17:59,800 --> 00:18:04,330
So there's a certain art, or a
certain element of thinking, a
339
00:18:04,330 --> 00:18:07,400
non-mechanical aspect into
problems of this kind, to
340
00:18:07,400 --> 00:18:11,840
figure out which is the
right state variable.
341
00:18:11,840 --> 00:18:14,670
When you define the state of
your system, you need to
342
00:18:14,670 --> 00:18:19,870
define it in such a way that
includes all information that
343
00:18:19,870 --> 00:18:23,735
has been accumulated that has
some relevance for the future.
344
00:18:23,735 --> 00:18:27,380
345
00:18:27,380 --> 00:18:31,250
So the general process for
coming up with a Markov model
346
00:18:31,250 --> 00:18:34,970
is to first make this big
decision of what your state
347
00:18:34,970 --> 00:18:37,480
variable is going to be.
348
00:18:37,480 --> 00:18:41,570
Then you write down if it
may be a picture of
349
00:18:41,570 --> 00:18:43,150
the different states.
350
00:18:43,150 --> 00:18:45,720
Then you identify the possible
transitions.
351
00:18:45,720 --> 00:18:48,810
So sometimes the diagram that
you're going to have will not
352
00:18:48,810 --> 00:18:50,970
include all the possible arcs.
353
00:18:50,970 --> 00:18:54,040
You would only show those arcs
that correspond to transitions
354
00:18:54,040 --> 00:18:54,770
that are possible.
355
00:18:54,770 --> 00:18:57,850
For example, in the supermarket
example, we did
356
00:18:57,850 --> 00:19:01,660
not have a transition from state
2 to state 5, because
357
00:19:01,660 --> 00:19:02,590
that cannot happen.
358
00:19:02,590 --> 00:19:05,360
You can only have 1 arrival
at any time.
359
00:19:05,360 --> 00:19:08,330
So in the diagram, we only
showed the possible
360
00:19:08,330 --> 00:19:09,280
transitions.
361
00:19:09,280 --> 00:19:12,200
And for each of the possible
transitions, then you work
362
00:19:12,200 --> 00:19:15,060
with the description of the
model to figure out the
363
00:19:15,060 --> 00:19:17,380
correct transition
probability.
364
00:19:17,380 --> 00:19:21,090
So you got the diagram by
writing down transition
365
00:19:21,090 --> 00:19:22,340
probabilities.
366
00:19:22,340 --> 00:19:26,890
367
00:19:26,890 --> 00:19:30,930
OK, so suppose you got
your Markov model.
368
00:19:30,930 --> 00:19:32,900
What will you do with it?
369
00:19:32,900 --> 00:19:34,900
Well, what do we need
models for?
370
00:19:34,900 --> 00:19:38,580
We need models in order to
make predictions, to make
371
00:19:38,580 --> 00:19:39,890
probabilistic predictions.
372
00:19:39,890 --> 00:19:42,750
So for example, I tell you that
the process started in
373
00:19:42,750 --> 00:19:43,790
that state.
374
00:19:43,790 --> 00:19:46,070
You let it run for some time.
375
00:19:46,070 --> 00:19:49,980
Where do you think it's going to
be 10 time steps from now?
376
00:19:49,980 --> 00:19:52,540
That's a question that you
might want to answer.
377
00:19:52,540 --> 00:19:55,660
Since the process is random,
there's no way for you to tell
378
00:19:55,660 --> 00:19:58,610
me exactly where it's
going to be.
379
00:19:58,610 --> 00:20:00,480
But maybe you can give
me probabilities.
380
00:20:00,480 --> 00:20:02,880
You can tell me, with so
much probability, the
381
00:20:02,880 --> 00:20:04,240
state would be there.
382
00:20:04,240 --> 00:20:06,080
With so much probability,
the state would be
383
00:20:06,080 --> 00:20:07,680
there, and so on.
384
00:20:07,680 --> 00:20:12,010
So our first exercise is to
calculate those probabilities
385
00:20:12,010 --> 00:20:16,720
about what may happen to the
process a number of steps in
386
00:20:16,720 --> 00:20:18,790
the future.
387
00:20:18,790 --> 00:20:21,800
It's handy to have some
notation in here.
388
00:20:21,800 --> 00:20:25,700
So somebody tells us that this
process starts at the
389
00:20:25,700 --> 00:20:27,560
particular state i.
390
00:20:27,560 --> 00:20:31,800
We let the process run
for n transitions.
391
00:20:31,800 --> 00:20:36,190
It may land at some state j, but
that state j at which it's
392
00:20:36,190 --> 00:20:38,060
going to land is going
to be random.
393
00:20:38,060 --> 00:20:40,440
So we want to give
probabilities.
394
00:20:40,440 --> 00:20:44,750
Tell me, with what probability
the state, n times steps
395
00:20:44,750 --> 00:20:49,100
later, is going to be that
particular state j?
396
00:20:49,100 --> 00:20:54,830
The shorthand notation is to use
this symbol here for the
397
00:20:54,830 --> 00:20:58,730
n-step transition probabilities
that you find
398
00:20:58,730 --> 00:21:02,610
yourself at state j given that
you started at state i.
399
00:21:02,610 --> 00:21:05,930
So the way these two indices are
ordered, the way to think
400
00:21:05,930 --> 00:21:09,130
about them is that from
i, you go to j.
401
00:21:09,130 --> 00:21:13,040
So the probability that from
i you go to j if you have n
402
00:21:13,040 --> 00:21:16,210
steps in front of you.
403
00:21:16,210 --> 00:21:18,890
Some of these transition
probabilities are, of course
404
00:21:18,890 --> 00:21:20,190
easy to write.
405
00:21:20,190 --> 00:21:29,530
For example, in 0 transitions,
you're going to be exactly
406
00:21:29,530 --> 00:21:30,860
where you started.
407
00:21:30,860 --> 00:21:35,590
So this probability is going to
be equal to 1 if i is equal
408
00:21:35,590 --> 00:21:40,870
to j, And 0 if i is
different than j.
409
00:21:40,870 --> 00:21:43,360
That's an easy one
to write down.
410
00:21:43,360 --> 00:21:48,250
If you have only 1 transition,
what's the probability that 1
411
00:21:48,250 --> 00:21:51,740
step later you find yourself
in state j given that you
412
00:21:51,740 --> 00:21:54,310
started at state i?
413
00:21:54,310 --> 00:21:56,830
What is this?
414
00:21:56,830 --> 00:22:00,450
These are just the ordinary
1-step transition
415
00:22:00,450 --> 00:22:03,180
probabilities that we are given
in the description of
416
00:22:03,180 --> 00:22:04,780
the problem.
417
00:22:04,780 --> 00:22:08,965
So by definition, the 1-step
transition probabilities are
418
00:22:08,965 --> 00:22:10,215
of this form.
419
00:22:10,215 --> 00:22:14,070
420
00:22:14,070 --> 00:22:17,980
This equality is correct just
because of the way that we
421
00:22:17,980 --> 00:22:20,680
defined those two quantities.
422
00:22:20,680 --> 00:22:24,670
Now we want to say something
about the n-step transition
423
00:22:24,670 --> 00:22:28,760
probabilities when n
is a bigger number.
424
00:22:28,760 --> 00:22:31,320
425
00:22:31,320 --> 00:22:31,700
OK.
426
00:22:31,700 --> 00:22:36,450
So here, we're going to use the
total probability theorem.
427
00:22:36,450 --> 00:22:39,750
So we're going to condition in
two different scenarios, and
428
00:22:39,750 --> 00:22:43,580
break up the calculation of this
quantity, by considering
429
00:22:43,580 --> 00:22:46,850
the different ways that
this event can happen.
430
00:22:46,850 --> 00:22:49,110
So what is the event
of interest?
431
00:22:49,110 --> 00:22:51,040
The event of interest
is the following.
432
00:22:51,040 --> 00:22:54,070
At time 0 we start i.
433
00:22:54,070 --> 00:22:57,310
We are interested in landing
at time n at the
434
00:22:57,310 --> 00:22:59,640
particular state j.
435
00:22:59,640 --> 00:23:03,860
Now this event can happen in
several different ways, in
436
00:23:03,860 --> 00:23:05,120
lots of different ways.
437
00:23:05,120 --> 00:23:08,630
But let us group them
into subgroups.
438
00:23:08,630 --> 00:23:12,640
One group, or one sort of
scenario, is the following.
439
00:23:12,640 --> 00:23:18,200
During the first n-1 time steps,
things happen, and
440
00:23:18,200 --> 00:23:20,750
somehow you end up at state 1.
441
00:23:20,750 --> 00:23:24,180
And then from state 1, in the
next time step you make a
442
00:23:24,180 --> 00:23:27,160
transition to state j.
443
00:23:27,160 --> 00:23:32,770
This particular arc here
actually corresponds to lots
444
00:23:32,770 --> 00:23:36,600
and lots of different possible
scenarios, or different spots,
445
00:23:36,600 --> 00:23:38,110
or different transitions.
446
00:23:38,110 --> 00:23:43,770
In n-1 time steps, there's lots
of possible ways by which
447
00:23:43,770 --> 00:23:46,010
you could end up at state 1.
448
00:23:46,010 --> 00:23:48,650
Different paths through
the state space.
449
00:23:48,650 --> 00:23:51,630
But all of them together
collectively have a
450
00:23:51,630 --> 00:23:55,360
probability, which is the
(n-1)-step transition
451
00:23:55,360 --> 00:24:02,200
probability, that from state
i, you end up at state 1
452
00:24:02,200 --> 00:24:05,960
And then there's other
possible scenarios.
453
00:24:05,960 --> 00:24:10,120
Perhaps in the first n-1 time
steps, you follow the
454
00:24:10,120 --> 00:24:13,370
trajectory that took
you at state m.
455
00:24:13,370 --> 00:24:17,430
And then from state m, you did
this transition, and you ended
456
00:24:17,430 --> 00:24:18,980
up at state j.
457
00:24:18,980 --> 00:24:22,580
So this diagram breaks up
the set of all possible
458
00:24:22,580 --> 00:24:27,360
trajectories from i to j into
different collections, where
459
00:24:27,360 --> 00:24:31,340
each collection has to do with
which one happens to be the
460
00:24:31,340 --> 00:24:37,070
state just before the last time
step, just before time n.
461
00:24:37,070 --> 00:24:40,040
And we're going to condition
on the state at time n-1.
462
00:24:40,040 --> 00:24:42,620
463
00:24:42,620 --> 00:24:48,180
So the total probability of
ending up at state j is the
464
00:24:48,180 --> 00:24:53,090
sum of the probabilities of
the different scenarios --
465
00:24:53,090 --> 00:24:56,380
the different ways that you
can get to state j.
466
00:24:56,380 --> 00:25:00,650
If we look at that type of
scenario, what's the
467
00:25:00,650 --> 00:25:03,030
probability of that scenario
happening?
468
00:25:03,030 --> 00:25:08,290
With probability Ri1(n-1),
I find myself at
469
00:25:08,290 --> 00:25:10,810
state 1 at time n-1.
470
00:25:10,810 --> 00:25:15,000
This is just by the definition
of these multi-step transition
471
00:25:15,000 --> 00:25:16,160
probabilities.
472
00:25:16,160 --> 00:25:17,990
This is the number
of transitions.
473
00:25:17,990 --> 00:25:22,690
The probability that from state
i, I end up at state 1.
474
00:25:22,690 --> 00:25:27,130
And then given that I found
myself at state 1, with
475
00:25:27,130 --> 00:25:31,350
probability P1j, that's the
transition probability, next
476
00:25:31,350 --> 00:25:34,710
time I'm going to find
myself at state j.
477
00:25:34,710 --> 00:25:39,610
So the product of these two is
the total probability of my
478
00:25:39,610 --> 00:25:43,500
getting from state i to
state j through state
479
00:25:43,500 --> 00:25:47,340
1 at the time before.
480
00:25:47,340 --> 00:25:53,160
Now where exactly did we use
the Markov assumption here?
481
00:25:53,160 --> 00:25:57,750
No matter which particular path
we used to get from i to
482
00:25:57,750 --> 00:26:01,660
state 1, the probability that
next I'm going to make this
483
00:26:01,660 --> 00:26:05,510
transition is that
same number, P1j.
484
00:26:05,510 --> 00:26:09,170
So that number does not depend
on the particular path that I
485
00:26:09,170 --> 00:26:11,240
followed in order
to get there.
486
00:26:11,240 --> 00:26:15,090
If we didn't have the Markov
assumption, we should have
487
00:26:15,090 --> 00:26:18,610
considered all possible
individual trajectories here,
488
00:26:18,610 --> 00:26:21,360
and then we would need to use
the transition probability
489
00:26:21,360 --> 00:26:23,840
that corresponds to that
particular trajectory.
490
00:26:23,840 --> 00:26:26,130
But because of the Markov
assumption, the only thing
491
00:26:26,130 --> 00:26:29,930
that matters is that right
now we are at state 1.
492
00:26:29,930 --> 00:26:33,100
It does not matter
how we got there.
493
00:26:33,100 --> 00:26:37,240
So now once you see this
scenario, then this scenario,
494
00:26:37,240 --> 00:26:40,160
and that scenario, and you add
the probabilities of these
495
00:26:40,160 --> 00:26:43,820
different scenarios, you end
up with this formula here,
496
00:26:43,820 --> 00:26:45,540
which is a recursion.
497
00:26:45,540 --> 00:26:49,810
It tells us that once you have
computed the (n-1)-step
498
00:26:49,810 --> 00:26:53,830
transition probabilities, then
you can compute also the
499
00:26:53,830 --> 00:26:55,990
n-step transition
probabilities.
500
00:26:55,990 --> 00:27:01,390
This is a recursion that you
execute or you run for all i's
501
00:27:01,390 --> 00:27:03,320
and j's simultaneously.
502
00:27:03,320 --> 00:27:04,180
That is fixed.
503
00:27:04,180 --> 00:27:08,280
And for a particular n, you
calculate this quantity for
504
00:27:08,280 --> 00:27:10,620
all possible i's, j's, k's.
505
00:27:10,620 --> 00:27:13,710
You have all of those
quantities, and then you use
506
00:27:13,710 --> 00:27:16,730
this equation to find those
numbers again for all the
507
00:27:16,730 --> 00:27:20,340
possible i's and j's.
508
00:27:20,340 --> 00:27:26,620
Now this is formula which is
always true, and there's a big
509
00:27:26,620 --> 00:27:28,810
idea behind the formula.
510
00:27:28,810 --> 00:27:32,050
And now there's variations of
this formula, depending on
511
00:27:32,050 --> 00:27:33,610
whether you're interested
in something
512
00:27:33,610 --> 00:27:35,200
that's slightly different.
513
00:27:35,200 --> 00:27:42,070
So for example, if you were to
have a random initial state,
514
00:27:42,070 --> 00:27:44,850
somebody gives you the
probability distribution of
515
00:27:44,850 --> 00:27:48,300
the initial state, so you're
told that with probability
516
00:27:48,300 --> 00:27:51,250
such and such, you're going
to start at state 1.
517
00:27:51,250 --> 00:27:52,760
With that probability, you're
going to start at
518
00:27:52,760 --> 00:27:54,200
state 2, and so on.
519
00:27:54,200 --> 00:27:56,560
And you want to find the
probability at the time n you
520
00:27:56,560 --> 00:27:58,530
find yourself at state j.
521
00:27:58,530 --> 00:28:01,880
Well again, total probability
theorem, you condition on the
522
00:28:01,880 --> 00:28:03,120
initial state.
523
00:28:03,120 --> 00:28:05,430
With this probability you find
yourself at that particular
524
00:28:05,430 --> 00:28:08,570
initial state, and given that
this is your initial state,
525
00:28:08,570 --> 00:28:11,840
this is the probability that
n time steps later you find
526
00:28:11,840 --> 00:28:14,980
yourself at state j.
527
00:28:14,980 --> 00:28:20,080
Now building again on the same
idea, you can run every
528
00:28:20,080 --> 00:28:23,330
recursion of this kind
by conditioning
529
00:28:23,330 --> 00:28:24,950
at different times.
530
00:28:24,950 --> 00:28:26,200
So here's a variation.
531
00:28:26,200 --> 00:28:29,260
532
00:28:29,260 --> 00:28:31,520
You start at state i.
533
00:28:31,520 --> 00:28:36,240
After 1 time step, you find
yourself at state 1, with
534
00:28:36,240 --> 00:28:40,630
probability pi1, and you find
yourself at state m with
535
00:28:40,630 --> 00:28:43,930
probability Pim.
536
00:28:43,930 --> 00:28:49,250
And once that happens, then
you're going to follow some
537
00:28:49,250 --> 00:28:51,070
trajectories.
538
00:28:51,070 --> 00:28:54,390
And there is a possibility that
you're going to end up at
539
00:28:54,390 --> 00:28:58,285
state j after n-1 time steps.
540
00:28:58,285 --> 00:29:02,160
541
00:29:02,160 --> 00:29:05,160
This scenario can happen
in many possible ways.
542
00:29:05,160 --> 00:29:08,130
There's lots of possible paths
from state 1 to state j.
543
00:29:08,130 --> 00:29:12,680
There's many paths from
state 1 to state j.
544
00:29:12,680 --> 00:29:15,940
What is the collective
probability of all these
545
00:29:15,940 --> 00:29:17,190
transitions?
546
00:29:17,190 --> 00:29:19,250
547
00:29:19,250 --> 00:29:23,150
This is the event that, starting
from state 1, I end
548
00:29:23,150 --> 00:29:27,560
up at state j in
n-1 time steps.
549
00:29:27,560 --> 00:29:34,240
So this one has here probability
R1j of n-1.
550
00:29:34,240 --> 00:29:37,980
And similarly down here.
551
00:29:37,980 --> 00:29:41,350
And then by using the same way
of thinking as before, we get
552
00:29:41,350 --> 00:29:48,650
the formula that Rij(n) is the
sum over all k's of Pik, and
553
00:29:48,650 --> 00:29:49,940
then the Rkj(n-1).
554
00:29:49,940 --> 00:29:54,800
555
00:29:54,800 --> 00:29:59,050
So this formula looks almost the
same as this one, but it's
556
00:29:59,050 --> 00:30:00,810
actually different.
557
00:30:00,810 --> 00:30:05,570
The indices and the way things
work out are a bit different,
558
00:30:05,570 --> 00:30:08,500
but the basic idea
is the same.
559
00:30:08,500 --> 00:30:10,940
Here we use the total
probability theory by
560
00:30:10,940 --> 00:30:15,770
conditioning on the state just
1 step before the end of our
561
00:30:15,770 --> 00:30:17,020
time horizon.
562
00:30:17,020 --> 00:30:21,260
Here we use total probability
theorem by conditioning on the
563
00:30:21,260 --> 00:30:24,300
state right after the
first transition.
564
00:30:24,300 --> 00:30:28,340
So this generally idea has
different variations.
565
00:30:28,340 --> 00:30:30,920
They're all valid, and depending
on the context that
566
00:30:30,920 --> 00:30:34,600
you're dealing with, you might
want to work with one of these
567
00:30:34,600 --> 00:30:37,130
or another.
568
00:30:37,130 --> 00:30:40,070
So let's illustrate
these calculations
569
00:30:40,070 --> 00:30:42,090
in terms of an example.
570
00:30:42,090 --> 00:30:46,910
So in this example, we just have
2 states, and somebody
571
00:30:46,910 --> 00:30:49,510
gives us transition
probabilities to be those
572
00:30:49,510 --> 00:30:51,740
particular numbers.
573
00:30:51,740 --> 00:30:55,530
Let's write down
the equations.
574
00:30:55,530 --> 00:31:02,760
So the probability that starting
from state 1, I find
575
00:31:02,760 --> 00:31:06,580
myself at state 1 n
time steps later.
576
00:31:06,580 --> 00:31:09,270
This can happen in 2 ways.
577
00:31:09,270 --> 00:31:15,440
At time n-1, I might find
myself at state 2.
578
00:31:15,440 --> 00:31:21,370
And then from state 2, I make a
transition back to state 1,
579
00:31:21,370 --> 00:31:24,050
which happens with
probability--
580
00:31:24,050 --> 00:31:25,260
why'd I put 2 there --
581
00:31:25,260 --> 00:31:27,890
anyway, 0.2.
582
00:31:27,890 --> 00:31:32,230
And another way is that from
state 1, I go to state 1 in
583
00:31:32,230 --> 00:31:38,170
n-1 steps, and then from state
1 I stay where I am, which
584
00:31:38,170 --> 00:31:42,830
happens with probability 0.5.
585
00:31:42,830 --> 00:31:44,810
So this is for R11(n).
586
00:31:44,810 --> 00:31:48,730
587
00:31:48,730 --> 00:31:54,740
Now R12(n), we can
write a similar
588
00:31:54,740 --> 00:31:56,780
recursion for this one.
589
00:31:56,780 --> 00:32:00,010
On the other hand, seems these
are probabilities.
590
00:32:00,010 --> 00:32:02,270
The state at time n is
going to be either
591
00:32:02,270 --> 00:32:04,420
state 1 or state 2.
592
00:32:04,420 --> 00:32:09,270
So these 2 numbers need to add
to 1, so we can just write
593
00:32:09,270 --> 00:32:10,520
this as 1 - R11(n).
594
00:32:10,520 --> 00:32:13,080
595
00:32:13,080 --> 00:32:19,660
And this is an enough of a
recursion to propagate R11 and
596
00:32:19,660 --> 00:32:22,010
R12 as time goes on.
597
00:32:22,010 --> 00:32:24,830
598
00:32:24,830 --> 00:32:29,105
So after n-1 transitions, either
I find myself in state
599
00:32:29,105 --> 00:32:33,910
2, and then there's a point to
transition that I go to 1, or
600
00:32:33,910 --> 00:32:37,810
I find myself in state 1, which
with that probability,
601
00:32:37,810 --> 00:32:41,230
and from there, I have
probability 0.5 of staying
602
00:32:41,230 --> 00:32:42,880
where I am.
603
00:32:42,880 --> 00:32:45,830
Now let's start calculating.
604
00:32:45,830 --> 00:32:49,500
As we discussed before, if I
start at state 1, after 0
605
00:32:49,500 --> 00:32:53,320
transitions I'm certain to be at
state , and I'm certain not
606
00:32:53,320 --> 00:32:55,100
to be at state 1.
607
00:32:55,100 --> 00:32:59,390
If I start from state 1, I'm
certain to not to be at state
608
00:32:59,390 --> 00:33:01,980
at that time, and I'm certain
that I am right
609
00:33:01,980 --> 00:33:03,520
now, it's state 1.
610
00:33:03,520 --> 00:33:09,970
After I make transition,
starting from state 1, there's
611
00:33:09,970 --> 00:33:13,790
probability 0.5 that
I stay at state 1.
612
00:33:13,790 --> 00:33:17,830
And there's probability 0.5
that I stay at state 2.
613
00:33:17,830 --> 00:33:22,060
If I were to start from state
2, the probability that I go
614
00:33:22,060 --> 00:33:25,690
to 1 in 1 time step is this
transition that has
615
00:33:25,690 --> 00:33:30,160
probability 0.2, and
the other 0.8.
616
00:33:30,160 --> 00:33:30,510
OK.
617
00:33:30,510 --> 00:33:33,850
So the calculation now becomes
more interesting, if we want
618
00:33:33,850 --> 00:33:36,920
to calculate the next term.
619
00:33:36,920 --> 00:33:41,890
How likely is that at time 2,
I find myself at state 1?
620
00:33:41,890 --> 00:33:44,610
621
00:33:44,610 --> 00:33:50,060
In order to be here at state 1,
this can happen in 2 ways.
622
00:33:50,060 --> 00:33:54,510
Either the first transition left
me there, and the second
623
00:33:54,510 --> 00:33:57,620
transition is the same.
624
00:33:57,620 --> 00:34:01,380
So these correspond to this 0.5,
that the first transition
625
00:34:01,380 --> 00:34:04,660
took me there, and the
next transition was
626
00:34:04,660 --> 00:34:07,140
also of the same kind.
627
00:34:07,140 --> 00:34:08,880
That's one possibility.
628
00:34:08,880 --> 00:34:10,510
But there's another scenario.
629
00:34:10,510 --> 00:34:15,100
In order to be at state 1
at time 2 -- this can
630
00:34:15,100 --> 00:34:17,239
also happen this way.
631
00:34:17,239 --> 00:34:19,690
So that's the event
that, after 1
632
00:34:19,690 --> 00:34:22,449
transition, I got there.
633
00:34:22,449 --> 00:34:26,920
And the next transition happened
to be this one.
634
00:34:26,920 --> 00:34:31,020
So this corresponds
to 0.5 times 0.2.
635
00:34:31,020 --> 00:34:34,250
It corresponds to taking the
1-step transition probability
636
00:34:34,250 --> 00:34:39,070
of getting there, times the
probability that from state 2
637
00:34:39,070 --> 00:34:43,070
I move to state 1, which
in this case, is 0.2.
638
00:34:43,070 --> 00:34:47,480
So basically we take this
number, multiplied with 0.2,
639
00:34:47,480 --> 00:34:50,250
and then add those 2 numbers.
640
00:34:50,250 --> 00:34:54,090
And after you add them,
you get 0.35.
641
00:34:54,090 --> 00:34:59,090
And similarly here, you're
going to get 0.65.
642
00:34:59,090 --> 00:35:02,390
And now to continue with the
recursion, we keep doing the
643
00:35:02,390 --> 00:35:02,960
same thing.
644
00:35:02,960 --> 00:35:08,290
We take this number times 0.5
plus this number times 0.2.
645
00:35:08,290 --> 00:35:10,780
Add them up, you get
the next entry.
646
00:35:10,780 --> 00:35:15,390
Keep doing that, keep doing
that, and eventually you will
647
00:35:15,390 --> 00:35:19,970
notice that the numbers
start settling into a
648
00:35:19,970 --> 00:35:23,810
limiting value at 2/7.
649
00:35:23,810 --> 00:35:25,690
And let's verify this.
650
00:35:25,690 --> 00:35:29,220
If this number is 2/7, what is
the next number going to be?
651
00:35:29,220 --> 00:35:33,720
652
00:35:33,720 --> 00:35:41,100
The next number is going to
be 2/7 -- (not 2.7) --
653
00:35:41,100 --> 00:35:42,410
it's going to be 2/7.
654
00:35:42,410 --> 00:35:44,870
That's the probability that I
find myself at that state,
655
00:35:44,870 --> 00:35:46,890
times 0.5--
656
00:35:46,890 --> 00:35:51,200
that's the next transition that
takes me to state 1 --
657
00:35:51,200 --> 00:35:53,020
plus 5/7--
658
00:35:53,020 --> 00:35:56,700
that would be the remaining
probability that I find myself
659
00:35:56,700 --> 00:35:58,180
in state 2 --
660
00:35:58,180 --> 00:36:02,760
times 1/5.
661
00:36:02,760 --> 00:36:06,360
662
00:36:06,360 --> 00:36:11,210
And so that gives
me, again, 2/7.
663
00:36:11,210 --> 00:36:15,840
So this calculation basically
illustrates, if this number
664
00:36:15,840 --> 00:36:19,360
has become 2/7, then
the next number is
665
00:36:19,360 --> 00:36:21,570
also going to be 2/7.
666
00:36:21,570 --> 00:36:24,260
And of course this number here
is going to have to be 5/7.
667
00:36:24,260 --> 00:36:26,900
668
00:36:26,900 --> 00:36:32,020
And this one would have to
be again, the same, 5/7.
669
00:36:32,020 --> 00:36:36,620
So the probability that I find
myself at state 1, after a
670
00:36:36,620 --> 00:36:41,830
long time has elapsed, settles
into some steady state value.
671
00:36:41,830 --> 00:36:44,140
So that's an interesting
phenomenon.
672
00:36:44,140 --> 00:36:46,850
We just make this observation.
673
00:36:46,850 --> 00:36:50,390
Now we can also do the
calculation about the
674
00:36:50,390 --> 00:36:53,660
probability, starting
from state 2.
675
00:36:53,660 --> 00:36:57,050
And here, you do the
calculations --
676
00:36:57,050 --> 00:36:58,460
I'm not going to do them.
677
00:36:58,460 --> 00:37:02,040
But after you do them, you find
this probability also
678
00:37:02,040 --> 00:37:07,405
settles to 2/7 and this one
also settles to 5/7.
679
00:37:07,405 --> 00:37:11,130
680
00:37:11,130 --> 00:37:15,320
So these numbers here are the
same as those numbers.
681
00:37:15,320 --> 00:37:19,770
What's the difference
between these?
682
00:37:19,770 --> 00:37:24,790
This is the probability that I
find myself at state 1 given
683
00:37:24,790 --> 00:37:27,050
that I started at 1.
684
00:37:27,050 --> 00:37:30,890
This is the probability that I
find myself at state 1 given
685
00:37:30,890 --> 00:37:34,220
that I started at state 2.
686
00:37:34,220 --> 00:37:39,110
These probabilities are the
same, no matter where I
687
00:37:39,110 --> 00:37:40,460
started from.
688
00:37:40,460 --> 00:37:45,790
So this numerical example sort
of illustrates the idea that
689
00:37:45,790 --> 00:37:51,070
after the chain has run for a
long time, what the state of
690
00:37:51,070 --> 00:37:55,010
the chain is, does not care
about the initial
691
00:37:55,010 --> 00:37:56,530
state of the chain.
692
00:37:56,530 --> 00:38:02,590
So if you start here, you know
that you're going to stay here
693
00:38:02,590 --> 00:38:05,350
for some time, a few
transitions, because this
694
00:38:05,350 --> 00:38:07,090
probability is kind of small.
695
00:38:07,090 --> 00:38:10,440
So the initial state does that's
tell you something.
696
00:38:10,440 --> 00:38:13,710
But in the very long run,
transitions of this kind are
697
00:38:13,710 --> 00:38:14,340
going to happen.
698
00:38:14,340 --> 00:38:17,510
Transitions of that kind
are going to happen.
699
00:38:17,510 --> 00:38:20,920
There's a lot of randomness
that comes in, and that
700
00:38:20,920 --> 00:38:24,820
randomness washes out any
information that could come
701
00:38:24,820 --> 00:38:28,580
from the initial state
of the system.
702
00:38:28,580 --> 00:38:33,290
We describe this situation by
saying that the Markov chain
703
00:38:33,290 --> 00:38:37,210
eventually enters
a steady state.
704
00:38:37,210 --> 00:38:41,050
Where a steady state, what
does it mean it?
705
00:38:41,050 --> 00:38:46,630
Does it mean the state itself
becomes steady and
706
00:38:46,630 --> 00:38:48,750
stops at one place?
707
00:38:48,750 --> 00:38:52,490
No, the state of the chain
keeps jumping forever.
708
00:38:52,490 --> 00:38:55,380
The state of the chain will keep
making transitions, will
709
00:38:55,380 --> 00:38:58,780
keep going back and forth
between 1 and 2.
710
00:38:58,780 --> 00:39:02,920
So the state itself, the
Xn, does not become
711
00:39:02,920 --> 00:39:04,970
steady in any sense.
712
00:39:04,970 --> 00:39:07,950
What becomes steady are
the probabilities
713
00:39:07,950 --> 00:39:09,860
that describe Xn.
714
00:39:09,860 --> 00:39:12,900
That is, after a long time
elapses, the probability that
715
00:39:12,900 --> 00:39:19,700
you find yourself at state 1
becomes a constant 2/7, and
716
00:39:19,700 --> 00:39:21,520
the probability that you
find yourself in
717
00:39:21,520 --> 00:39:23,810
state 2 becomes a constant.
718
00:39:23,810 --> 00:39:28,000
So jumps will keep happening,
but at any given time, if you
719
00:39:28,000 --> 00:39:30,590
ask what's the probability that
right now I am at state
720
00:39:30,590 --> 00:39:34,630
1, the answer is going
to be 2/7.
721
00:39:34,630 --> 00:39:37,650
Incidentally, do the numbers
sort of makes sense?
722
00:39:37,650 --> 00:39:42,270
Why is this number bigger
than that number?
723
00:39:42,270 --> 00:39:46,500
Well, this state is a little
more sticky than that state.
724
00:39:46,500 --> 00:39:50,000
Once you enter here, it's kind
of harder to get out.
725
00:39:50,000 --> 00:39:53,380
So when you enter here, you
spend a lot of time here.
726
00:39:53,380 --> 00:39:56,240
This one is easier to get out,
because the probability is
727
00:39:56,240 --> 00:40:00,370
0.5, so when you enter there,
you tend to get out faster.
728
00:40:00,370 --> 00:40:04,150
So you keep moving from one to
the other, but you tend to
729
00:40:04,150 --> 00:40:08,510
spend more time on that state,
and this is reflected in this
730
00:40:08,510 --> 00:40:10,930
probability being bigger
than that one.
731
00:40:10,930 --> 00:40:14,540
So no matter where you start,
there's 5/7 probability of
732
00:40:14,540 --> 00:40:18,650
being here, 2/7 probability
being there.
733
00:40:18,650 --> 00:40:20,480
So there were some really
nice things that
734
00:40:20,480 --> 00:40:24,730
happened in this example.
735
00:40:24,730 --> 00:40:28,830
The question is, whether things
are always as nice for
736
00:40:28,830 --> 00:40:30,410
general Markov chains.
737
00:40:30,410 --> 00:40:33,380
The two nice things that
happened where the following--
738
00:40:33,380 --> 00:40:36,020
as we keep doing this
calculation, this number
739
00:40:36,020 --> 00:40:37,660
settles to something.
740
00:40:37,660 --> 00:40:39,620
The limit exists.
741
00:40:39,620 --> 00:40:42,520
The other thing that happens
is that this number is the
742
00:40:42,520 --> 00:40:45,740
same as that number, which means
that the initial state
743
00:40:45,740 --> 00:40:47,280
does not matter.
744
00:40:47,280 --> 00:40:50,130
Is this always the case?
745
00:40:50,130 --> 00:40:54,570
Is it always the case that as
n goes to infinity, the
746
00:40:54,570 --> 00:40:58,490
transition probabilities
converge to something?
747
00:40:58,490 --> 00:41:02,680
And if they do converge to
something, is it the case that
748
00:41:02,680 --> 00:41:07,830
the limit is not affected by the
initial state i at which
749
00:41:07,830 --> 00:41:09,400
the chain started?
750
00:41:09,400 --> 00:41:12,970
So mathematically speaking, the
question we are raising is
751
00:41:12,970 --> 00:41:19,180
whether Rij(n) converges
to something.
752
00:41:19,180 --> 00:41:25,440
And whether that something to
which it converges to has only
753
00:41:25,440 --> 00:41:26,780
to do with j.
754
00:41:26,780 --> 00:41:30,680
It's the probability that you
find yourself at state j, and
755
00:41:30,680 --> 00:41:34,170
that probability doesn't care
about the initial state.
756
00:41:34,170 --> 00:41:36,950
So it's the question of whether
the initial state gets
757
00:41:36,950 --> 00:41:39,970
forgotten in the long run.
758
00:41:39,970 --> 00:41:46,260
So the answer is that usually,
or for nice chains, both of
759
00:41:46,260 --> 00:41:49,420
these things will be true.
760
00:41:49,420 --> 00:41:51,500
You get the limit which
does not depend
761
00:41:51,500 --> 00:41:52,900
on the initial state.
762
00:41:52,900 --> 00:41:59,020
But if your chain has some
peculiar or unique structure,
763
00:41:59,020 --> 00:42:01,160
this might not happen.
764
00:42:01,160 --> 00:42:03,905
So let's think first about
the issue of convergence.
765
00:42:03,905 --> 00:42:06,980
766
00:42:06,980 --> 00:42:12,720
So convergence, as n goes to
infinity at a steady value,
767
00:42:12,720 --> 00:42:14,520
really means the following.
768
00:42:14,520 --> 00:42:18,710
If I tell you a lot of time has
passed, then you tell me,
769
00:42:18,710 --> 00:42:21,220
OK, the state of the
probabilities are equal to
770
00:42:21,220 --> 00:42:25,630
that value without having
to consult your clock.
771
00:42:25,630 --> 00:42:29,340
If you don't have convergence,
it means that Rij can keep
772
00:42:29,340 --> 00:42:32,230
going up and down, without
settling to something.
773
00:42:32,230 --> 00:42:35,640
So in order for you to tell me
the value of Rij, you need to
774
00:42:35,640 --> 00:42:38,200
consult your clock to
check if, right now,
775
00:42:38,200 --> 00:42:40,670
it's up or is it down.
776
00:42:40,670 --> 00:42:43,230
So there's some kind of periodic
behavior that you
777
00:42:43,230 --> 00:42:46,020
might get when you do not get
convergence, and this example
778
00:42:46,020 --> 00:42:47,490
here illustrates it.
779
00:42:47,490 --> 00:42:50,120
So what's happened
in this example?
780
00:42:50,120 --> 00:42:54,470
Starting from state 2, next time
you go here, or there,
781
00:42:54,470 --> 00:42:56,760
with probability half.
782
00:42:56,760 --> 00:43:00,820
And then next time, no matter
where you are, you move back
783
00:43:00,820 --> 00:43:02,200
to state 2.
784
00:43:02,200 --> 00:43:05,950
So this chain has some
randomness, but the randomness
785
00:43:05,950 --> 00:43:08,050
is kind of limited type.
786
00:43:08,050 --> 00:43:09,260
You go out, you come in.
787
00:43:09,260 --> 00:43:10,500
You go out, you come in.
788
00:43:10,500 --> 00:43:14,290
So there's a periodic pattern
that gets repeated.
789
00:43:14,290 --> 00:43:19,850
It means that if you start at
state 2 after an even number
790
00:43:19,850 --> 00:43:24,100
of steps, you are certain
to be back at state 2.
791
00:43:24,100 --> 00:43:26,730
So this probability here is 1.
792
00:43:26,730 --> 00:43:30,430
On the other hand, if the number
of transitions is odd,
793
00:43:30,430 --> 00:43:33,900
there's no way that you can
be at your initial state.
794
00:43:33,900 --> 00:43:37,150
If you start here, at even times
you would be here, at
795
00:43:37,150 --> 00:43:39,570
odd times you would
be there or there.
796
00:43:39,570 --> 00:43:42,170
So this probability is 0.
797
00:43:42,170 --> 00:43:46,540
As n goes to infinity, these
probabilities, the n-step
798
00:43:46,540 --> 00:43:49,220
transition probability does
not converge to anything.
799
00:43:49,220 --> 00:43:51,950
It keeps alternating
between 0 and 1.
800
00:43:51,950 --> 00:43:54,090
So convergence fails.
801
00:43:54,090 --> 00:43:57,440
This is the main mechanism by
which convergence can fail if
802
00:43:57,440 --> 00:43:59,720
your chain has a periodic
structure.
803
00:43:59,720 --> 00:44:03,520
And we're going to discuss next
time that, if periodicity
804
00:44:03,520 --> 00:44:08,790
absent, then we don't have an
issue with convergence.
805
00:44:08,790 --> 00:44:13,950
The second question if we have
convergence, whether the
806
00:44:13,950 --> 00:44:16,850
initial state matters or not.
807
00:44:16,850 --> 00:44:19,560
In the previous chain, where you
could keep going back and
808
00:44:19,560 --> 00:44:22,830
forth between states 1 and 2
numerically, one finds that
809
00:44:22,830 --> 00:44:25,040
the initial state
does not matter.
810
00:44:25,040 --> 00:44:27,460
But you can think of situations
where the initial
811
00:44:27,460 --> 00:44:30,070
state does matter.
812
00:44:30,070 --> 00:44:33,120
Look at this chain here.
813
00:44:33,120 --> 00:44:37,520
If you start at state 1, you
stay at state 1 forever.
814
00:44:37,520 --> 00:44:39,840
There's no way to escape.
815
00:44:39,840 --> 00:44:46,040
So this means that R11(n)
is 1 for all n.
816
00:44:46,040 --> 00:44:50,390
If you start at state 3, you
will be moving between stage 3
817
00:44:50,390 --> 00:44:54,400
and 4, but there's no way to
go in that direction, so
818
00:44:54,400 --> 00:44:57,730
there's no way that
you go to state 1.
819
00:44:57,730 --> 00:45:01,410
And for that reason,
R31 is 0 for all n.
820
00:45:01,410 --> 00:45:06,570
821
00:45:06,570 --> 00:45:16,100
OK So this is a case where the
initial state matters.
822
00:45:16,100 --> 00:45:21,640
R11 goes to a limit, as
n goes to infinity,
823
00:45:21,640 --> 00:45:22,720
because it's constant.
824
00:45:22,720 --> 00:45:25,460
It's always 1 so
the limit is 1.
825
00:45:25,460 --> 00:45:28,050
R31 also has a limit.
826
00:45:28,050 --> 00:45:30,010
It's 0 for all times.
827
00:45:30,010 --> 00:45:32,830
So these are the long term
probabilities of finding
828
00:45:32,830 --> 00:45:34,730
yourself at state 1.
829
00:45:34,730 --> 00:45:37,990
But those long-term
probabilities are affected by
830
00:45:37,990 --> 00:45:39,470
where you started.
831
00:45:39,470 --> 00:45:41,830
If you start here, you're sure
that's, in the long term,
832
00:45:41,830 --> 00:45:42,860
you'll be here.
833
00:45:42,860 --> 00:45:45,260
If you start here, you're sure
that, in the long term, you
834
00:45:45,260 --> 00:45:47,120
will not be there.
835
00:45:47,120 --> 00:45:50,550
So the initial state
does matter here.
836
00:45:50,550 --> 00:45:53,780
And this is a situation where
certain states are not
837
00:45:53,780 --> 00:45:57,120
accessible from certain other
states, so it has something to
838
00:45:57,120 --> 00:46:00,150
do with the graph structure
of our Markov chain.
839
00:46:00,150 --> 00:46:04,640
Finally let's answer this
question here, at
840
00:46:04,640 --> 00:46:07,560
least for large n's.
841
00:46:07,560 --> 00:46:12,140
What do you think is going to
happen in the long term if you
842
00:46:12,140 --> 00:46:14,555
start at state 2?
843
00:46:14,555 --> 00:46:18,840
If you start at state 2, you
may stay at state 2 for a
844
00:46:18,840 --> 00:46:22,860
random amount of time, but
eventually this transition
845
00:46:22,860 --> 00:46:25,680
will happen, or that transition
would happen.
846
00:46:25,680 --> 00:46:30,720
Because of the symmetry, you are
as likely to escape from
847
00:46:30,720 --> 00:46:34,170
state 2 in this direction, or in
that direction, so there's
848
00:46:34,170 --> 00:46:37,510
probability 1/2 that, when the
transition happens, the
849
00:46:37,510 --> 00:46:40,020
transition happens in
that direction.
850
00:46:40,020 --> 00:46:47,660
So for large N, you're
certain that the
851
00:46:47,660 --> 00:46:50,640
transition does happen.
852
00:46:50,640 --> 00:46:54,380
And given that the transition
has happened, it has
853
00:46:54,380 --> 00:46:57,560
probability 1/2 that it has
gone that particular way.
854
00:46:57,560 --> 00:47:00,380
So clearly here, you see that
the probability of finding
855
00:47:00,380 --> 00:47:03,770
yourself in a particular state
is very much affected by where
856
00:47:03,770 --> 00:47:05,400
you started from.
857
00:47:05,400 --> 00:47:09,310
So what we want to do next is
to abstract from these two
858
00:47:09,310 --> 00:47:13,550
examples and describe the
general structural properties
859
00:47:13,550 --> 00:47:16,360
that have to do with
periodicity, and that have to
860
00:47:16,360 --> 00:47:18,990
do with what happened here with
certain states, not being
861
00:47:18,990 --> 00:47:20,720
accessible from the others.
862
00:47:20,720 --> 00:47:23,530
We're going to leave periodicity
for next time.
863
00:47:23,530 --> 00:47:25,380
But let's talk about
the second kind of
864
00:47:25,380 --> 00:47:28,540
phenomenon that we have.
865
00:47:28,540 --> 00:47:32,420
So here, what we're going to do
is to classify the states
866
00:47:32,420 --> 00:47:35,230
in a transition diagram
into two types,
867
00:47:35,230 --> 00:47:38,000
recurrent and transient.
868
00:47:38,000 --> 00:47:41,330
So a state is said to
be recurrent if the
869
00:47:41,330 --> 00:47:43,560
following is true.
870
00:47:43,560 --> 00:47:49,470
If you start from the state i,
you can go to some places, but
871
00:47:49,470 --> 00:47:55,390
no matter where you go, there
is a way of coming back.
872
00:47:55,390 --> 00:47:59,390
So what's an example for
the recurrent state?
873
00:47:59,390 --> 00:48:02,000
This one.
874
00:48:02,000 --> 00:48:04,510
Starting from here, you
can go elsewhere.
875
00:48:04,510 --> 00:48:06,510
You can go to state 7.
876
00:48:06,510 --> 00:48:08,540
You can go to state 6.
877
00:48:08,540 --> 00:48:11,050
That's all where
you can go to.
878
00:48:11,050 --> 00:48:15,730
But no matter where you go,
there is a path that can take
879
00:48:15,730 --> 00:48:17,190
you back there.
880
00:48:17,190 --> 00:48:20,640
So no matter where you go, there
is a chance, and there
881
00:48:20,640 --> 00:48:23,470
is a way for returning
where you started.
882
00:48:23,470 --> 00:48:25,770
Those states we call
recurrent.
883
00:48:25,770 --> 00:48:28,750
And by this, 8 is recurrent.
884
00:48:28,750 --> 00:48:31,960
All of these are recurrent.
885
00:48:31,960 --> 00:48:34,080
So this is recurrent,
this is recurrent.
886
00:48:34,080 --> 00:48:36,570
And this state 5 is
also recurrent.
887
00:48:36,570 --> 00:48:40,080
You cannot go anywhere from
5 except to 5 itself.
888
00:48:40,080 --> 00:48:43,860
Wherever you can go, you can
go back to where you start.
889
00:48:43,860 --> 00:48:45,830
So this is recurrent.
890
00:48:45,830 --> 00:48:49,190
If it is not the recurrent, we
say that it is transient.
891
00:48:49,190 --> 00:48:50,890
So what does transient mean?
892
00:48:50,890 --> 00:48:53,900
You need to take this
definition, and reverse it.
893
00:48:53,900 --> 00:48:58,860
Transient means that, starting
from i, there is a place to
894
00:48:58,860 --> 00:49:05,010
which you could go, and from
which you cannot return.
895
00:49:05,010 --> 00:49:07,160
If it's recurrent, anywhere
you go, you
896
00:49:07,160 --> 00:49:09,170
can always come back.
897
00:49:09,170 --> 00:49:12,100
Transient means there are places
where you can go from
898
00:49:12,100 --> 00:49:14,270
which you cannot come back.
899
00:49:14,270 --> 00:49:18,320
So state 1 is recurrent -
because starting from here,
900
00:49:18,320 --> 00:49:20,880
there's a possibility that
you get there, and then
901
00:49:20,880 --> 00:49:22,310
there's no way back.
902
00:49:22,310 --> 00:49:26,120
State 4 is recurrent, starting
from 4, there's somewhere you
903
00:49:26,120 --> 00:49:28,520
can go and--
904
00:49:28,520 --> 00:49:30,260
sorry, transient, correct.
905
00:49:30,260 --> 00:49:33,180
State 4 is transient starting
from here, there are places
906
00:49:33,180 --> 00:49:36,670
where you could go, and from
which you cannot come back.
907
00:49:36,670 --> 00:49:40,380
And in this particular diagram,
all these 4 states
908
00:49:40,380 --> 00:49:43,110
are transients.
909
00:49:43,110 --> 00:49:49,800
Now if the state is transient,
it means that there is a way
910
00:49:49,800 --> 00:49:53,150
to go somewhere where you're
going to get stuck and not to
911
00:49:53,150 --> 00:49:54,840
be able to come.
912
00:49:54,840 --> 00:49:59,350
As long as your state keeps
circulating around here,
913
00:49:59,350 --> 00:50:02,460
eventually one of these
transitions is going to
914
00:50:02,460 --> 00:50:05,820
happen, and once that happens,
then there's no way that you
915
00:50:05,820 --> 00:50:06,960
can come back.
916
00:50:06,960 --> 00:50:11,360
So that transient state will
be visited only a finite
917
00:50:11,360 --> 00:50:12,650
number of times.
918
00:50:12,650 --> 00:50:15,100
You will not be able
to return to it.
919
00:50:15,100 --> 00:50:17,760
And in the long run, you're
certain that you're going to
920
00:50:17,760 --> 00:50:22,150
get out of the transient states,
and get to some class
921
00:50:22,150 --> 00:50:25,100
of recurrent states, and
get stuck forever.
922
00:50:25,100 --> 00:50:29,050
So, let's see, in this diagram,
if I start here,
923
00:50:29,050 --> 00:50:32,580
could I stay in this lump
of states forever?
924
00:50:32,580 --> 00:50:35,780
Well as long as I'm staying in
this type of states, I would
925
00:50:35,780 --> 00:50:40,000
keep visiting states 1 and 2
Each time that I visit state
926
00:50:40,000 --> 00:50:41,480
2, there's going to be positive
927
00:50:41,480 --> 00:50:43,550
probability that I escape.
928
00:50:43,550 --> 00:50:47,935
So in the long run, if I were
to stay here, I would visit
929
00:50:47,935 --> 00:50:50,130
state 2 an infinite number
of times, and I would get
930
00:50:50,130 --> 00:50:52,210
infinite chances to escape.
931
00:50:52,210 --> 00:50:56,840
But if you have infinite chances
to escape, eventually
932
00:50:56,840 --> 00:50:57,810
you will escape.
933
00:50:57,810 --> 00:51:01,760
So you are certain that with
probability 1, starting from
934
00:51:01,760 --> 00:51:05,050
here, you're going to move
either to those states, or to
935
00:51:05,050 --> 00:51:06,150
those states.
936
00:51:06,150 --> 00:51:09,710
So starting from transient
states, you only stay at the
937
00:51:09,710 --> 00:51:14,660
transient states for random
but finite amount of time.
938
00:51:14,660 --> 00:51:19,000
And after that happens,
you end up in a class
939
00:51:19,000 --> 00:51:20,180
of recurrent states.
940
00:51:20,180 --> 00:51:23,090
And when I say class, what they
mean is that, in this
941
00:51:23,090 --> 00:51:26,440
picture, I divide the recurrent
states into 2
942
00:51:26,440 --> 00:51:28,280
classes, or categories.
943
00:51:28,280 --> 00:51:30,220
What's special about them?
944
00:51:30,220 --> 00:51:31,300
These states are recurrent.
945
00:51:31,300 --> 00:51:33,000
These states are recurrent.
946
00:51:33,000 --> 00:51:35,150
But there's no communication
between the 2.
947
00:51:35,150 --> 00:51:36,780
If you start here, you're
stuck here.
948
00:51:36,780 --> 00:51:39,750
If you start here, you
are stuck there.
949
00:51:39,750 --> 00:51:42,970
And this is a case where the
initial state does matter,
950
00:51:42,970 --> 00:51:45,180
because if you start here,
you get stuck here.
951
00:51:45,180 --> 00:51:47,130
You start here, you
get stuck there.
952
00:51:47,130 --> 00:51:49,970
So depending on the initial
state, that's going to affect
953
00:51:49,970 --> 00:51:52,590
the long term behavior
of your chain.
954
00:51:52,590 --> 00:51:55,470
So the guess you can make at
this point is that, for the
955
00:51:55,470 --> 00:51:59,210
initial state to not matter,
we should not have multiple
956
00:51:59,210 --> 00:52:00,160
recurrent classes.
957
00:52:00,160 --> 00:52:01,710
We should have only 1.
958
00:52:01,710 --> 00:52:04,030
But we're going to get back
to this point next time.
959
00:52:04,030 --> 00:52:05,280