1
00:00:01,680 --> 00:00:04,660
In this video, we
continue our exploration
2
00:00:04,660 --> 00:00:06,650
of quantities of
interest associated
3
00:00:06,650 --> 00:00:09,740
with the short-term
behavior of Markov chain.
4
00:00:09,740 --> 00:00:13,060
This time, we suppose that we
have a Markov chain composed
5
00:00:13,060 --> 00:00:16,700
of a single recurrent
class, such as this one.
6
00:00:16,700 --> 00:00:18,160
Here, we have our
recurrent class.
7
00:00:18,160 --> 00:00:22,830
And these are transient states.
8
00:00:22,830 --> 00:00:24,770
We also assume that
we're interested
9
00:00:24,770 --> 00:00:27,820
in a specific recurrent
state-- let's say 9.
10
00:00:27,820 --> 00:00:31,170
So this is my state s.
11
00:00:31,170 --> 00:00:33,620
And we will assume that
the Markov chain starts
12
00:00:33,620 --> 00:00:36,120
from a given initial
state, state i,
13
00:00:36,120 --> 00:00:40,470
and assume that i is 1.
14
00:00:40,470 --> 00:00:45,280
Now the question we ask is,
given that you started in 1,
15
00:00:45,280 --> 00:00:50,070
how long is it going to take
to reach 9 for the first time?
16
00:00:50,070 --> 00:00:51,840
We know this is a
recurrent class,
17
00:00:51,840 --> 00:00:53,950
so we know that the
Markov chain eventually
18
00:00:53,950 --> 00:00:57,610
will come to that class
and circulate here.
19
00:00:57,610 --> 00:01:00,950
So you know that the Markov
chain will get to that state 9.
20
00:01:00,950 --> 00:01:02,880
You are interested
in knowing when
21
00:01:02,880 --> 00:01:04,819
it does so for the first time.
22
00:01:04,819 --> 00:01:06,740
Of course, we don't
know for sure.
23
00:01:06,740 --> 00:01:07,810
This is random.
24
00:01:07,810 --> 00:01:09,420
It is a random variable.
25
00:01:09,420 --> 00:01:12,640
And let's try to calculate
the expected value
26
00:01:12,640 --> 00:01:14,220
of this random variable.
27
00:01:14,220 --> 00:01:17,230
More precisely, this is
what we would like to do.
28
00:01:17,230 --> 00:01:21,090
We would like to find the
mean first passage time from i
29
00:01:21,090 --> 00:01:24,080
to s, from any starting state i.
30
00:01:24,080 --> 00:01:27,440
Here, we're going to
illustrate that with 1.
31
00:01:27,440 --> 00:01:30,030
And mathematically, this
is what we have here.
32
00:01:30,030 --> 00:01:33,440
So here again, what you have
is that the Markov chain
33
00:01:33,440 --> 00:01:36,890
started in state i, at time 0.
34
00:01:36,890 --> 00:01:39,950
And now you are looking
at this set here.
35
00:01:39,950 --> 00:01:43,620
And the set records
all the time n such
36
00:01:43,620 --> 00:01:48,570
that your Markov chain, Xn,
is in the state s of interest.
37
00:01:48,570 --> 00:01:52,630
So out of this set, you're
looking at the minimum, n,
38
00:01:52,630 --> 00:01:54,030
that verifies that.
39
00:01:54,030 --> 00:01:56,520
So this is going to
be a random variable.
40
00:01:56,520 --> 00:01:59,100
And you take the
expected value of that.
41
00:01:59,100 --> 00:02:02,920
And this is what we call the
mean first passage time from i
42
00:02:02,920 --> 00:02:04,230
to s.
43
00:02:04,230 --> 00:02:07,040
Again, this is the
expected number of steps
44
00:02:07,040 --> 00:02:11,200
in order to get from i
to s for the first time.
45
00:02:11,200 --> 00:02:15,830
So how do we go about
calculating such a quantity?
46
00:02:15,830 --> 00:02:18,590
Well, let us think a little bit.
47
00:02:18,590 --> 00:02:20,700
We are looking at
the event that you
48
00:02:20,700 --> 00:02:25,200
will visit 9 from 1
for the first time.
49
00:02:25,200 --> 00:02:27,930
What happened
after visiting 9 is
50
00:02:27,930 --> 00:02:30,420
of no relevance in
our calculation.
51
00:02:30,420 --> 00:02:34,510
In other words, our calculation
will never involve this arc
52
00:02:34,510 --> 00:02:37,200
and will not involve
this arc either.
53
00:02:37,200 --> 00:02:39,079
Because in order
for the Markov chain
54
00:02:39,079 --> 00:02:42,680
to traverse this
arc or this one,
55
00:02:42,680 --> 00:02:45,980
it would have to visit 9 first.
56
00:02:45,980 --> 00:02:49,740
So what it means is that we
can forget about this arc,
57
00:02:49,740 --> 00:02:52,860
and we can forget about
this arc-- in a sense,
58
00:02:52,860 --> 00:02:54,890
that they don't matter
in the calculation
59
00:02:54,890 --> 00:02:57,550
of the mean first
passage time to s.
60
00:02:57,550 --> 00:03:00,230
Now, removing
these arcs entirely
61
00:03:00,230 --> 00:03:02,290
means that you would
have to increase
62
00:03:02,290 --> 00:03:08,050
the probability of that self
transition from 0.2 to 1.
63
00:03:08,050 --> 00:03:09,500
So what do you get?
64
00:03:09,500 --> 00:03:12,660
Well, you get this
following graph.
65
00:03:12,660 --> 00:03:14,980
We have removed
these arcs, and we
66
00:03:14,980 --> 00:03:18,210
have increased the
probability 0.2 here to 1.
67
00:03:18,210 --> 00:03:21,860
And again, we have
argued that calculating
68
00:03:21,860 --> 00:03:26,420
the mean first passage time
from i to s in this graph
69
00:03:26,420 --> 00:03:30,270
is the same as doing the
same thing for this graph.
70
00:03:30,270 --> 00:03:34,610
But here, we have a
very special structure.
71
00:03:34,610 --> 00:03:39,030
We have only one recurrent
state left, which is state 9.
72
00:03:39,030 --> 00:03:41,920
And that state is
an absorbing state.
73
00:03:41,920 --> 00:03:43,390
All the other
state-- this one was
74
00:03:43,390 --> 00:03:44,960
transient, transient, transient.
75
00:03:44,960 --> 00:03:47,280
This one becomes
a transient state.
76
00:03:47,280 --> 00:03:50,490
And this one is also a transient
state in this new transition
77
00:03:50,490 --> 00:03:52,150
probability diagram.
78
00:03:52,150 --> 00:03:55,630
So we get a situation where
we have one single absorbing
79
00:03:55,630 --> 00:03:58,750
state, and we are
interested in calculating
80
00:03:58,750 --> 00:04:03,790
the probability starting from
i to reach that absorbing state
81
00:04:03,790 --> 00:04:07,140
and also the number
of steps it takes
82
00:04:07,140 --> 00:04:11,140
to do that, which is what we
did in the previous video.
83
00:04:11,140 --> 00:04:14,030
So let us repeat what
we had seen before.
84
00:04:14,030 --> 00:04:19,010
So this is going to be the
unique solution to this system.
85
00:04:19,010 --> 00:04:20,820
t of s equals 0, of course.
86
00:04:20,820 --> 00:04:22,900
Since you start from
s, you are in s.
87
00:04:22,900 --> 00:04:26,180
And so the amount of time
it takes to get to s is 0.
88
00:04:26,180 --> 00:04:29,390
And otherwise, for all the
other states that are not s,
89
00:04:29,390 --> 00:04:33,100
this is the resulting system
of equation that we have.
90
00:04:33,100 --> 00:04:35,190
And the unique
solution of this system
91
00:04:35,190 --> 00:04:40,080
gives you the result of finding
the expected amount of time
92
00:04:40,080 --> 00:04:43,100
to go to the absorbing state s.
93
00:04:43,100 --> 00:04:45,290
So using this
simple trick, we've
94
00:04:45,290 --> 00:04:47,960
been able to use the
previous calculation
95
00:04:47,960 --> 00:04:50,610
to calculate something else.
96
00:04:50,610 --> 00:04:53,380
Let us consider a
related question, which
97
00:04:53,380 --> 00:04:57,490
we called the mean
recurrence time of s.
98
00:04:57,490 --> 00:05:00,220
Here, let's go back
to the original graph.
99
00:05:00,220 --> 00:05:02,480
And this is our state s.
100
00:05:02,480 --> 00:05:04,440
And now the question
is the following--
101
00:05:04,440 --> 00:05:08,210
given that you are in s,
what is the amount of time
102
00:05:08,210 --> 00:05:10,540
it will take for
the Markov chain
103
00:05:10,540 --> 00:05:14,080
to return to s for
the first time?
104
00:05:14,080 --> 00:05:17,740
So for this question, the
Markov chain is currently in 9.
105
00:05:17,740 --> 00:05:20,320
And you ask yourself,
how long is it
106
00:05:20,320 --> 00:05:25,620
going to take, once you
leave 9, to return to 9?
107
00:05:25,620 --> 00:05:27,730
Here again, we
don't know for sure.
108
00:05:27,730 --> 00:05:29,000
It's a random variable.
109
00:05:29,000 --> 00:05:31,810
And we are interested
in the expectation
110
00:05:31,810 --> 00:05:33,900
of that random variable--
in other words,
111
00:05:33,900 --> 00:05:37,100
in the expected number of
steps it takes in order
112
00:05:37,100 --> 00:05:39,310
to get back to 9
once you are in 9.
113
00:05:39,310 --> 00:05:43,380
And this is what we mean by
the mean recurrence time of s.
114
00:05:43,380 --> 00:05:46,780
And mathematically,
this is what it is.
115
00:05:46,780 --> 00:05:50,650
It is almost the same formula
as the one that you had here,
116
00:05:50,650 --> 00:05:53,750
except that here you
have n greater than/equal
117
00:05:53,750 --> 00:05:56,960
to 1 as opposed to n
greater than/equal to 0.
118
00:05:56,960 --> 00:06:01,130
It simply means that you're not
interested in the first time
119
00:06:01,130 --> 00:06:03,560
that you're in s,
because you start from s.
120
00:06:03,560 --> 00:06:06,590
So you want to have n
greater than or equal to 1. n
121
00:06:06,590 --> 00:06:10,600
equals 0 would not work.
ts star would have been 0.
122
00:06:10,600 --> 00:06:13,540
So how would you
solve that problem?
123
00:06:13,540 --> 00:06:17,750
Well, here again, you use the
same trick that we used before.
124
00:06:17,750 --> 00:06:19,990
And think in terms of tree.
125
00:06:19,990 --> 00:06:23,440
Let's look at the
Markov chain in state 9.
126
00:06:23,440 --> 00:06:24,940
What can happen next?
127
00:06:24,940 --> 00:06:32,034
From 9, it can go to
3 or it can go to 5.
128
00:06:35,860 --> 00:06:39,740
Or it can jump on itself.
129
00:06:39,740 --> 00:06:41,710
So this is with a
probability 0.2.
130
00:06:41,710 --> 00:06:44,130
This is with a probability 0.2.
131
00:06:44,130 --> 00:06:47,820
And this is with a
probability of 0.6.
132
00:06:47,820 --> 00:06:50,640
And now, after you
make one transmission--
133
00:06:50,640 --> 00:06:56,680
so let's-- you are in 3 now--
what is the time to get to 9?
134
00:06:56,680 --> 00:07:00,600
Well, it's exactly
t3-- where the t
135
00:07:00,600 --> 00:07:04,930
is, the solution here
of that previous system.
136
00:07:04,930 --> 00:07:07,170
What about from 5?
137
00:07:07,170 --> 00:07:08,890
t5.
138
00:07:08,890 --> 00:07:10,560
And what about from 9?
139
00:07:10,560 --> 00:07:11,470
Well, t9.
140
00:07:11,470 --> 00:07:14,120
And t9 was 0.
141
00:07:14,120 --> 00:07:17,310
So from that tree,
what you have is
142
00:07:17,310 --> 00:07:33,330
t9 star will be 0.2 times t3
plus 0.6 times t5 plus 0.2
143
00:07:33,330 --> 00:07:37,420
times t9 plus, of
course, 1, because you
144
00:07:37,420 --> 00:07:39,640
have done one transition.
145
00:07:39,640 --> 00:07:46,380
Where, again, this value of
t3, t5, and t9 are the ones
146
00:07:46,380 --> 00:07:49,030
corresponding to
this solution here.
147
00:07:49,030 --> 00:07:52,990
So of course, t9 is 0.
148
00:07:52,990 --> 00:07:56,040
So what we have shown
here, that in general,
149
00:07:56,040 --> 00:07:58,380
this is actually
what you would have.
150
00:07:58,380 --> 00:08:00,850
And this is exactly what
we have written here.
151
00:08:00,850 --> 00:08:09,100
ts star is 1 plus the summation
of psj of t of j, where t of j
152
00:08:09,100 --> 00:08:11,900
is the solution to this system.
153
00:08:11,900 --> 00:08:14,750
So again, you started from 9.
154
00:08:14,750 --> 00:08:17,140
You have to do a
transition first.
155
00:08:17,140 --> 00:08:19,640
You can do a transition
unto yourself.
156
00:08:19,640 --> 00:08:23,190
You can go to 3 or
you can go to 6.
157
00:08:23,190 --> 00:08:25,520
After that transition,
which stands for the number
158
00:08:25,520 --> 00:08:29,180
1 here, what happens
is you're trying
159
00:08:29,180 --> 00:08:31,560
to find the solution
to this system, which
160
00:08:31,560 --> 00:08:34,419
is the mean first passage time,
from the current state where
161
00:08:34,419 --> 00:08:36,990
you are, to 9.
162
00:08:36,990 --> 00:08:39,340
And then when you put
these two things together,
163
00:08:39,340 --> 00:08:43,209
you get the mean
recurrence time of 9.