1
00:00:00,000 --> 00:00:00,590
2
00:00:00,590 --> 00:00:01,420
Hi, everyone.
3
00:00:01,420 --> 00:00:03,270
Today, I'm going to talk
about Markov Chain
4
00:00:03,270 --> 00:00:05,050
Practice number one.
5
00:00:05,050 --> 00:00:07,240
Before we start, let's
first take a look
6
00:00:07,240 --> 00:00:09,080
at this Markov chain.
7
00:00:09,080 --> 00:00:11,380
This Markov chain
has six states.
8
00:00:11,380 --> 00:00:13,720
In this problem, we always
assume the process
9
00:00:13,720 --> 00:00:15,980
starts from state S0.
10
00:00:15,980 --> 00:00:18,470
On the first trial, the process
can either make a
11
00:00:18,470 --> 00:00:23,080
transition from S0 to S1 with
probability 1/3 or from S0 to
12
00:00:23,080 --> 00:00:26,740
S3 with probability 1/3 third
or from S0 to S5 with
13
00:00:26,740 --> 00:00:28,460
probability 1/3.
14
00:00:28,460 --> 00:00:31,310
If on the first trial, the
process makes the transition
15
00:00:31,310 --> 00:00:35,870
from S0 to S1 or from S0 to S5,
it will always be stuck in
16
00:00:35,870 --> 00:00:40,110
either S1 or S5 forever, because
both of the states S1
17
00:00:40,110 --> 00:00:44,220
and S5 have a self-transition
probability of one.
18
00:00:44,220 --> 00:00:47,160
On the other hand, if on the
first trial, the process makes
19
00:00:47,160 --> 00:00:50,950
the transition from S0 to S3, it
can then either transition
20
00:00:50,950 --> 00:00:53,920
to the left or transition
to the right or make
21
00:00:53,920 --> 00:00:57,430
self-transition back
to the state S3.
22
00:00:57,430 --> 00:01:00,570
If the process ever enters the
left of the chain, it will
23
00:01:00,570 --> 00:01:03,120
never be able to come
to the right.
24
00:01:03,120 --> 00:01:06,030
On the other hand, if the
process ever enters the right
25
00:01:06,030 --> 00:01:10,280
of the chain, it would never
be able to go to the left.
26
00:01:10,280 --> 00:01:13,200
For part A of the problem,
we have to calculate the
27
00:01:13,200 --> 00:01:17,140
probability that the process
enter S2 for the first time at
28
00:01:17,140 --> 00:01:18,790
the case trial.
29
00:01:18,790 --> 00:01:22,890
First, notice that it would take
at least two trials for
30
00:01:22,890 --> 00:01:26,980
the process to make a transition
from S0 to S2.
31
00:01:26,980 --> 00:01:30,830
Therefore, for k equal to 1,
the probability of ak is
32
00:01:30,830 --> 00:01:32,080
simply equal to 0.
33
00:01:32,080 --> 00:01:34,740
34
00:01:34,740 --> 00:01:41,860
For k equal to 1, probability
of a1 is equal to 0.
35
00:01:41,860 --> 00:01:44,710
36
00:01:44,710 --> 00:01:49,250
Then for k equal to 2, 3 and on,
the probability that the
37
00:01:49,250 --> 00:01:52,870
process enters S2 for the first
time at a case trial is
38
00:01:52,870 --> 00:01:56,600
equivalent to the probability
that the process first makes a
39
00:01:56,600 --> 00:02:01,430
transition from S0 to S3 and
then stays in S3 for the next
40
00:02:01,430 --> 00:02:05,060
two k minus 2 trials and finally
makes a transition
41
00:02:05,060 --> 00:02:09,520
from S3 to S2 on
the kth trial.
42
00:02:09,520 --> 00:02:12,730
So let's write this out.
43
00:02:12,730 --> 00:02:21,530
For k equal to 2, 3, and on, the
probability of ak is equal
44
00:02:21,530 --> 00:02:25,100
to the probability that the
process first makes transition
45
00:02:25,100 --> 00:02:29,140
from S0 to S3 on the first
trial, which is probability
46
00:02:29,140 --> 00:02:34,510
03, times the probability
that the process makes
47
00:02:34,510 --> 00:02:38,850
self-transition for the next
k minus 2 trials, which is
48
00:02:38,850 --> 00:02:45,860
probability 33 to the power of k
minus 2, and finally makes a
49
00:02:45,860 --> 00:02:54,020
transition from S3 to S2 on the
kth trial, which is p32.
50
00:02:54,020 --> 00:03:03,340
And this gives us 1/3 times 1/4
to the power of k minus 2
51
00:03:03,340 --> 00:03:11,590
times 1/4, which is equal
to 1/3 times 1/4 to
52
00:03:11,590 --> 00:03:13,180
the power of k minus--
53
00:03:13,180 --> 00:03:15,980
For part B of the problem,
we have to calculate the
54
00:03:15,980 --> 00:03:20,550
probability that the process
never enters as four.
55
00:03:20,550 --> 00:03:23,180
This event can happen
in three ways.
56
00:03:23,180 --> 00:03:25,800
The first way is that the
process makes a transition
57
00:03:25,800 --> 00:03:28,530
from S0 to S1 on the
first trial and
58
00:03:28,530 --> 00:03:31,060
be stuck in S1 forever.
59
00:03:31,060 --> 00:03:34,070
The second way that the process
makes a transition
60
00:03:34,070 --> 00:03:36,700
from S0 to S5 on the
first trial and
61
00:03:36,700 --> 00:03:39,170
be stuck at S5 forever.
62
00:03:39,170 --> 00:03:42,120
The third way is that the
process makes a transition
63
00:03:42,120 --> 00:03:45,790
from S0 to S3 on the first
trial and then it makes a
64
00:03:45,790 --> 00:03:49,920
transition from S3 to S2 on the
next state change so that
65
00:03:49,920 --> 00:03:53,700
it would never be able
to go to S4.
66
00:03:53,700 --> 00:03:56,880
Therefore, the probability of
B is equal to the sum of
67
00:03:56,880 --> 00:03:59,260
probabilities of this
three events.
68
00:03:59,260 --> 00:04:02,360
69
00:04:02,360 --> 00:04:08,120
So the probability of B is equal
to the probability that
70
00:04:08,120 --> 00:04:10,960
the process makes a transition
from S0 to S1 on the first
71
00:04:10,960 --> 00:04:15,670
trial, which is 1/3, plus the
probability that the process
72
00:04:15,670 --> 00:04:19,130
makes a transition from S0 to
S5 on the first trial, which
73
00:04:19,130 --> 00:04:23,140
is also 1/3, plus the
probability that the process
74
00:04:23,140 --> 00:04:28,590
makes a transition from S0 to
S3 on the first trial times
75
00:04:28,590 --> 00:04:32,060
the probability that the
process then makes a
76
00:04:32,060 --> 00:04:37,090
transition from S3 to S2 on
the next state change.
77
00:04:37,090 --> 00:04:45,740
So transition to S2, given that
the processes are already
78
00:04:45,740 --> 00:04:49,110
in state S3 and there's
a state change.
79
00:04:49,110 --> 00:04:57,860
80
00:04:57,860 --> 00:05:00,950
Let's take a look at this
conditional probability.
81
00:05:00,950 --> 00:05:03,350
The condition that the processes
are already in state
82
00:05:03,350 --> 00:05:07,660
S3 and there's a state change
imply two possible events,
83
00:05:07,660 --> 00:05:12,880
which are the transition from
S3 to S2 and the transition
84
00:05:12,880 --> 00:05:16,380
from S3 to S4.
85
00:05:16,380 --> 00:05:23,850
Therefore, we can write this
conditional probability as the
86
00:05:23,850 --> 00:05:32,020
conditional probability of
transition from as S3 to S2,
87
00:05:32,020 --> 00:05:39,730
given that another event,
S3 to S2 or
88
00:05:39,730 --> 00:05:46,000
S3 to S4 has happened.
89
00:05:46,000 --> 00:05:56,270
And this is simply equal to the
proportion of p32 and p32
90
00:05:56,270 --> 00:06:07,300
plus p34, which is equal to 1/4
over 1/4 plus 1/2, which
91
00:06:07,300 --> 00:06:09,830
is equal to 1/3.
92
00:06:09,830 --> 00:06:17,870
Therefore, the probability of
B is equal to 1/3 plus 1/3
93
00:06:17,870 --> 00:06:26,345
plus 1/3 times the 1/3 here,
which is equal to 7/9.
94
00:06:26,345 --> 00:06:29,390
95
00:06:29,390 --> 00:06:32,350
For part C of the problem,
we have to calculate the
96
00:06:32,350 --> 00:06:36,170
probability that the process
enter S2 and leaves S2 on the
97
00:06:36,170 --> 00:06:37,820
next trial.
98
00:06:37,820 --> 00:06:40,560
This probability can be written
as the product of two
99
00:06:40,560 --> 00:06:42,240
probabilities--
100
00:06:42,240 --> 00:06:45,430
the probability that the process
enters S2 and the
101
00:06:45,430 --> 00:06:49,440
probability that it leaves S2 on
the next trial, given it's
102
00:06:49,440 --> 00:06:51,800
already in S2.
103
00:06:51,800 --> 00:06:54,630
Let's first look at the
probability that the
104
00:06:54,630 --> 00:06:57,200
process enters S2.
105
00:06:57,200 --> 00:07:00,550
Using a similar approach as
part B, we know that the
106
00:07:00,550 --> 00:07:04,890
probability the process ever
enters S2 is equal to the
107
00:07:04,890 --> 00:07:08,220
probability of the event that
the process first makes a
108
00:07:08,220 --> 00:07:12,650
transition from S0 to S3 on the
first trial and then makes
109
00:07:12,650 --> 00:07:18,350
a transition from S3 to S2
on the next state change.
110
00:07:18,350 --> 00:07:23,900
So the probability that the
process enters S2 is equal to
111
00:07:23,900 --> 00:07:27,430
the probability that it first
makes a transition from S0 to
112
00:07:27,430 --> 00:07:32,350
S3 on the first trial, which is
P03, times the probability
113
00:07:32,350 --> 00:07:42,175
that it makes a transition to
S2, given that it's already in
114
00:07:42,175 --> 00:07:45,850
S3 and there is a
state change.
115
00:07:45,850 --> 00:07:52,620
116
00:07:52,620 --> 00:07:56,090
We have already calculated this
conditional probability
117
00:07:56,090 --> 00:08:01,610
in part B. Let's then look at
the second probability term,
118
00:08:01,610 --> 00:08:04,160
the probability that the process
leaves S2 on the next
119
00:08:04,160 --> 00:08:07,690
trial, given that it's
already in S2.
120
00:08:07,690 --> 00:08:12,630
So given that the process is
already in S2, it can take two
121
00:08:12,630 --> 00:08:14,020
transitions.
122
00:08:14,020 --> 00:08:18,170
In can either transition
from S2 to S1 or make a
123
00:08:18,170 --> 00:08:22,090
self-transition from
S2 back to S2.
124
00:08:22,090 --> 00:08:26,660
Therefore, this conditional
probability that it leaves S2
125
00:08:26,660 --> 00:08:29,700
on the next trial, given that it
was already in S2 is simply
126
00:08:29,700 --> 00:08:33,230
equal to the transition
probability from S2 to S1,
127
00:08:33,230 --> 00:08:35,575
which is P21.
128
00:08:35,575 --> 00:08:38,270
129
00:08:38,270 --> 00:08:46,510
Therefore, this is equal to P03,
which is 1/3, times 1/3
130
00:08:46,510 --> 00:08:54,040
from the result from part B
times P21, which is 1/2, and
131
00:08:54,040 --> 00:08:55,460
gives us 1/18.
132
00:08:55,460 --> 00:08:58,360
133
00:08:58,360 --> 00:09:01,200
For part D of the problem,
we have to calculate the
134
00:09:01,200 --> 00:09:04,510
probability that the process
enters S1 for the first time
135
00:09:04,510 --> 00:09:06,840
on the third trial.
136
00:09:06,840 --> 00:09:09,240
So if you take a look at this
Markov chain, you'll notice
137
00:09:09,240 --> 00:09:12,510
that the only way for this
event to happen is when a
138
00:09:12,510 --> 00:09:16,420
process first makes a transition
from S0 to S3 on
139
00:09:16,420 --> 00:09:20,450
the first trial and from S3 to
S2 on the second trial and
140
00:09:20,450 --> 00:09:23,260
from S2 to S1 on the
third trial.
141
00:09:23,260 --> 00:09:29,900
Therefore, the probability of D
is equal to the probability
142
00:09:29,900 --> 00:09:35,280
of the event that the process
makes a transition from S0 to
143
00:09:35,280 --> 00:09:41,040
S3 on the first trial and from
S3 to S2 on the second trial
144
00:09:41,040 --> 00:09:45,210
and finally from S2 to S1
on the third trial.
145
00:09:45,210 --> 00:09:56,140
So this is equal to P03 times
P32 times P21, which is equal
146
00:09:56,140 --> 00:10:07,570
to 1/3 times 1/4 times 1/2,
which is equal to 1/24.
147
00:10:07,570 --> 00:10:11,300
148
00:10:11,300 --> 00:10:14,140
For part E of the problem,
we have to calculate the
149
00:10:14,140 --> 00:10:17,540
probability that the process
is in S3 immediately
150
00:10:17,540 --> 00:10:19,390
after the nth trial.
151
00:10:19,390 --> 00:10:22,040
If you take a look at this
Markov chain, you'll notice
152
00:10:22,040 --> 00:10:25,500
that if on the first trial, the
process makes a transition
153
00:10:25,500 --> 00:10:29,510
from S0 to S1 or from S0
to S5, it will never be
154
00:10:29,510 --> 00:10:31,620
able to go to S3.
155
00:10:31,620 --> 00:10:34,600
On the other hand, if on the
first trial, the process makes
156
00:10:34,600 --> 00:10:39,950
a transition from S0 to S3 and
if it leaves S3 at some point,
157
00:10:39,950 --> 00:10:43,290
it will never be able
to come back to S3.
158
00:10:43,290 --> 00:10:47,130
Therefore, in order for the
process to be S3 immediately
159
00:10:47,130 --> 00:10:50,410
after the nth trial, we will
need the process to first make
160
00:10:50,410 --> 00:10:55,585
transition from S0 to S3 on the
first trial and then stay
161
00:10:55,585 --> 00:10:58,810
in S3 for the next
n minus 1 trials.
162
00:10:58,810 --> 00:11:06,880
163
00:11:06,880 --> 00:11:12,330
Therefore, the probability of
the event e is simply equal to
164
00:11:12,330 --> 00:11:19,620
the probability of this event,
which is equal to P03 times
165
00:11:19,620 --> 00:11:31,780
P33 to the power n minus 1,
which is equal to 1/3 times
166
00:11:31,780 --> 00:11:38,420
1/4 to the power of n minus 1.
167
00:11:38,420 --> 00:11:40,410
And this concludes
our practice on
168
00:11:40,410 --> 00:11:41,660
Markov chain today.
169
00:11:41,660 --> 00:11:42,600