WEBVTT

00:00:00.590 --> 00:00:01.420
Hi, everyone.

00:00:01.420 --> 00:00:03.270
Today, I'm going to talk
about Markov Chain

00:00:03.270 --> 00:00:05.050
Practice number one.

00:00:05.050 --> 00:00:07.240
Before we start, let's
first take a look

00:00:07.240 --> 00:00:09.080
at this Markov chain.

00:00:09.080 --> 00:00:11.380
This Markov chain
has six states.

00:00:11.380 --> 00:00:13.720
In this problem, we always
assume the process

00:00:13.720 --> 00:00:15.980
starts from state S0.

00:00:15.980 --> 00:00:18.470
On the first trial, the process
can either make a

00:00:18.470 --> 00:00:23.080
transition from S0 to S1 with
probability 1/3 or from S0 to

00:00:23.080 --> 00:00:26.740
S3 with probability 1/3 third
or from S0 to S5 with

00:00:26.740 --> 00:00:28.460
probability 1/3.

00:00:28.460 --> 00:00:31.310
If on the first trial, the
process makes the transition

00:00:31.310 --> 00:00:35.870
from S0 to S1 or from S0 to S5,
it will always be stuck in

00:00:35.870 --> 00:00:40.110
either S1 or S5 forever, because
both of the states S1

00:00:40.110 --> 00:00:44.220
and S5 have a self-transition
probability of one.

00:00:44.220 --> 00:00:47.160
On the other hand, if on the
first trial, the process makes

00:00:47.160 --> 00:00:50.950
the transition from S0 to S3, it
can then either transition

00:00:50.950 --> 00:00:53.920
to the left or transition
to the right or make

00:00:53.920 --> 00:00:57.430
self-transition back
to the state S3.

00:00:57.430 --> 00:01:00.570
If the process ever enters the
left of the chain, it will

00:01:00.570 --> 00:01:03.120
never be able to come
to the right.

00:01:03.120 --> 00:01:06.030
On the other hand, if the
process ever enters the right

00:01:06.030 --> 00:01:10.280
of the chain, it would never
be able to go to the left.

00:01:10.280 --> 00:01:13.200
For part A of the problem,
we have to calculate the

00:01:13.200 --> 00:01:17.140
probability that the process
enter S2 for the first time at

00:01:17.140 --> 00:01:18.790
the case trial.

00:01:18.790 --> 00:01:22.890
First, notice that it would take
at least two trials for

00:01:22.890 --> 00:01:26.980
the process to make a transition
from S0 to S2.

00:01:26.980 --> 00:01:30.830
Therefore, for k equal to 1,
the probability of ak is

00:01:30.830 --> 00:01:32.080
simply equal to 0.

00:01:34.740 --> 00:01:41.860
For k equal to 1, probability
of a1 is equal to 0.

00:01:44.710 --> 00:01:49.250
Then for k equal to 2, 3 and on,
the probability that the

00:01:49.250 --> 00:01:52.870
process enters S2 for the first
time at a case trial is

00:01:52.870 --> 00:01:56.600
equivalent to the probability
that the process first makes a

00:01:56.600 --> 00:02:01.430
transition from S0 to S3 and
then stays in S3 for the next

00:02:01.430 --> 00:02:05.060
two k minus 2 trials and finally
makes a transition

00:02:05.060 --> 00:02:09.520
from S3 to S2 on
the kth trial.

00:02:09.520 --> 00:02:12.730
So let's write this out.

00:02:12.730 --> 00:02:21.530
For k equal to 2, 3, and on, the
probability of ak is equal

00:02:21.530 --> 00:02:25.100
to the probability that the
process first makes transition

00:02:25.100 --> 00:02:29.140
from S0 to S3 on the first
trial, which is probability

00:02:29.140 --> 00:02:34.510
03, times the probability
that the process makes

00:02:34.510 --> 00:02:38.850
self-transition for the next
k minus 2 trials, which is

00:02:38.850 --> 00:02:45.860
probability 33 to the power of k
minus 2, and finally makes a

00:02:45.860 --> 00:02:54.020
transition from S3 to S2 on the
kth trial, which is p32.

00:02:54.020 --> 00:03:03.340
And this gives us 1/3 times 1/4
to the power of k minus 2

00:03:03.340 --> 00:03:11.590
times 1/4, which is equal
to 1/3 times 1/4 to

00:03:11.590 --> 00:03:13.180
the power of k minus--

00:03:13.180 --> 00:03:15.980
For part B of the problem,
we have to calculate the

00:03:15.980 --> 00:03:20.550
probability that the process
never enters as four.

00:03:20.550 --> 00:03:23.180
This event can happen
in three ways.

00:03:23.180 --> 00:03:25.800
The first way is that the
process makes a transition

00:03:25.800 --> 00:03:28.530
from S0 to S1 on the
first trial and

00:03:28.530 --> 00:03:31.060
be stuck in S1 forever.

00:03:31.060 --> 00:03:34.070
The second way that the process
makes a transition

00:03:34.070 --> 00:03:36.700
from S0 to S5 on the
first trial and

00:03:36.700 --> 00:03:39.170
be stuck at S5 forever.

00:03:39.170 --> 00:03:42.120
The third way is that the
process makes a transition

00:03:42.120 --> 00:03:45.790
from S0 to S3 on the first
trial and then it makes a

00:03:45.790 --> 00:03:49.920
transition from S3 to S2 on the
next state change so that

00:03:49.920 --> 00:03:53.700
it would never be able
to go to S4.

00:03:53.700 --> 00:03:56.880
Therefore, the probability of
B is equal to the sum of

00:03:56.880 --> 00:03:59.260
probabilities of this
three events.

00:04:02.360 --> 00:04:08.120
So the probability of B is equal
to the probability that

00:04:08.120 --> 00:04:10.960
the process makes a transition
from S0 to S1 on the first

00:04:10.960 --> 00:04:15.670
trial, which is 1/3, plus the
probability that the process

00:04:15.670 --> 00:04:19.130
makes a transition from S0 to
S5 on the first trial, which

00:04:19.130 --> 00:04:23.140
is also 1/3, plus the
probability that the process

00:04:23.140 --> 00:04:28.590
makes a transition from S0 to
S3 on the first trial times

00:04:28.590 --> 00:04:32.060
the probability that the
process then makes a

00:04:32.060 --> 00:04:37.090
transition from S3 to S2 on
the next state change.

00:04:37.090 --> 00:04:45.740
So transition to S2, given that
the processes are already

00:04:45.740 --> 00:04:49.110
in state S3 and there's
a state change.

00:04:57.860 --> 00:05:00.950
Let's take a look at this
conditional probability.

00:05:00.950 --> 00:05:03.350
The condition that the processes
are already in state

00:05:03.350 --> 00:05:07.660
S3 and there's a state change
imply two possible events,

00:05:07.660 --> 00:05:12.880
which are the transition from
S3 to S2 and the transition

00:05:12.880 --> 00:05:16.380
from S3 to S4.

00:05:16.380 --> 00:05:23.850
Therefore, we can write this
conditional probability as the

00:05:23.850 --> 00:05:32.020
conditional probability of
transition from as S3 to S2,

00:05:32.020 --> 00:05:39.730
given that another event,
S3 to S2 or

00:05:39.730 --> 00:05:46.000
S3 to S4 has happened.

00:05:46.000 --> 00:05:56.270
And this is simply equal to the
proportion of p32 and p32

00:05:56.270 --> 00:06:07.300
plus p34, which is equal to 1/4
over 1/4 plus 1/2, which

00:06:07.300 --> 00:06:09.830
is equal to 1/3.

00:06:09.830 --> 00:06:17.870
Therefore, the probability of
B is equal to 1/3 plus 1/3

00:06:17.870 --> 00:06:26.345
plus 1/3 times the 1/3 here,
which is equal to 7/9.

00:06:29.390 --> 00:06:32.350
For part C of the problem,
we have to calculate the

00:06:32.350 --> 00:06:36.170
probability that the process
enter S2 and leaves S2 on the

00:06:36.170 --> 00:06:37.820
next trial.

00:06:37.820 --> 00:06:40.560
This probability can be written
as the product of two

00:06:40.560 --> 00:06:42.240
probabilities--

00:06:42.240 --> 00:06:45.430
the probability that the process
enters S2 and the

00:06:45.430 --> 00:06:49.440
probability that it leaves S2 on
the next trial, given it's

00:06:49.440 --> 00:06:51.800
already in S2.

00:06:51.800 --> 00:06:54.630
Let's first look at the
probability that the

00:06:54.630 --> 00:06:57.200
process enters S2.

00:06:57.200 --> 00:07:00.550
Using a similar approach as
part B, we know that the

00:07:00.550 --> 00:07:04.890
probability the process ever
enters S2 is equal to the

00:07:04.890 --> 00:07:08.220
probability of the event that
the process first makes a

00:07:08.220 --> 00:07:12.650
transition from S0 to S3 on the
first trial and then makes

00:07:12.650 --> 00:07:18.350
a transition from S3 to S2
on the next state change.

00:07:18.350 --> 00:07:23.900
So the probability that the
process enters S2 is equal to

00:07:23.900 --> 00:07:27.430
the probability that it first
makes a transition from S0 to

00:07:27.430 --> 00:07:32.350
S3 on the first trial, which is
P03, times the probability

00:07:32.350 --> 00:07:42.175
that it makes a transition to
S2, given that it's already in

00:07:42.175 --> 00:07:45.850
S3 and there is a
state change.

00:07:52.620 --> 00:07:56.090
We have already calculated this
conditional probability

00:07:56.090 --> 00:08:01.610
in part B. Let's then look at
the second probability term,

00:08:01.610 --> 00:08:04.160
the probability that the process
leaves S2 on the next

00:08:04.160 --> 00:08:07.690
trial, given that it's
already in S2.

00:08:07.690 --> 00:08:12.630
So given that the process is
already in S2, it can take two

00:08:12.630 --> 00:08:14.020
transitions.

00:08:14.020 --> 00:08:18.170
In can either transition
from S2 to S1 or make a

00:08:18.170 --> 00:08:22.090
self-transition from
S2 back to S2.

00:08:22.090 --> 00:08:26.660
Therefore, this conditional
probability that it leaves S2

00:08:26.660 --> 00:08:29.700
on the next trial, given that it
was already in S2 is simply

00:08:29.700 --> 00:08:33.230
equal to the transition
probability from S2 to S1,

00:08:33.230 --> 00:08:35.575
which is P21.

00:08:38.270 --> 00:08:46.510
Therefore, this is equal to P03,
which is 1/3, times 1/3

00:08:46.510 --> 00:08:54.040
from the result from part B
times P21, which is 1/2, and

00:08:54.040 --> 00:08:55.460
gives us 1/18.

00:08:58.360 --> 00:09:01.200
For part D of the problem,
we have to calculate the

00:09:01.200 --> 00:09:04.510
probability that the process
enters S1 for the first time

00:09:04.510 --> 00:09:06.840
on the third trial.

00:09:06.840 --> 00:09:09.240
So if you take a look at this
Markov chain, you'll notice

00:09:09.240 --> 00:09:12.510
that the only way for this
event to happen is when a

00:09:12.510 --> 00:09:16.420
process first makes a transition
from S0 to S3 on

00:09:16.420 --> 00:09:20.450
the first trial and from S3 to
S2 on the second trial and

00:09:20.450 --> 00:09:23.260
from S2 to S1 on the
third trial.

00:09:23.260 --> 00:09:29.900
Therefore, the probability of D
is equal to the probability

00:09:29.900 --> 00:09:35.280
of the event that the process
makes a transition from S0 to

00:09:35.280 --> 00:09:41.040
S3 on the first trial and from
S3 to S2 on the second trial

00:09:41.040 --> 00:09:45.210
and finally from S2 to S1
on the third trial.

00:09:45.210 --> 00:09:56.140
So this is equal to P03 times
P32 times P21, which is equal

00:09:56.140 --> 00:10:07.570
to 1/3 times 1/4 times 1/2,
which is equal to 1/24.

00:10:11.300 --> 00:10:14.140
For part E of the problem,
we have to calculate the

00:10:14.140 --> 00:10:17.540
probability that the process
is in S3 immediately

00:10:17.540 --> 00:10:19.390
after the nth trial.

00:10:19.390 --> 00:10:22.040
If you take a look at this
Markov chain, you'll notice

00:10:22.040 --> 00:10:25.500
that if on the first trial, the
process makes a transition

00:10:25.500 --> 00:10:29.510
from S0 to S1 or from S0
to S5, it will never be

00:10:29.510 --> 00:10:31.620
able to go to S3.

00:10:31.620 --> 00:10:34.600
On the other hand, if on the
first trial, the process makes

00:10:34.600 --> 00:10:39.950
a transition from S0 to S3 and
if it leaves S3 at some point,

00:10:39.950 --> 00:10:43.290
it will never be able
to come back to S3.

00:10:43.290 --> 00:10:47.130
Therefore, in order for the
process to be S3 immediately

00:10:47.130 --> 00:10:50.410
after the nth trial, we will
need the process to first make

00:10:50.410 --> 00:10:55.585
transition from S0 to S3 on the
first trial and then stay

00:10:55.585 --> 00:10:58.810
in S3 for the next
n minus 1 trials.

00:11:06.880 --> 00:11:12.330
Therefore, the probability of
the event e is simply equal to

00:11:12.330 --> 00:11:19.620
the probability of this event,
which is equal to P03 times

00:11:19.620 --> 00:11:31.780
P33 to the power n minus 1,
which is equal to 1/3 times

00:11:31.780 --> 00:11:38.420
1/4 to the power of n minus 1.

00:11:38.420 --> 00:11:40.410
And this concludes
our practice on

00:11:40.410 --> 00:11:41.660
Markov chain today.