Hi, everyone.
Today, I'm going to talk about Markov Chain
Practice number one.
Before we start, let's first take a look
at this Markov chain.
This Markov chain has six states.
In this problem, we always assume the process
starts from state S0.
On the first trial, the process can either make a
transition from S0 to S1 with probability 1/3 or from S0 to
S3 with probability 1/3 third or from S0 to S5 with
probability 1/3.
If on the first trial, the process makes the transition
from S0 to S1 or from S0 to S5, it will always be stuck in
either S1 or S5 forever, because both of the states S1
and S5 have a self-transition probability of one.
On the other hand, if on the first trial, the process makes
the transition from S0 to S3, it can then either transition
to the left or transition to the right or make
self-transition back to the state S3.
If the process ever enters the left of the chain, it will
never be able to come to the right.
On the other hand, if the process ever enters the right
of the chain, it would never be able to go to the left.
For part A of the problem, we have to calculate the
probability that the process enter S2 for the first time at
the case trial.
First, notice that it would take at least two trials for
the process to make a transition from S0 to S2.
Therefore, for k equal to 1, the probability of ak is
simply equal to 0.
For k equal to 1, probability of a1 is equal to 0.
Then for k equal to 2, 3 and on, the probability that the
process enters S2 for the first time at a case trial is
equivalent to the probability that the process first makes a
transition from S0 to S3 and then stays in S3 for the next
two k minus 2 trials and finally makes a transition
from S3 to S2 on the kth trial.
So let's write this out.
For k equal to 2, 3, and on, the probability of ak is equal
to the probability that the process first makes transition
from S0 to S3 on the first trial, which is probability
03, times the probability that the process makes
self-transition for the next k minus 2 trials, which is
probability 33 to the power of k minus 2, and finally makes a
transition from S3 to S2 on the kth trial, which is p32.
And this gives us 1/3 times 1/4 to the power of k minus 2
times 1/4, which is equal to 1/3 times 1/4 to
the power of k minus--
For part B of the problem, we have to calculate the
probability that the process never enters as four.
This event can happen in three ways.
The first way is that the process makes a transition
from S0 to S1 on the first trial and
be stuck in S1 forever.
The second way that the process makes a transition
from S0 to S5 on the first trial and
be stuck at S5 forever.
The third way is that the process makes a transition
from S0 to S3 on the first trial and then it makes a
transition from S3 to S2 on the next state change so that
it would never be able to go to S4.
Therefore, the probability of B is equal to the sum of
probabilities of this three events.
So the probability of B is equal to the probability that
the process makes a transition from S0 to S1 on the first
trial, which is 1/3, plus the probability that the process
makes a transition from S0 to S5 on the first trial, which
is also 1/3, plus the probability that the process
makes a transition from S0 to S3 on the first trial times
the probability that the process then makes a
transition from S3 to S2 on the next state change.
So transition to S2, given that the processes are already
in state S3 and there's a state change.
Let's take a look at this conditional probability.
The condition that the processes are already in state
S3 and there's a state change imply two possible events,
which are the transition from S3 to S2 and the transition
from S3 to S4.
Therefore, we can write this conditional probability as the
conditional probability of transition from as S3 to S2,
given that another event, S3 to S2 or
S3 to S4 has happened.
And this is simply equal to the proportion of p32 and p32
plus p34, which is equal to 1/4 over 1/4 plus 1/2, which
is equal to 1/3.
Therefore, the probability of B is equal to 1/3 plus 1/3
plus 1/3 times the 1/3 here, which is equal to 7/9.
For part C of the problem, we have to calculate the
probability that the process enter S2 and leaves S2 on the
next trial.
This probability can be written as the product of two
probabilities--
the probability that the process enters S2 and the
probability that it leaves S2 on the next trial, given it's
already in S2.
Let's first look at the probability that the
process enters S2.
Using a similar approach as part B, we know that the
probability the process ever enters S2 is equal to the
probability of the event that the process first makes a
transition from S0 to S3 on the first trial and then makes
a transition from S3 to S2 on the next state change.
So the probability that the process enters S2 is equal to
the probability that it first makes a transition from S0 to
S3 on the first trial, which is P03, times the probability
that it makes a transition to S2, given that it's already in
S3 and there is a state change.
We have already calculated this conditional probability
in part B. Let's then look at the second probability term,
the probability that the process leaves S2 on the next
trial, given that it's already in S2.
So given that the process is already in S2, it can take two
transitions.
In can either transition from S2 to S1 or make a
self-transition from S2 back to S2.
Therefore, this conditional probability that it leaves S2
on the next trial, given that it was already in S2 is simply
equal to the transition probability from S2 to S1,
which is P21.
Therefore, this is equal to P03, which is 1/3, times 1/3
from the result from part B times P21, which is 1/2, and
gives us 1/18.
For part D of the problem, we have to calculate the
probability that the process enters S1 for the first time
on the third trial.
So if you take a look at this Markov chain, you'll notice
that the only way for this event to happen is when a
process first makes a transition from S0 to S3 on
the first trial and from S3 to S2 on the second trial and
from S2 to S1 on the third trial.
Therefore, the probability of D is equal to the probability
of the event that the process makes a transition from S0 to
S3 on the first trial and from S3 to S2 on the second trial
and finally from S2 to S1 on the third trial.
So this is equal to P03 times P32 times P21, which is equal
to 1/3 times 1/4 times 1/2, which is equal to 1/24.
For part E of the problem, we have to calculate the
probability that the process is in S3 immediately
after the nth trial.
If you take a look at this Markov chain, you'll notice
that if on the first trial, the process makes a transition
from S0 to S1 or from S0 to S5, it will never be
able to go to S3.
On the other hand, if on the first trial, the process makes
a transition from S0 to S3 and if it leaves S3 at some point,
it will never be able to come back to S3.
Therefore, in order for the process to be S3 immediately
after the nth trial, we will need the process to first make
transition from S0 to S3 on the first trial and then stay
in S3 for the next n minus 1 trials.
Therefore, the probability of the event e is simply equal to
the probability of this event, which is equal to P03 times
P33 to the power n minus 1, which is equal to 1/3 times
1/4 to the power of n minus 1.
And this concludes our practice on
Markov chain today.