Hi.
In this problem, we're going to practice setting up a
Markov chain by going fishing in this lake, which has n fish
in it, some of which are green.
And the rest of the fish are blue.
So, what we do is, every day we go to this lake, and we
catch exactly 1 fish.
And all the fish are equally likely to be
the 1 that's caught.
Now, if we catch a green fish, we paint it blue, and we throw
back into the lake.
And if we catch a blue fish, we just keep it blue, and we
also throw it back.
Now, what we're interested in modeling is, how does this
lake evolve over time?
And specifically what we're interested in is the number of
green fish that are left in the lake.
So, let's let Gi be the event that there are i green fish
left in the lake.
And we want to know, how does Gi evolve over time?
Now, one thing that we've learned that we can use to
model this is a Markov chain.
But before we can use it, we need to make sure that this
actually satisfies the Markov property.
Now, recall that the Markov property essentially says
that, given the current state of the system, that's all you
need in order to predict the future states.
So, any past history of the previous states that it was
in, that's all irrelevant.
All you need is the current state.
Now, in the context of this particular problem, what that
means is that if I tell you that there are 10 green fish
left, that's all the information you need in order
to predict how many fish there will be tomorrow.
So, why is that?
Well, it's because what influences the number of green
fish that are left?
What influences it is which fish you catch because,
depending on which fish you catch, you may paint the green
fish blue, in which case the number of green fish decrease.
But what affects which fish you catch?
Well, that probability is dictated solely based on just
the number of green fish in the lake right now, today.
So, it doesn't matter that there were 20 fish yesterday.
All that matters is how many green fish there are in the
lake today.
And so, because of that argument, the
number of green fish--
this does satisfy the Markov property, so we can use this
and model it as a Markov chain.
So, like we alluded to just now, the key dynamic that we
need to look at is, how does the number
of green fish change?
And if we look at it, we notice that after each day,
the number of green fish can only have two possible
transitions.
One possible transition is that it goes down by exactly
1, which happens if you happen to catch a green fish and
paint it blue.
So, that green fish is no longer green, so the number of
green fish goes down by 1.
The other possible transition is that Gi doesn't change
because you caught a blue fish that day.
So, all the green fish are still green.
So, now given that, let's see if we can come up with a
Markov chain.
So, the first thing we've done is we've written down all the
different states, right?
So, this represents the number of green
fish left in the lake.
So, there could be 0 green fish left, 1 green fish, all
the way through n, which means that every single fish in the
lake is green.
Now, we have the states.
What we need to do now is to fill in the transition
probabilities, which are the Pij's.
And remember, the Pij is the probability of transitioning
from state i to state j in the next transition.
So, what that means in this context is, what's the
probability that there will be j green fish tomorrow given
that there are i green fish today?
Now, if we go back to our earlier argument, we see that
for any given i, you can only transition to
two possible j's.
One of them is you stay at i because the number of green
fish doesn't change because you caught a blue fish.
And the other is that you'd go from i to i minus 1.
The number of green fish decreases by 1.
Now, what we need to do now is fill in what those
probabilities are.
So, if j equals i, meaning that the number of green fish
doesn't change, well, what's the probability that you have
the same number of green fish tomorrow as you do today?
Well, if you have i green fish today, that happens if you
catch 1 of the n minus i blue fish.
So, what's the probability of catching one of the n
minus i blue fish?
Well, it's n minus i over n.
Now, the other possible transition is you go from a i
to j equals i minus 1, so i goes down by 1.
And that happens when you catch a green fish.
So, given that there are i green fish, what's the
probability that you catch 1 of those?
Well, it's going to be i/n.
And finally, every other transition has 0 probability.
All right.
So, now we can add those transitions on
to our Markov chain.
So, for example, we have these.
So, let's look at this general case i.
So, if you're state i, you have i green fish left.
You will transition to i minus 1 green fish left if that day
you caught a green fish.
And we said that that probability is i/n.
And the self transition probability is you caught a
blue fish that day, so you still stay a i green fish.
And that probability, we said, was n minus i over n.
All right.
Now, it's helpful to verify that this formula works by
looking at some cases where it's intuitive to calculate
what these probabilities should be.
So, let's look at state n.
That is the state where every single fish
in the lake is green.
So, if ever single fish in the lake is green, then no matter
what fish you catch, it's going to be green.
And you're going to paint it blue and return it, so you're
guaranteed to go down to n minus 1 green fish.
And so, this transition probability down to n minus 1
is guaranteed to be 1.
And so, the self transition probability has to be 0.
Now, let's go back to our formula and verify that
actually gives us the right value.
So, if i is n, then there's only these transition
probabilities.
So, if i is n, then the transition probability to j,
for j is also n, is n minus n over n, which is 0.
And that's exactly what we said.
We argued that the self transition
probability should be 0.
And also, if i is in, the probability of transitioning
to n minus 1 should be n over n, which is 1.
And that's exactly what we argued here.
So, it seems like these transition probabilities do
make sense.
And if we wanted to, we could fill in the rest of these.
So, for example, this would be 2/n, 1/n, n minus 1 over n, n
minus 2 over n.
And now, let's also consider the case of state 0, which
means that every single fish is blue.
There are 0 green fish left.
Well, if that's the case, then what's the probability of
staying at 0?
Well, that's n minus 0 over n is 1, all right?
So, the self transition probability is 1.
And that makes sense because if you have 0 green fish,
there's no way to generate more green fish because you
don't paint blue fish green.
And so, you're going to stay at 0 green fish forever.
All right.
So, we've characterized the entire Markov chain now.
And so, now let's just answer some simple
questions about this.
So, the problem asks us to identify, what are the
recurrent and transient states?
So, remember that recurrent state means that if you start
out at that state, no matter where you go, what other
states you end up at, there is some positive probability path
that will take you back to your original state.
And if you're not recurrent, then you're transient, which
means that if you're transient, if you start out at
the transient state, there is some other state that you can
go to, from which there's no way to come back to the
original transient state.
All right.
So, now let's look at this and see which states are recurrent
and which are transient.
And we can fill this in more.
And if we look at it, let's look at state n.
Well, we're guaranteed to go from state n to
state n minus 1.
And once we're in state n minus 1, there's no way for us
to go back to state n because we can't
generate more green fish.
And so, n is transient.
And similarly, we can use the same argument to show that
everything from 1 through n, all of these states, are
transient for the same reason because there's no way to
generate more green fish.
And so, the chain can only stay at a given
state or go down 1.
And so, it always goes down.
It can only go left, and it can never go right.
So, once you leave a certain state, there's
no way to come back.
And so, states 1 through n are all transient.
And 0 the only recurrent state because, well, the only place
you go from 0 is itself.
So, you always stay at 0.
And in fact, 0 is not only recurrent, it's absorbing
because every single other state, no matter where you
start out at, you will always end up at 0.
So, this was just an example of how to
set up a Markov chain.
You just think about the actual dynamics of what's
going on and make sure that it satisfies the Markov property.
Then, figure out what all the states are and calculate all
the transition probabilities.
And once you have that, you've specified your Markov chain.