WEBVTT
00:00:00.590 --> 00:00:01.080
Hi.
00:00:01.080 --> 00:00:02.930
In this problem, we're going
to practice setting up a
00:00:02.930 --> 00:00:06.720
Markov chain by going fishing in
this lake, which has n fish
00:00:06.720 --> 00:00:08.800
in it, some of which
are green.
00:00:08.800 --> 00:00:10.800
And the rest of the
fish are blue.
00:00:10.800 --> 00:00:14.260
So, what we do is, every day
we go to this lake, and we
00:00:14.260 --> 00:00:16.560
catch exactly 1 fish.
00:00:16.560 --> 00:00:18.900
And all the fish are equally
likely to be
00:00:18.900 --> 00:00:20.500
the 1 that's caught.
00:00:20.500 --> 00:00:24.560
Now, if we catch a green fish,
we paint it blue, and we throw
00:00:24.560 --> 00:00:26.460
back into the lake.
00:00:26.460 --> 00:00:30.110
And if we catch a blue fish, we
just keep it blue, and we
00:00:30.110 --> 00:00:31.790
also throw it back.
00:00:31.790 --> 00:00:36.200
Now, what we're interested in
modeling is, how does this
00:00:36.200 --> 00:00:37.470
lake evolve over time?
00:00:37.470 --> 00:00:40.315
And specifically what we're
interested in is the number of
00:00:40.315 --> 00:00:41.720
green fish that are
left in the lake.
00:00:41.720 --> 00:00:45.320
So, let's let Gi be the event
that there are i green fish
00:00:45.320 --> 00:00:46.390
left in the lake.
00:00:46.390 --> 00:00:49.710
And we want to know, how does
Gi evolve over time?
00:00:49.710 --> 00:00:51.530
Now, one thing that we've
learned that we can use to
00:00:51.530 --> 00:00:53.230
model this is a Markov chain.
00:00:53.230 --> 00:00:55.550
But before we can use it, we
need to make sure that this
00:00:55.550 --> 00:00:57.640
actually satisfies the
Markov property.
00:00:57.640 --> 00:00:59.880
Now, recall that the Markov
property essentially says
00:00:59.880 --> 00:01:05.840
that, given the current state of
the system, that's all you
00:01:05.840 --> 00:01:07.970
need in order to predict
the future states.
00:01:07.970 --> 00:01:11.870
So, any past history of the
previous states that it was
00:01:11.870 --> 00:01:14.050
in, that's all irrelevant.
00:01:14.050 --> 00:01:15.740
All you need is the
current state.
00:01:15.740 --> 00:01:17.950
Now, in the context of this
particular problem, what that
00:01:17.950 --> 00:01:21.080
means is that if I tell you that
there are 10 green fish
00:01:21.080 --> 00:01:24.670
left, that's all the information
you need in order
00:01:24.670 --> 00:01:27.750
to predict how many fish
there will be tomorrow.
00:01:27.750 --> 00:01:29.030
So, why is that?
00:01:29.030 --> 00:01:33.750
Well, it's because what
influences the number of green
00:01:33.750 --> 00:01:35.210
fish that are left?
00:01:35.210 --> 00:01:39.030
What influences it is which
fish you catch because,
00:01:39.030 --> 00:01:43.100
depending on which fish you
catch, you may paint the green
00:01:43.100 --> 00:01:47.830
fish blue, in which case the
number of green fish decrease.
00:01:47.830 --> 00:01:52.750
But what affects which
fish you catch?
00:01:52.750 --> 00:01:55.820
Well, that probability is
dictated solely based on just
00:01:55.820 --> 00:01:59.070
the number of green fish in
the lake right now, today.
00:01:59.070 --> 00:02:03.220
So, it doesn't matter that there
were 20 fish yesterday.
00:02:03.220 --> 00:02:07.560
All that matters is how many
green fish there are in the
00:02:07.560 --> 00:02:08.840
lake today.
00:02:08.840 --> 00:02:11.020
And so, because of that
argument, the
00:02:11.020 --> 00:02:13.240
number of green fish--
00:02:13.240 --> 00:02:16.225
this does satisfy the Markov
property, so we can use this
00:02:16.225 --> 00:02:18.480
and model it as a
Markov chain.
00:02:18.480 --> 00:02:23.130
So, like we alluded to just now,
the key dynamic that we
00:02:23.130 --> 00:02:25.916
need to look at is, how
does the number
00:02:25.916 --> 00:02:27.050
of green fish change?
00:02:27.050 --> 00:02:30.360
And if we look at it, we notice
that after each day,
00:02:30.360 --> 00:02:36.840
the number of green fish can
only have two possible
00:02:36.840 --> 00:02:38.330
transitions.
00:02:38.330 --> 00:02:40.730
One possible transition is that
it goes down by exactly
00:02:40.730 --> 00:02:45.230
1, which happens if you happen
to catch a green fish and
00:02:45.230 --> 00:02:46.490
paint it blue.
00:02:46.490 --> 00:02:50.440
So, that green fish is no longer
green, so the number of
00:02:50.440 --> 00:02:52.080
green fish goes down by 1.
00:02:52.080 --> 00:02:56.670
The other possible transition
is that Gi doesn't change
00:02:56.670 --> 00:02:58.960
because you caught a
blue fish that day.
00:02:58.960 --> 00:03:01.820
So, all the green fish
are still green.
00:03:01.820 --> 00:03:05.095
So, now given that, let's see
if we can come up with a
00:03:05.095 --> 00:03:06.420
Markov chain.
00:03:06.420 --> 00:03:09.410
So, the first thing we've done
is we've written down all the
00:03:09.410 --> 00:03:11.580
different states, right?
00:03:11.580 --> 00:03:13.090
So, this represents the
number of green
00:03:13.090 --> 00:03:14.460
fish left in the lake.
00:03:14.460 --> 00:03:18.220
So, there could be 0 green fish
left, 1 green fish, all
00:03:18.220 --> 00:03:20.975
the way through n, which means
that every single fish in the
00:03:20.975 --> 00:03:22.910
lake is green.
00:03:22.910 --> 00:03:24.010
Now, we have the states.
00:03:24.010 --> 00:03:26.690
What we need to do now is to
fill in the transition
00:03:26.690 --> 00:03:30.660
probabilities, which
are the Pij's.
00:03:30.660 --> 00:03:36.020
And remember, the Pij is the
probability of transitioning
00:03:36.020 --> 00:03:39.870
from state i to state j in
the next transition.
00:03:39.870 --> 00:03:42.240
So, what that means in this
context is, what's the
00:03:42.240 --> 00:03:51.910
probability that there will be
j green fish tomorrow given
00:03:51.910 --> 00:03:57.880
that there are i green
fish today?
00:04:01.230 --> 00:04:06.690
Now, if we go back to our
earlier argument, we see that
00:04:06.690 --> 00:04:09.420
for any given i, you can
only transition to
00:04:09.420 --> 00:04:10.870
two possible j's.
00:04:10.870 --> 00:04:15.310
One of them is you stay at i
because the number of green
00:04:15.310 --> 00:04:17.740
fish doesn't change because
you caught a blue fish.
00:04:17.740 --> 00:04:20.420
And the other is that you'd
go from i to i minus 1.
00:04:23.700 --> 00:04:26.550
The number of green fish
decreases by 1.
00:04:26.550 --> 00:04:29.020
Now, what we need to do now
is fill in what those
00:04:29.020 --> 00:04:30.110
probabilities are.
00:04:30.110 --> 00:04:36.620
So, if j equals i, meaning that
the number of green fish
00:04:36.620 --> 00:04:39.410
doesn't change, well, what's the
probability that you have
00:04:39.410 --> 00:04:42.380
the same number of green fish
tomorrow as you do today?
00:04:42.380 --> 00:04:46.560
Well, if you have i green fish
today, that happens if you
00:04:46.560 --> 00:04:50.180
catch 1 of the n minus
i blue fish.
00:04:50.180 --> 00:04:52.380
So, what's the probability
of catching one of the n
00:04:52.380 --> 00:04:53.610
minus i blue fish?
00:04:53.610 --> 00:04:59.530
Well, it's n minus i over n.
00:04:59.530 --> 00:05:03.920
Now, the other possible
transition is you go from a i
00:05:03.920 --> 00:05:07.400
to j equals i minus 1,
so i goes down by 1.
00:05:07.400 --> 00:05:10.560
And that happens when you
catch a green fish.
00:05:10.560 --> 00:05:13.000
So, given that there are i
green fish, what's the
00:05:13.000 --> 00:05:15.850
probability that you
catch 1 of those?
00:05:15.850 --> 00:05:19.360
Well, it's going to be i/n.
00:05:19.360 --> 00:05:26.160
And finally, every other
transition has 0 probability.
00:05:26.160 --> 00:05:26.390
All right.
00:05:26.390 --> 00:05:28.170
So, now we can add those
transitions on
00:05:28.170 --> 00:05:31.370
to our Markov chain.
00:05:31.370 --> 00:05:35.590
So, for example,
we have these.
00:05:35.590 --> 00:05:38.900
So, let's look at this
general case i.
00:05:38.900 --> 00:05:41.650
So, if you're state i, you
have i green fish left.
00:05:41.650 --> 00:05:45.690
You will transition to i minus
1 green fish left if that day
00:05:45.690 --> 00:05:46.970
you caught a green fish.
00:05:46.970 --> 00:05:51.260
And we said that that
probability is i/n.
00:05:51.260 --> 00:05:54.045
And the self transition
probability is you caught a
00:05:54.045 --> 00:05:57.170
blue fish that day, so you still
stay a i green fish.
00:05:57.170 --> 00:06:01.115
And that probability, we said,
was n minus i over n.
00:06:01.115 --> 00:06:01.530
All right.
00:06:01.530 --> 00:06:04.680
Now, it's helpful to verify
that this formula works by
00:06:04.680 --> 00:06:08.280
looking at some cases where it's
intuitive to calculate
00:06:08.280 --> 00:06:09.890
what these probabilities
should be.
00:06:09.890 --> 00:06:12.290
So, let's look at state n.
00:06:12.290 --> 00:06:14.845
That is the state where
every single fish
00:06:14.845 --> 00:06:16.550
in the lake is green.
00:06:16.550 --> 00:06:21.240
So, if ever single fish in the
lake is green, then no matter
00:06:21.240 --> 00:06:23.370
what fish you catch, it's
going to be green.
00:06:23.370 --> 00:06:27.580
And you're going to paint it
blue and return it, so you're
00:06:27.580 --> 00:06:30.610
guaranteed to go down to
n minus 1 green fish.
00:06:30.610 --> 00:06:33.750
And so, this transition
probability down to n minus 1
00:06:33.750 --> 00:06:35.580
is guaranteed to be 1.
00:06:35.580 --> 00:06:38.880
And so, the self transition
probability has to be 0.
00:06:38.880 --> 00:06:40.760
Now, let's go back to our
formula and verify that
00:06:40.760 --> 00:06:43.710
actually gives us
the right value.
00:06:43.710 --> 00:06:52.570
So, if i is n, then there's
only these transition
00:06:52.570 --> 00:06:53.460
probabilities.
00:06:53.460 --> 00:06:58.020
So, if i is n, then the
transition probability to j,
00:06:58.020 --> 00:07:03.240
for j is also n, is n minus
n over n, which is 0.
00:07:03.240 --> 00:07:04.480
And that's exactly
what we said.
00:07:04.480 --> 00:07:05.770
We argued that the
self transition
00:07:05.770 --> 00:07:07.570
probability should be 0.
00:07:07.570 --> 00:07:14.090
And also, if i is in, the
probability of transitioning
00:07:14.090 --> 00:07:19.050
to n minus 1 should be
n over n, which is 1.
00:07:19.050 --> 00:07:21.960
And that's exactly what
we argued here.
00:07:21.960 --> 00:07:25.080
So, it seems like these
transition probabilities do
00:07:25.080 --> 00:07:26.140
make sense.
00:07:26.140 --> 00:07:28.160
And if we wanted to, we could
fill in the rest of these.
00:07:28.160 --> 00:07:36.380
So, for example, this would be
2/n, 1/n, n minus 1 over n, n
00:07:36.380 --> 00:07:39.320
minus 2 over n.
00:07:39.320 --> 00:07:43.030
And now, let's also consider
the case of state 0, which
00:07:43.030 --> 00:07:45.180
means that every single
fish is blue.
00:07:45.180 --> 00:07:47.540
There are 0 green fish left.
00:07:47.540 --> 00:07:52.150
Well, if that's the case, then
what's the probability of
00:07:52.150 --> 00:07:53.610
staying at 0?
00:07:53.610 --> 00:07:56.860
Well, that's n minus 0 over
n is 1, all right?
00:07:56.860 --> 00:07:58.890
So, the self transition
probability is 1.
00:07:58.890 --> 00:08:02.710
And that makes sense because
if you have 0 green fish,
00:08:02.710 --> 00:08:04.520
there's no way to generate more
green fish because you
00:08:04.520 --> 00:08:06.160
don't paint blue fish green.
00:08:06.160 --> 00:08:10.120
And so, you're going to stay
at 0 green fish forever.
00:08:10.120 --> 00:08:10.390
All right.
00:08:10.390 --> 00:08:14.520
So, we've characterized the
entire Markov chain now.
00:08:14.520 --> 00:08:17.070
And so, now let's just
answer some simple
00:08:17.070 --> 00:08:19.090
questions about this.
00:08:19.090 --> 00:08:21.510
So, the problem asks us to
identify, what are the
00:08:21.510 --> 00:08:23.460
recurrent and transient
states?
00:08:23.460 --> 00:08:26.430
So, remember that recurrent
state means that if you start
00:08:26.430 --> 00:08:29.880
out at that state, no matter
where you go, what other
00:08:29.880 --> 00:08:33.659
states you end up at, there is
some positive probability path
00:08:33.659 --> 00:08:36.980
that will take you back to
your original state.
00:08:36.980 --> 00:08:40.100
And if you're not recurrent,
then you're transient, which
00:08:40.100 --> 00:08:42.370
means that if you're transient,
if you start out at
00:08:42.370 --> 00:08:45.110
the transient state, there is
some other state that you can
00:08:45.110 --> 00:08:49.090
go to, from which there's no
way to come back to the
00:08:49.090 --> 00:08:51.122
original transient state.
00:08:51.122 --> 00:08:51.620
All right.
00:08:51.620 --> 00:08:54.315
So, now let's look at this and
see which states are recurrent
00:08:54.315 --> 00:08:56.715
and which are transient.
00:08:56.715 --> 00:09:00.120
And we can fill this in more.
00:09:08.690 --> 00:09:13.730
And if we look at it, let's
look at state n.
00:09:13.730 --> 00:09:16.790
Well, we're guaranteed
to go from state n to
00:09:16.790 --> 00:09:18.050
state n minus 1.
00:09:18.050 --> 00:09:20.410
And once we're in state n minus
1, there's no way for us
00:09:20.410 --> 00:09:22.100
to go back to state
n because we can't
00:09:22.100 --> 00:09:23.540
generate more green fish.
00:09:23.540 --> 00:09:25.750
And so, n is transient.
00:09:25.750 --> 00:09:28.060
And similarly, we can use the
same argument to show that
00:09:28.060 --> 00:09:31.360
everything from 1 through n,
all of these states, are
00:09:31.360 --> 00:09:34.120
transient for the same reason
because there's no way to
00:09:34.120 --> 00:09:35.490
generate more green fish.
00:09:35.490 --> 00:09:39.740
And so, the chain can
only stay at a given
00:09:39.740 --> 00:09:43.330
state or go down 1.
00:09:43.330 --> 00:09:45.770
And so, it always goes down.
00:09:45.770 --> 00:09:48.700
It can only go left, and
it can never go right.
00:09:48.700 --> 00:09:50.280
So, once you leave a certain
state, there's
00:09:50.280 --> 00:09:52.240
no way to come back.
00:09:52.240 --> 00:09:55.550
And so, states 1 through
n are all transient.
00:09:55.550 --> 00:10:01.090
And 0 the only recurrent state
because, well, the only place
00:10:01.090 --> 00:10:02.750
you go from 0 is itself.
00:10:02.750 --> 00:10:04.180
So, you always stay at 0.
00:10:04.180 --> 00:10:07.900
And in fact, 0 is not only
recurrent, it's absorbing
00:10:07.900 --> 00:10:11.410
because every single other
state, no matter where you
00:10:11.410 --> 00:10:15.450
start out at, you will
always end up at 0.
00:10:15.450 --> 00:10:17.670
So, this was just an
example of how to
00:10:17.670 --> 00:10:19.100
set up a Markov chain.
00:10:19.100 --> 00:10:21.770
You just think about the actual
dynamics of what's
00:10:21.770 --> 00:10:24.350
going on and make sure that it
satisfies the Markov property.
00:10:24.350 --> 00:10:28.080
Then, figure out what all the
states are and calculate all
00:10:28.080 --> 00:10:29.260
the transition probabilities.
00:10:29.260 --> 00:10:31.720
And once you have that, you've
specified your Markov chain.