1
00:00:00,000 --> 00:00:00,590
2
00:00:00,590 --> 00:00:01,080
Hi.
3
00:00:01,080 --> 00:00:02,930
In this problem, we're going
to practice setting up a
4
00:00:02,930 --> 00:00:06,720
Markov chain by going fishing in
this lake, which has n fish
5
00:00:06,720 --> 00:00:08,800
in it, some of which
are green.
6
00:00:08,800 --> 00:00:10,800
And the rest of the
fish are blue.
7
00:00:10,800 --> 00:00:14,260
So, what we do is, every day
we go to this lake, and we
8
00:00:14,260 --> 00:00:16,560
catch exactly 1 fish.
9
00:00:16,560 --> 00:00:18,900
And all the fish are equally
likely to be
10
00:00:18,900 --> 00:00:20,500
the 1 that's caught.
11
00:00:20,500 --> 00:00:24,560
Now, if we catch a green fish,
we paint it blue, and we throw
12
00:00:24,560 --> 00:00:26,460
back into the lake.
13
00:00:26,460 --> 00:00:30,110
And if we catch a blue fish, we
just keep it blue, and we
14
00:00:30,110 --> 00:00:31,790
also throw it back.
15
00:00:31,790 --> 00:00:36,200
Now, what we're interested in
modeling is, how does this
16
00:00:36,200 --> 00:00:37,470
lake evolve over time?
17
00:00:37,470 --> 00:00:40,315
And specifically what we're
interested in is the number of
18
00:00:40,315 --> 00:00:41,720
green fish that are
left in the lake.
19
00:00:41,720 --> 00:00:45,320
So, let's let Gi be the event
that there are i green fish
20
00:00:45,320 --> 00:00:46,390
left in the lake.
21
00:00:46,390 --> 00:00:49,710
And we want to know, how does
Gi evolve over time?
22
00:00:49,710 --> 00:00:51,530
Now, one thing that we've
learned that we can use to
23
00:00:51,530 --> 00:00:53,230
model this is a Markov chain.
24
00:00:53,230 --> 00:00:55,550
But before we can use it, we
need to make sure that this
25
00:00:55,550 --> 00:00:57,640
actually satisfies the
Markov property.
26
00:00:57,640 --> 00:00:59,880
Now, recall that the Markov
property essentially says
27
00:00:59,880 --> 00:01:05,840
that, given the current state of
the system, that's all you
28
00:01:05,840 --> 00:01:07,970
need in order to predict
the future states.
29
00:01:07,970 --> 00:01:11,870
So, any past history of the
previous states that it was
30
00:01:11,870 --> 00:01:14,050
in, that's all irrelevant.
31
00:01:14,050 --> 00:01:15,740
All you need is the
current state.
32
00:01:15,740 --> 00:01:17,950
Now, in the context of this
particular problem, what that
33
00:01:17,950 --> 00:01:21,080
means is that if I tell you that
there are 10 green fish
34
00:01:21,080 --> 00:01:24,670
left, that's all the information
you need in order
35
00:01:24,670 --> 00:01:27,750
to predict how many fish
there will be tomorrow.
36
00:01:27,750 --> 00:01:29,030
So, why is that?
37
00:01:29,030 --> 00:01:33,750
Well, it's because what
influences the number of green
38
00:01:33,750 --> 00:01:35,210
fish that are left?
39
00:01:35,210 --> 00:01:39,030
What influences it is which
fish you catch because,
40
00:01:39,030 --> 00:01:43,100
depending on which fish you
catch, you may paint the green
41
00:01:43,100 --> 00:01:47,830
fish blue, in which case the
number of green fish decrease.
42
00:01:47,830 --> 00:01:52,750
But what affects which
fish you catch?
43
00:01:52,750 --> 00:01:55,820
Well, that probability is
dictated solely based on just
44
00:01:55,820 --> 00:01:59,070
the number of green fish in
the lake right now, today.
45
00:01:59,070 --> 00:02:03,220
So, it doesn't matter that there
were 20 fish yesterday.
46
00:02:03,220 --> 00:02:07,560
All that matters is how many
green fish there are in the
47
00:02:07,560 --> 00:02:08,840
lake today.
48
00:02:08,840 --> 00:02:11,020
And so, because of that
argument, the
49
00:02:11,020 --> 00:02:13,240
number of green fish--
50
00:02:13,240 --> 00:02:16,225
this does satisfy the Markov
property, so we can use this
51
00:02:16,225 --> 00:02:18,480
and model it as a
Markov chain.
52
00:02:18,480 --> 00:02:23,130
So, like we alluded to just now,
the key dynamic that we
53
00:02:23,130 --> 00:02:25,916
need to look at is, how
does the number
54
00:02:25,916 --> 00:02:27,050
of green fish change?
55
00:02:27,050 --> 00:02:30,360
And if we look at it, we notice
that after each day,
56
00:02:30,360 --> 00:02:36,840
the number of green fish can
only have two possible
57
00:02:36,840 --> 00:02:38,330
transitions.
58
00:02:38,330 --> 00:02:40,730
One possible transition is that
it goes down by exactly
59
00:02:40,730 --> 00:02:45,230
1, which happens if you happen
to catch a green fish and
60
00:02:45,230 --> 00:02:46,490
paint it blue.
61
00:02:46,490 --> 00:02:50,440
So, that green fish is no longer
green, so the number of
62
00:02:50,440 --> 00:02:52,080
green fish goes down by 1.
63
00:02:52,080 --> 00:02:56,670
The other possible transition
is that Gi doesn't change
64
00:02:56,670 --> 00:02:58,960
because you caught a
blue fish that day.
65
00:02:58,960 --> 00:03:01,820
So, all the green fish
are still green.
66
00:03:01,820 --> 00:03:05,095
So, now given that, let's see
if we can come up with a
67
00:03:05,095 --> 00:03:06,420
Markov chain.
68
00:03:06,420 --> 00:03:09,410
So, the first thing we've done
is we've written down all the
69
00:03:09,410 --> 00:03:11,580
different states, right?
70
00:03:11,580 --> 00:03:13,090
So, this represents the
number of green
71
00:03:13,090 --> 00:03:14,460
fish left in the lake.
72
00:03:14,460 --> 00:03:18,220
So, there could be 0 green fish
left, 1 green fish, all
73
00:03:18,220 --> 00:03:20,975
the way through n, which means
that every single fish in the
74
00:03:20,975 --> 00:03:22,910
lake is green.
75
00:03:22,910 --> 00:03:24,010
Now, we have the states.
76
00:03:24,010 --> 00:03:26,690
What we need to do now is to
fill in the transition
77
00:03:26,690 --> 00:03:30,660
probabilities, which
are the Pij's.
78
00:03:30,660 --> 00:03:36,020
And remember, the Pij is the
probability of transitioning
79
00:03:36,020 --> 00:03:39,870
from state i to state j in
the next transition.
80
00:03:39,870 --> 00:03:42,240
So, what that means in this
context is, what's the
81
00:03:42,240 --> 00:03:51,910
probability that there will be
j green fish tomorrow given
82
00:03:51,910 --> 00:03:57,880
that there are i green
fish today?
83
00:03:57,880 --> 00:04:01,230
84
00:04:01,230 --> 00:04:06,690
Now, if we go back to our
earlier argument, we see that
85
00:04:06,690 --> 00:04:09,420
for any given i, you can
only transition to
86
00:04:09,420 --> 00:04:10,870
two possible j's.
87
00:04:10,870 --> 00:04:15,310
One of them is you stay at i
because the number of green
88
00:04:15,310 --> 00:04:17,740
fish doesn't change because
you caught a blue fish.
89
00:04:17,740 --> 00:04:20,420
And the other is that you'd
go from i to i minus 1.
90
00:04:20,420 --> 00:04:23,700
91
00:04:23,700 --> 00:04:26,550
The number of green fish
decreases by 1.
92
00:04:26,550 --> 00:04:29,020
Now, what we need to do now
is fill in what those
93
00:04:29,020 --> 00:04:30,110
probabilities are.
94
00:04:30,110 --> 00:04:36,620
So, if j equals i, meaning that
the number of green fish
95
00:04:36,620 --> 00:04:39,410
doesn't change, well, what's the
probability that you have
96
00:04:39,410 --> 00:04:42,380
the same number of green fish
tomorrow as you do today?
97
00:04:42,380 --> 00:04:46,560
Well, if you have i green fish
today, that happens if you
98
00:04:46,560 --> 00:04:50,180
catch 1 of the n minus
i blue fish.
99
00:04:50,180 --> 00:04:52,380
So, what's the probability
of catching one of the n
100
00:04:52,380 --> 00:04:53,610
minus i blue fish?
101
00:04:53,610 --> 00:04:59,530
Well, it's n minus i over n.
102
00:04:59,530 --> 00:05:03,920
Now, the other possible
transition is you go from a i
103
00:05:03,920 --> 00:05:07,400
to j equals i minus 1,
so i goes down by 1.
104
00:05:07,400 --> 00:05:10,560
And that happens when you
catch a green fish.
105
00:05:10,560 --> 00:05:13,000
So, given that there are i
green fish, what's the
106
00:05:13,000 --> 00:05:15,850
probability that you
catch 1 of those?
107
00:05:15,850 --> 00:05:19,360
Well, it's going to be i/n.
108
00:05:19,360 --> 00:05:26,160
And finally, every other
transition has 0 probability.
109
00:05:26,160 --> 00:05:26,390
All right.
110
00:05:26,390 --> 00:05:28,170
So, now we can add those
transitions on
111
00:05:28,170 --> 00:05:31,370
to our Markov chain.
112
00:05:31,370 --> 00:05:35,590
So, for example,
we have these.
113
00:05:35,590 --> 00:05:38,900
So, let's look at this
general case i.
114
00:05:38,900 --> 00:05:41,650
So, if you're state i, you
have i green fish left.
115
00:05:41,650 --> 00:05:45,690
You will transition to i minus
1 green fish left if that day
116
00:05:45,690 --> 00:05:46,970
you caught a green fish.
117
00:05:46,970 --> 00:05:51,260
And we said that that
probability is i/n.
118
00:05:51,260 --> 00:05:54,045
And the self transition
probability is you caught a
119
00:05:54,045 --> 00:05:57,170
blue fish that day, so you still
stay a i green fish.
120
00:05:57,170 --> 00:06:01,115
And that probability, we said,
was n minus i over n.
121
00:06:01,115 --> 00:06:01,530
All right.
122
00:06:01,530 --> 00:06:04,680
Now, it's helpful to verify
that this formula works by
123
00:06:04,680 --> 00:06:08,280
looking at some cases where it's
intuitive to calculate
124
00:06:08,280 --> 00:06:09,890
what these probabilities
should be.
125
00:06:09,890 --> 00:06:12,290
So, let's look at state n.
126
00:06:12,290 --> 00:06:14,845
That is the state where
every single fish
127
00:06:14,845 --> 00:06:16,550
in the lake is green.
128
00:06:16,550 --> 00:06:21,240
So, if ever single fish in the
lake is green, then no matter
129
00:06:21,240 --> 00:06:23,370
what fish you catch, it's
going to be green.
130
00:06:23,370 --> 00:06:27,580
And you're going to paint it
blue and return it, so you're
131
00:06:27,580 --> 00:06:30,610
guaranteed to go down to
n minus 1 green fish.
132
00:06:30,610 --> 00:06:33,750
And so, this transition
probability down to n minus 1
133
00:06:33,750 --> 00:06:35,580
is guaranteed to be 1.
134
00:06:35,580 --> 00:06:38,880
And so, the self transition
probability has to be 0.
135
00:06:38,880 --> 00:06:40,760
Now, let's go back to our
formula and verify that
136
00:06:40,760 --> 00:06:43,710
actually gives us
the right value.
137
00:06:43,710 --> 00:06:52,570
So, if i is n, then there's
only these transition
138
00:06:52,570 --> 00:06:53,460
probabilities.
139
00:06:53,460 --> 00:06:58,020
So, if i is n, then the
transition probability to j,
140
00:06:58,020 --> 00:07:03,240
for j is also n, is n minus
n over n, which is 0.
141
00:07:03,240 --> 00:07:04,480
And that's exactly
what we said.
142
00:07:04,480 --> 00:07:05,770
We argued that the
self transition
143
00:07:05,770 --> 00:07:07,570
probability should be 0.
144
00:07:07,570 --> 00:07:14,090
And also, if i is in, the
probability of transitioning
145
00:07:14,090 --> 00:07:19,050
to n minus 1 should be
n over n, which is 1.
146
00:07:19,050 --> 00:07:21,960
And that's exactly what
we argued here.
147
00:07:21,960 --> 00:07:25,080
So, it seems like these
transition probabilities do
148
00:07:25,080 --> 00:07:26,140
make sense.
149
00:07:26,140 --> 00:07:28,160
And if we wanted to, we could
fill in the rest of these.
150
00:07:28,160 --> 00:07:36,380
So, for example, this would be
2/n, 1/n, n minus 1 over n, n
151
00:07:36,380 --> 00:07:39,320
minus 2 over n.
152
00:07:39,320 --> 00:07:43,030
And now, let's also consider
the case of state 0, which
153
00:07:43,030 --> 00:07:45,180
means that every single
fish is blue.
154
00:07:45,180 --> 00:07:47,540
There are 0 green fish left.
155
00:07:47,540 --> 00:07:52,150
Well, if that's the case, then
what's the probability of
156
00:07:52,150 --> 00:07:53,610
staying at 0?
157
00:07:53,610 --> 00:07:56,860
Well, that's n minus 0 over
n is 1, all right?
158
00:07:56,860 --> 00:07:58,890
So, the self transition
probability is 1.
159
00:07:58,890 --> 00:08:02,710
And that makes sense because
if you have 0 green fish,
160
00:08:02,710 --> 00:08:04,520
there's no way to generate more
green fish because you
161
00:08:04,520 --> 00:08:06,160
don't paint blue fish green.
162
00:08:06,160 --> 00:08:10,120
And so, you're going to stay
at 0 green fish forever.
163
00:08:10,120 --> 00:08:10,390
All right.
164
00:08:10,390 --> 00:08:14,520
So, we've characterized the
entire Markov chain now.
165
00:08:14,520 --> 00:08:17,070
And so, now let's just
answer some simple
166
00:08:17,070 --> 00:08:19,090
questions about this.
167
00:08:19,090 --> 00:08:21,510
So, the problem asks us to
identify, what are the
168
00:08:21,510 --> 00:08:23,460
recurrent and transient
states?
169
00:08:23,460 --> 00:08:26,430
So, remember that recurrent
state means that if you start
170
00:08:26,430 --> 00:08:29,880
out at that state, no matter
where you go, what other
171
00:08:29,880 --> 00:08:33,659
states you end up at, there is
some positive probability path
172
00:08:33,659 --> 00:08:36,980
that will take you back to
your original state.
173
00:08:36,980 --> 00:08:40,100
And if you're not recurrent,
then you're transient, which
174
00:08:40,100 --> 00:08:42,370
means that if you're transient,
if you start out at
175
00:08:42,370 --> 00:08:45,110
the transient state, there is
some other state that you can
176
00:08:45,110 --> 00:08:49,090
go to, from which there's no
way to come back to the
177
00:08:49,090 --> 00:08:51,122
original transient state.
178
00:08:51,122 --> 00:08:51,620
All right.
179
00:08:51,620 --> 00:08:54,315
So, now let's look at this and
see which states are recurrent
180
00:08:54,315 --> 00:08:56,715
and which are transient.
181
00:08:56,715 --> 00:09:00,120
And we can fill this in more.
182
00:09:00,120 --> 00:09:08,690
183
00:09:08,690 --> 00:09:13,730
And if we look at it, let's
look at state n.
184
00:09:13,730 --> 00:09:16,790
Well, we're guaranteed
to go from state n to
185
00:09:16,790 --> 00:09:18,050
state n minus 1.
186
00:09:18,050 --> 00:09:20,410
And once we're in state n minus
1, there's no way for us
187
00:09:20,410 --> 00:09:22,100
to go back to state
n because we can't
188
00:09:22,100 --> 00:09:23,540
generate more green fish.
189
00:09:23,540 --> 00:09:25,750
And so, n is transient.
190
00:09:25,750 --> 00:09:28,060
And similarly, we can use the
same argument to show that
191
00:09:28,060 --> 00:09:31,360
everything from 1 through n,
all of these states, are
192
00:09:31,360 --> 00:09:34,120
transient for the same reason
because there's no way to
193
00:09:34,120 --> 00:09:35,490
generate more green fish.
194
00:09:35,490 --> 00:09:39,740
And so, the chain can
only stay at a given
195
00:09:39,740 --> 00:09:43,330
state or go down 1.
196
00:09:43,330 --> 00:09:45,770
And so, it always goes down.
197
00:09:45,770 --> 00:09:48,700
It can only go left, and
it can never go right.
198
00:09:48,700 --> 00:09:50,280
So, once you leave a certain
state, there's
199
00:09:50,280 --> 00:09:52,240
no way to come back.
200
00:09:52,240 --> 00:09:55,550
And so, states 1 through
n are all transient.
201
00:09:55,550 --> 00:10:01,090
And 0 the only recurrent state
because, well, the only place
202
00:10:01,090 --> 00:10:02,750
you go from 0 is itself.
203
00:10:02,750 --> 00:10:04,180
So, you always stay at 0.
204
00:10:04,180 --> 00:10:07,900
And in fact, 0 is not only
recurrent, it's absorbing
205
00:10:07,900 --> 00:10:11,410
because every single other
state, no matter where you
206
00:10:11,410 --> 00:10:15,450
start out at, you will
always end up at 0.
207
00:10:15,450 --> 00:10:17,670
So, this was just an
example of how to
208
00:10:17,670 --> 00:10:19,100
set up a Markov chain.
209
00:10:19,100 --> 00:10:21,770
You just think about the actual
dynamics of what's
210
00:10:21,770 --> 00:10:24,350
going on and make sure that it
satisfies the Markov property.
211
00:10:24,350 --> 00:10:28,080
Then, figure out what all the
states are and calculate all
212
00:10:28,080 --> 00:10:29,260
the transition probabilities.
213
00:10:29,260 --> 00:10:31,720
And once you have that, you've
specified your Markov chain.
214
00:10:31,720 --> 00:10:32,970