1 00:00:00,000 --> 00:00:00,590 2 00:00:00,590 --> 00:00:01,080 Hi. 3 00:00:01,080 --> 00:00:02,930 In this problem, we're going to practice setting up a 4 00:00:02,930 --> 00:00:06,720 Markov chain by going fishing in this lake, which has n fish 5 00:00:06,720 --> 00:00:08,800 in it, some of which are green. 6 00:00:08,800 --> 00:00:10,800 And the rest of the fish are blue. 7 00:00:10,800 --> 00:00:14,260 So, what we do is, every day we go to this lake, and we 8 00:00:14,260 --> 00:00:16,560 catch exactly 1 fish. 9 00:00:16,560 --> 00:00:18,900 And all the fish are equally likely to be 10 00:00:18,900 --> 00:00:20,500 the 1 that's caught. 11 00:00:20,500 --> 00:00:24,560 Now, if we catch a green fish, we paint it blue, and we throw 12 00:00:24,560 --> 00:00:26,460 back into the lake. 13 00:00:26,460 --> 00:00:30,110 And if we catch a blue fish, we just keep it blue, and we 14 00:00:30,110 --> 00:00:31,790 also throw it back. 15 00:00:31,790 --> 00:00:36,200 Now, what we're interested in modeling is, how does this 16 00:00:36,200 --> 00:00:37,470 lake evolve over time? 17 00:00:37,470 --> 00:00:40,315 And specifically what we're interested in is the number of 18 00:00:40,315 --> 00:00:41,720 green fish that are left in the lake. 19 00:00:41,720 --> 00:00:45,320 So, let's let Gi be the event that there are i green fish 20 00:00:45,320 --> 00:00:46,390 left in the lake. 21 00:00:46,390 --> 00:00:49,710 And we want to know, how does Gi evolve over time? 22 00:00:49,710 --> 00:00:51,530 Now, one thing that we've learned that we can use to 23 00:00:51,530 --> 00:00:53,230 model this is a Markov chain. 24 00:00:53,230 --> 00:00:55,550 But before we can use it, we need to make sure that this 25 00:00:55,550 --> 00:00:57,640 actually satisfies the Markov property. 26 00:00:57,640 --> 00:00:59,880 Now, recall that the Markov property essentially says 27 00:00:59,880 --> 00:01:05,840 that, given the current state of the system, that's all you 28 00:01:05,840 --> 00:01:07,970 need in order to predict the future states. 29 00:01:07,970 --> 00:01:11,870 So, any past history of the previous states that it was 30 00:01:11,870 --> 00:01:14,050 in, that's all irrelevant. 31 00:01:14,050 --> 00:01:15,740 All you need is the current state. 32 00:01:15,740 --> 00:01:17,950 Now, in the context of this particular problem, what that 33 00:01:17,950 --> 00:01:21,080 means is that if I tell you that there are 10 green fish 34 00:01:21,080 --> 00:01:24,670 left, that's all the information you need in order 35 00:01:24,670 --> 00:01:27,750 to predict how many fish there will be tomorrow. 36 00:01:27,750 --> 00:01:29,030 So, why is that? 37 00:01:29,030 --> 00:01:33,750 Well, it's because what influences the number of green 38 00:01:33,750 --> 00:01:35,210 fish that are left? 39 00:01:35,210 --> 00:01:39,030 What influences it is which fish you catch because, 40 00:01:39,030 --> 00:01:43,100 depending on which fish you catch, you may paint the green 41 00:01:43,100 --> 00:01:47,830 fish blue, in which case the number of green fish decrease. 42 00:01:47,830 --> 00:01:52,750 But what affects which fish you catch? 43 00:01:52,750 --> 00:01:55,820 Well, that probability is dictated solely based on just 44 00:01:55,820 --> 00:01:59,070 the number of green fish in the lake right now, today. 45 00:01:59,070 --> 00:02:03,220 So, it doesn't matter that there were 20 fish yesterday. 46 00:02:03,220 --> 00:02:07,560 All that matters is how many green fish there are in the 47 00:02:07,560 --> 00:02:08,840 lake today. 48 00:02:08,840 --> 00:02:11,020 And so, because of that argument, the 49 00:02:11,020 --> 00:02:13,240 number of green fish-- 50 00:02:13,240 --> 00:02:16,225 this does satisfy the Markov property, so we can use this 51 00:02:16,225 --> 00:02:18,480 and model it as a Markov chain. 52 00:02:18,480 --> 00:02:23,130 So, like we alluded to just now, the key dynamic that we 53 00:02:23,130 --> 00:02:25,916 need to look at is, how does the number 54 00:02:25,916 --> 00:02:27,050 of green fish change? 55 00:02:27,050 --> 00:02:30,360 And if we look at it, we notice that after each day, 56 00:02:30,360 --> 00:02:36,840 the number of green fish can only have two possible 57 00:02:36,840 --> 00:02:38,330 transitions. 58 00:02:38,330 --> 00:02:40,730 One possible transition is that it goes down by exactly 59 00:02:40,730 --> 00:02:45,230 1, which happens if you happen to catch a green fish and 60 00:02:45,230 --> 00:02:46,490 paint it blue. 61 00:02:46,490 --> 00:02:50,440 So, that green fish is no longer green, so the number of 62 00:02:50,440 --> 00:02:52,080 green fish goes down by 1. 63 00:02:52,080 --> 00:02:56,670 The other possible transition is that Gi doesn't change 64 00:02:56,670 --> 00:02:58,960 because you caught a blue fish that day. 65 00:02:58,960 --> 00:03:01,820 So, all the green fish are still green. 66 00:03:01,820 --> 00:03:05,095 So, now given that, let's see if we can come up with a 67 00:03:05,095 --> 00:03:06,420 Markov chain. 68 00:03:06,420 --> 00:03:09,410 So, the first thing we've done is we've written down all the 69 00:03:09,410 --> 00:03:11,580 different states, right? 70 00:03:11,580 --> 00:03:13,090 So, this represents the number of green 71 00:03:13,090 --> 00:03:14,460 fish left in the lake. 72 00:03:14,460 --> 00:03:18,220 So, there could be 0 green fish left, 1 green fish, all 73 00:03:18,220 --> 00:03:20,975 the way through n, which means that every single fish in the 74 00:03:20,975 --> 00:03:22,910 lake is green. 75 00:03:22,910 --> 00:03:24,010 Now, we have the states. 76 00:03:24,010 --> 00:03:26,690 What we need to do now is to fill in the transition 77 00:03:26,690 --> 00:03:30,660 probabilities, which are the Pij's. 78 00:03:30,660 --> 00:03:36,020 And remember, the Pij is the probability of transitioning 79 00:03:36,020 --> 00:03:39,870 from state i to state j in the next transition. 80 00:03:39,870 --> 00:03:42,240 So, what that means in this context is, what's the 81 00:03:42,240 --> 00:03:51,910 probability that there will be j green fish tomorrow given 82 00:03:51,910 --> 00:03:57,880 that there are i green fish today? 83 00:03:57,880 --> 00:04:01,230 84 00:04:01,230 --> 00:04:06,690 Now, if we go back to our earlier argument, we see that 85 00:04:06,690 --> 00:04:09,420 for any given i, you can only transition to 86 00:04:09,420 --> 00:04:10,870 two possible j's. 87 00:04:10,870 --> 00:04:15,310 One of them is you stay at i because the number of green 88 00:04:15,310 --> 00:04:17,740 fish doesn't change because you caught a blue fish. 89 00:04:17,740 --> 00:04:20,420 And the other is that you'd go from i to i minus 1. 90 00:04:20,420 --> 00:04:23,700 91 00:04:23,700 --> 00:04:26,550 The number of green fish decreases by 1. 92 00:04:26,550 --> 00:04:29,020 Now, what we need to do now is fill in what those 93 00:04:29,020 --> 00:04:30,110 probabilities are. 94 00:04:30,110 --> 00:04:36,620 So, if j equals i, meaning that the number of green fish 95 00:04:36,620 --> 00:04:39,410 doesn't change, well, what's the probability that you have 96 00:04:39,410 --> 00:04:42,380 the same number of green fish tomorrow as you do today? 97 00:04:42,380 --> 00:04:46,560 Well, if you have i green fish today, that happens if you 98 00:04:46,560 --> 00:04:50,180 catch 1 of the n minus i blue fish. 99 00:04:50,180 --> 00:04:52,380 So, what's the probability of catching one of the n 100 00:04:52,380 --> 00:04:53,610 minus i blue fish? 101 00:04:53,610 --> 00:04:59,530 Well, it's n minus i over n. 102 00:04:59,530 --> 00:05:03,920 Now, the other possible transition is you go from a i 103 00:05:03,920 --> 00:05:07,400 to j equals i minus 1, so i goes down by 1. 104 00:05:07,400 --> 00:05:10,560 And that happens when you catch a green fish. 105 00:05:10,560 --> 00:05:13,000 So, given that there are i green fish, what's the 106 00:05:13,000 --> 00:05:15,850 probability that you catch 1 of those? 107 00:05:15,850 --> 00:05:19,360 Well, it's going to be i/n. 108 00:05:19,360 --> 00:05:26,160 And finally, every other transition has 0 probability. 109 00:05:26,160 --> 00:05:26,390 All right. 110 00:05:26,390 --> 00:05:28,170 So, now we can add those transitions on 111 00:05:28,170 --> 00:05:31,370 to our Markov chain. 112 00:05:31,370 --> 00:05:35,590 So, for example, we have these. 113 00:05:35,590 --> 00:05:38,900 So, let's look at this general case i. 114 00:05:38,900 --> 00:05:41,650 So, if you're state i, you have i green fish left. 115 00:05:41,650 --> 00:05:45,690 You will transition to i minus 1 green fish left if that day 116 00:05:45,690 --> 00:05:46,970 you caught a green fish. 117 00:05:46,970 --> 00:05:51,260 And we said that that probability is i/n. 118 00:05:51,260 --> 00:05:54,045 And the self transition probability is you caught a 119 00:05:54,045 --> 00:05:57,170 blue fish that day, so you still stay a i green fish. 120 00:05:57,170 --> 00:06:01,115 And that probability, we said, was n minus i over n. 121 00:06:01,115 --> 00:06:01,530 All right. 122 00:06:01,530 --> 00:06:04,680 Now, it's helpful to verify that this formula works by 123 00:06:04,680 --> 00:06:08,280 looking at some cases where it's intuitive to calculate 124 00:06:08,280 --> 00:06:09,890 what these probabilities should be. 125 00:06:09,890 --> 00:06:12,290 So, let's look at state n. 126 00:06:12,290 --> 00:06:14,845 That is the state where every single fish 127 00:06:14,845 --> 00:06:16,550 in the lake is green. 128 00:06:16,550 --> 00:06:21,240 So, if ever single fish in the lake is green, then no matter 129 00:06:21,240 --> 00:06:23,370 what fish you catch, it's going to be green. 130 00:06:23,370 --> 00:06:27,580 And you're going to paint it blue and return it, so you're 131 00:06:27,580 --> 00:06:30,610 guaranteed to go down to n minus 1 green fish. 132 00:06:30,610 --> 00:06:33,750 And so, this transition probability down to n minus 1 133 00:06:33,750 --> 00:06:35,580 is guaranteed to be 1. 134 00:06:35,580 --> 00:06:38,880 And so, the self transition probability has to be 0. 135 00:06:38,880 --> 00:06:40,760 Now, let's go back to our formula and verify that 136 00:06:40,760 --> 00:06:43,710 actually gives us the right value. 137 00:06:43,710 --> 00:06:52,570 So, if i is n, then there's only these transition 138 00:06:52,570 --> 00:06:53,460 probabilities. 139 00:06:53,460 --> 00:06:58,020 So, if i is n, then the transition probability to j, 140 00:06:58,020 --> 00:07:03,240 for j is also n, is n minus n over n, which is 0. 141 00:07:03,240 --> 00:07:04,480 And that's exactly what we said. 142 00:07:04,480 --> 00:07:05,770 We argued that the self transition 143 00:07:05,770 --> 00:07:07,570 probability should be 0. 144 00:07:07,570 --> 00:07:14,090 And also, if i is in, the probability of transitioning 145 00:07:14,090 --> 00:07:19,050 to n minus 1 should be n over n, which is 1. 146 00:07:19,050 --> 00:07:21,960 And that's exactly what we argued here. 147 00:07:21,960 --> 00:07:25,080 So, it seems like these transition probabilities do 148 00:07:25,080 --> 00:07:26,140 make sense. 149 00:07:26,140 --> 00:07:28,160 And if we wanted to, we could fill in the rest of these. 150 00:07:28,160 --> 00:07:36,380 So, for example, this would be 2/n, 1/n, n minus 1 over n, n 151 00:07:36,380 --> 00:07:39,320 minus 2 over n. 152 00:07:39,320 --> 00:07:43,030 And now, let's also consider the case of state 0, which 153 00:07:43,030 --> 00:07:45,180 means that every single fish is blue. 154 00:07:45,180 --> 00:07:47,540 There are 0 green fish left. 155 00:07:47,540 --> 00:07:52,150 Well, if that's the case, then what's the probability of 156 00:07:52,150 --> 00:07:53,610 staying at 0? 157 00:07:53,610 --> 00:07:56,860 Well, that's n minus 0 over n is 1, all right? 158 00:07:56,860 --> 00:07:58,890 So, the self transition probability is 1. 159 00:07:58,890 --> 00:08:02,710 And that makes sense because if you have 0 green fish, 160 00:08:02,710 --> 00:08:04,520 there's no way to generate more green fish because you 161 00:08:04,520 --> 00:08:06,160 don't paint blue fish green. 162 00:08:06,160 --> 00:08:10,120 And so, you're going to stay at 0 green fish forever. 163 00:08:10,120 --> 00:08:10,390 All right. 164 00:08:10,390 --> 00:08:14,520 So, we've characterized the entire Markov chain now. 165 00:08:14,520 --> 00:08:17,070 And so, now let's just answer some simple 166 00:08:17,070 --> 00:08:19,090 questions about this. 167 00:08:19,090 --> 00:08:21,510 So, the problem asks us to identify, what are the 168 00:08:21,510 --> 00:08:23,460 recurrent and transient states? 169 00:08:23,460 --> 00:08:26,430 So, remember that recurrent state means that if you start 170 00:08:26,430 --> 00:08:29,880 out at that state, no matter where you go, what other 171 00:08:29,880 --> 00:08:33,659 states you end up at, there is some positive probability path 172 00:08:33,659 --> 00:08:36,980 that will take you back to your original state. 173 00:08:36,980 --> 00:08:40,100 And if you're not recurrent, then you're transient, which 174 00:08:40,100 --> 00:08:42,370 means that if you're transient, if you start out at 175 00:08:42,370 --> 00:08:45,110 the transient state, there is some other state that you can 176 00:08:45,110 --> 00:08:49,090 go to, from which there's no way to come back to the 177 00:08:49,090 --> 00:08:51,122 original transient state. 178 00:08:51,122 --> 00:08:51,620 All right. 179 00:08:51,620 --> 00:08:54,315 So, now let's look at this and see which states are recurrent 180 00:08:54,315 --> 00:08:56,715 and which are transient. 181 00:08:56,715 --> 00:09:00,120 And we can fill this in more. 182 00:09:00,120 --> 00:09:08,690 183 00:09:08,690 --> 00:09:13,730 And if we look at it, let's look at state n. 184 00:09:13,730 --> 00:09:16,790 Well, we're guaranteed to go from state n to 185 00:09:16,790 --> 00:09:18,050 state n minus 1. 186 00:09:18,050 --> 00:09:20,410 And once we're in state n minus 1, there's no way for us 187 00:09:20,410 --> 00:09:22,100 to go back to state n because we can't 188 00:09:22,100 --> 00:09:23,540 generate more green fish. 189 00:09:23,540 --> 00:09:25,750 And so, n is transient. 190 00:09:25,750 --> 00:09:28,060 And similarly, we can use the same argument to show that 191 00:09:28,060 --> 00:09:31,360 everything from 1 through n, all of these states, are 192 00:09:31,360 --> 00:09:34,120 transient for the same reason because there's no way to 193 00:09:34,120 --> 00:09:35,490 generate more green fish. 194 00:09:35,490 --> 00:09:39,740 And so, the chain can only stay at a given 195 00:09:39,740 --> 00:09:43,330 state or go down 1. 196 00:09:43,330 --> 00:09:45,770 And so, it always goes down. 197 00:09:45,770 --> 00:09:48,700 It can only go left, and it can never go right. 198 00:09:48,700 --> 00:09:50,280 So, once you leave a certain state, there's 199 00:09:50,280 --> 00:09:52,240 no way to come back. 200 00:09:52,240 --> 00:09:55,550 And so, states 1 through n are all transient. 201 00:09:55,550 --> 00:10:01,090 And 0 the only recurrent state because, well, the only place 202 00:10:01,090 --> 00:10:02,750 you go from 0 is itself. 203 00:10:02,750 --> 00:10:04,180 So, you always stay at 0. 204 00:10:04,180 --> 00:10:07,900 And in fact, 0 is not only recurrent, it's absorbing 205 00:10:07,900 --> 00:10:11,410 because every single other state, no matter where you 206 00:10:11,410 --> 00:10:15,450 start out at, you will always end up at 0. 207 00:10:15,450 --> 00:10:17,670 So, this was just an example of how to 208 00:10:17,670 --> 00:10:19,100 set up a Markov chain. 209 00:10:19,100 --> 00:10:21,770 You just think about the actual dynamics of what's 210 00:10:21,770 --> 00:10:24,350 going on and make sure that it satisfies the Markov property. 211 00:10:24,350 --> 00:10:28,080 Then, figure out what all the states are and calculate all 212 00:10:28,080 --> 00:10:29,260 the transition probabilities. 213 00:10:29,260 --> 00:10:31,720 And once you have that, you've specified your Markov chain. 214 00:10:31,720 --> 00:10:32,970