1 00:00:00,000 --> 00:00:00,590 2 00:00:00,590 --> 00:00:01,420 Hi, everyone. 3 00:00:01,420 --> 00:00:03,270 Today, I'm going to talk about Markov Chain 4 00:00:03,270 --> 00:00:05,050 Practice number one. 5 00:00:05,050 --> 00:00:07,240 Before we start, let's first take a look 6 00:00:07,240 --> 00:00:09,080 at this Markov chain. 7 00:00:09,080 --> 00:00:11,380 This Markov chain has six states. 8 00:00:11,380 --> 00:00:13,720 In this problem, we always assume the process 9 00:00:13,720 --> 00:00:15,980 starts from state S0. 10 00:00:15,980 --> 00:00:18,470 On the first trial, the process can either make a 11 00:00:18,470 --> 00:00:23,080 transition from S0 to S1 with probability 1/3 or from S0 to 12 00:00:23,080 --> 00:00:26,740 S3 with probability 1/3 third or from S0 to S5 with 13 00:00:26,740 --> 00:00:28,460 probability 1/3. 14 00:00:28,460 --> 00:00:31,310 If on the first trial, the process makes the transition 15 00:00:31,310 --> 00:00:35,870 from S0 to S1 or from S0 to S5, it will always be stuck in 16 00:00:35,870 --> 00:00:40,110 either S1 or S5 forever, because both of the states S1 17 00:00:40,110 --> 00:00:44,220 and S5 have a self-transition probability of one. 18 00:00:44,220 --> 00:00:47,160 On the other hand, if on the first trial, the process makes 19 00:00:47,160 --> 00:00:50,950 the transition from S0 to S3, it can then either transition 20 00:00:50,950 --> 00:00:53,920 to the left or transition to the right or make 21 00:00:53,920 --> 00:00:57,430 self-transition back to the state S3. 22 00:00:57,430 --> 00:01:00,570 If the process ever enters the left of the chain, it will 23 00:01:00,570 --> 00:01:03,120 never be able to come to the right. 24 00:01:03,120 --> 00:01:06,030 On the other hand, if the process ever enters the right 25 00:01:06,030 --> 00:01:10,280 of the chain, it would never be able to go to the left. 26 00:01:10,280 --> 00:01:13,200 For part A of the problem, we have to calculate the 27 00:01:13,200 --> 00:01:17,140 probability that the process enter S2 for the first time at 28 00:01:17,140 --> 00:01:18,790 the case trial. 29 00:01:18,790 --> 00:01:22,890 First, notice that it would take at least two trials for 30 00:01:22,890 --> 00:01:26,980 the process to make a transition from S0 to S2. 31 00:01:26,980 --> 00:01:30,830 Therefore, for k equal to 1, the probability of ak is 32 00:01:30,830 --> 00:01:32,080 simply equal to 0. 33 00:01:32,080 --> 00:01:34,740 34 00:01:34,740 --> 00:01:41,860 For k equal to 1, probability of a1 is equal to 0. 35 00:01:41,860 --> 00:01:44,710 36 00:01:44,710 --> 00:01:49,250 Then for k equal to 2, 3 and on, the probability that the 37 00:01:49,250 --> 00:01:52,870 process enters S2 for the first time at a case trial is 38 00:01:52,870 --> 00:01:56,600 equivalent to the probability that the process first makes a 39 00:01:56,600 --> 00:02:01,430 transition from S0 to S3 and then stays in S3 for the next 40 00:02:01,430 --> 00:02:05,060 two k minus 2 trials and finally makes a transition 41 00:02:05,060 --> 00:02:09,520 from S3 to S2 on the kth trial. 42 00:02:09,520 --> 00:02:12,730 So let's write this out. 43 00:02:12,730 --> 00:02:21,530 For k equal to 2, 3, and on, the probability of ak is equal 44 00:02:21,530 --> 00:02:25,100 to the probability that the process first makes transition 45 00:02:25,100 --> 00:02:29,140 from S0 to S3 on the first trial, which is probability 46 00:02:29,140 --> 00:02:34,510 03, times the probability that the process makes 47 00:02:34,510 --> 00:02:38,850 self-transition for the next k minus 2 trials, which is 48 00:02:38,850 --> 00:02:45,860 probability 33 to the power of k minus 2, and finally makes a 49 00:02:45,860 --> 00:02:54,020 transition from S3 to S2 on the kth trial, which is p32. 50 00:02:54,020 --> 00:03:03,340 And this gives us 1/3 times 1/4 to the power of k minus 2 51 00:03:03,340 --> 00:03:11,590 times 1/4, which is equal to 1/3 times 1/4 to 52 00:03:11,590 --> 00:03:13,180 the power of k minus-- 53 00:03:13,180 --> 00:03:15,980 For part B of the problem, we have to calculate the 54 00:03:15,980 --> 00:03:20,550 probability that the process never enters as four. 55 00:03:20,550 --> 00:03:23,180 This event can happen in three ways. 56 00:03:23,180 --> 00:03:25,800 The first way is that the process makes a transition 57 00:03:25,800 --> 00:03:28,530 from S0 to S1 on the first trial and 58 00:03:28,530 --> 00:03:31,060 be stuck in S1 forever. 59 00:03:31,060 --> 00:03:34,070 The second way that the process makes a transition 60 00:03:34,070 --> 00:03:36,700 from S0 to S5 on the first trial and 61 00:03:36,700 --> 00:03:39,170 be stuck at S5 forever. 62 00:03:39,170 --> 00:03:42,120 The third way is that the process makes a transition 63 00:03:42,120 --> 00:03:45,790 from S0 to S3 on the first trial and then it makes a 64 00:03:45,790 --> 00:03:49,920 transition from S3 to S2 on the next state change so that 65 00:03:49,920 --> 00:03:53,700 it would never be able to go to S4. 66 00:03:53,700 --> 00:03:56,880 Therefore, the probability of B is equal to the sum of 67 00:03:56,880 --> 00:03:59,260 probabilities of this three events. 68 00:03:59,260 --> 00:04:02,360 69 00:04:02,360 --> 00:04:08,120 So the probability of B is equal to the probability that 70 00:04:08,120 --> 00:04:10,960 the process makes a transition from S0 to S1 on the first 71 00:04:10,960 --> 00:04:15,670 trial, which is 1/3, plus the probability that the process 72 00:04:15,670 --> 00:04:19,130 makes a transition from S0 to S5 on the first trial, which 73 00:04:19,130 --> 00:04:23,140 is also 1/3, plus the probability that the process 74 00:04:23,140 --> 00:04:28,590 makes a transition from S0 to S3 on the first trial times 75 00:04:28,590 --> 00:04:32,060 the probability that the process then makes a 76 00:04:32,060 --> 00:04:37,090 transition from S3 to S2 on the next state change. 77 00:04:37,090 --> 00:04:45,740 So transition to S2, given that the processes are already 78 00:04:45,740 --> 00:04:49,110 in state S3 and there's a state change. 79 00:04:49,110 --> 00:04:57,860 80 00:04:57,860 --> 00:05:00,950 Let's take a look at this conditional probability. 81 00:05:00,950 --> 00:05:03,350 The condition that the processes are already in state 82 00:05:03,350 --> 00:05:07,660 S3 and there's a state change imply two possible events, 83 00:05:07,660 --> 00:05:12,880 which are the transition from S3 to S2 and the transition 84 00:05:12,880 --> 00:05:16,380 from S3 to S4. 85 00:05:16,380 --> 00:05:23,850 Therefore, we can write this conditional probability as the 86 00:05:23,850 --> 00:05:32,020 conditional probability of transition from as S3 to S2, 87 00:05:32,020 --> 00:05:39,730 given that another event, S3 to S2 or 88 00:05:39,730 --> 00:05:46,000 S3 to S4 has happened. 89 00:05:46,000 --> 00:05:56,270 And this is simply equal to the proportion of p32 and p32 90 00:05:56,270 --> 00:06:07,300 plus p34, which is equal to 1/4 over 1/4 plus 1/2, which 91 00:06:07,300 --> 00:06:09,830 is equal to 1/3. 92 00:06:09,830 --> 00:06:17,870 Therefore, the probability of B is equal to 1/3 plus 1/3 93 00:06:17,870 --> 00:06:26,345 plus 1/3 times the 1/3 here, which is equal to 7/9. 94 00:06:26,345 --> 00:06:29,390 95 00:06:29,390 --> 00:06:32,350 For part C of the problem, we have to calculate the 96 00:06:32,350 --> 00:06:36,170 probability that the process enter S2 and leaves S2 on the 97 00:06:36,170 --> 00:06:37,820 next trial. 98 00:06:37,820 --> 00:06:40,560 This probability can be written as the product of two 99 00:06:40,560 --> 00:06:42,240 probabilities-- 100 00:06:42,240 --> 00:06:45,430 the probability that the process enters S2 and the 101 00:06:45,430 --> 00:06:49,440 probability that it leaves S2 on the next trial, given it's 102 00:06:49,440 --> 00:06:51,800 already in S2. 103 00:06:51,800 --> 00:06:54,630 Let's first look at the probability that the 104 00:06:54,630 --> 00:06:57,200 process enters S2. 105 00:06:57,200 --> 00:07:00,550 Using a similar approach as part B, we know that the 106 00:07:00,550 --> 00:07:04,890 probability the process ever enters S2 is equal to the 107 00:07:04,890 --> 00:07:08,220 probability of the event that the process first makes a 108 00:07:08,220 --> 00:07:12,650 transition from S0 to S3 on the first trial and then makes 109 00:07:12,650 --> 00:07:18,350 a transition from S3 to S2 on the next state change. 110 00:07:18,350 --> 00:07:23,900 So the probability that the process enters S2 is equal to 111 00:07:23,900 --> 00:07:27,430 the probability that it first makes a transition from S0 to 112 00:07:27,430 --> 00:07:32,350 S3 on the first trial, which is P03, times the probability 113 00:07:32,350 --> 00:07:42,175 that it makes a transition to S2, given that it's already in 114 00:07:42,175 --> 00:07:45,850 S3 and there is a state change. 115 00:07:45,850 --> 00:07:52,620 116 00:07:52,620 --> 00:07:56,090 We have already calculated this conditional probability 117 00:07:56,090 --> 00:08:01,610 in part B. Let's then look at the second probability term, 118 00:08:01,610 --> 00:08:04,160 the probability that the process leaves S2 on the next 119 00:08:04,160 --> 00:08:07,690 trial, given that it's already in S2. 120 00:08:07,690 --> 00:08:12,630 So given that the process is already in S2, it can take two 121 00:08:12,630 --> 00:08:14,020 transitions. 122 00:08:14,020 --> 00:08:18,170 In can either transition from S2 to S1 or make a 123 00:08:18,170 --> 00:08:22,090 self-transition from S2 back to S2. 124 00:08:22,090 --> 00:08:26,660 Therefore, this conditional probability that it leaves S2 125 00:08:26,660 --> 00:08:29,700 on the next trial, given that it was already in S2 is simply 126 00:08:29,700 --> 00:08:33,230 equal to the transition probability from S2 to S1, 127 00:08:33,230 --> 00:08:35,575 which is P21. 128 00:08:35,575 --> 00:08:38,270 129 00:08:38,270 --> 00:08:46,510 Therefore, this is equal to P03, which is 1/3, times 1/3 130 00:08:46,510 --> 00:08:54,040 from the result from part B times P21, which is 1/2, and 131 00:08:54,040 --> 00:08:55,460 gives us 1/18. 132 00:08:55,460 --> 00:08:58,360 133 00:08:58,360 --> 00:09:01,200 For part D of the problem, we have to calculate the 134 00:09:01,200 --> 00:09:04,510 probability that the process enters S1 for the first time 135 00:09:04,510 --> 00:09:06,840 on the third trial. 136 00:09:06,840 --> 00:09:09,240 So if you take a look at this Markov chain, you'll notice 137 00:09:09,240 --> 00:09:12,510 that the only way for this event to happen is when a 138 00:09:12,510 --> 00:09:16,420 process first makes a transition from S0 to S3 on 139 00:09:16,420 --> 00:09:20,450 the first trial and from S3 to S2 on the second trial and 140 00:09:20,450 --> 00:09:23,260 from S2 to S1 on the third trial. 141 00:09:23,260 --> 00:09:29,900 Therefore, the probability of D is equal to the probability 142 00:09:29,900 --> 00:09:35,280 of the event that the process makes a transition from S0 to 143 00:09:35,280 --> 00:09:41,040 S3 on the first trial and from S3 to S2 on the second trial 144 00:09:41,040 --> 00:09:45,210 and finally from S2 to S1 on the third trial. 145 00:09:45,210 --> 00:09:56,140 So this is equal to P03 times P32 times P21, which is equal 146 00:09:56,140 --> 00:10:07,570 to 1/3 times 1/4 times 1/2, which is equal to 1/24. 147 00:10:07,570 --> 00:10:11,300 148 00:10:11,300 --> 00:10:14,140 For part E of the problem, we have to calculate the 149 00:10:14,140 --> 00:10:17,540 probability that the process is in S3 immediately 150 00:10:17,540 --> 00:10:19,390 after the nth trial. 151 00:10:19,390 --> 00:10:22,040 If you take a look at this Markov chain, you'll notice 152 00:10:22,040 --> 00:10:25,500 that if on the first trial, the process makes a transition 153 00:10:25,500 --> 00:10:29,510 from S0 to S1 or from S0 to S5, it will never be 154 00:10:29,510 --> 00:10:31,620 able to go to S3. 155 00:10:31,620 --> 00:10:34,600 On the other hand, if on the first trial, the process makes 156 00:10:34,600 --> 00:10:39,950 a transition from S0 to S3 and if it leaves S3 at some point, 157 00:10:39,950 --> 00:10:43,290 it will never be able to come back to S3. 158 00:10:43,290 --> 00:10:47,130 Therefore, in order for the process to be S3 immediately 159 00:10:47,130 --> 00:10:50,410 after the nth trial, we will need the process to first make 160 00:10:50,410 --> 00:10:55,585 transition from S0 to S3 on the first trial and then stay 161 00:10:55,585 --> 00:10:58,810 in S3 for the next n minus 1 trials. 162 00:10:58,810 --> 00:11:06,880 163 00:11:06,880 --> 00:11:12,330 Therefore, the probability of the event e is simply equal to 164 00:11:12,330 --> 00:11:19,620 the probability of this event, which is equal to P03 times 165 00:11:19,620 --> 00:11:31,780 P33 to the power n minus 1, which is equal to 1/3 times 166 00:11:31,780 --> 00:11:38,420 1/4 to the power of n minus 1. 167 00:11:38,420 --> 00:11:40,410 And this concludes our practice on 168 00:11:40,410 --> 00:11:41,660 Markov chain today. 169 00:11:41,660 --> 00:11:42,600