1 00:00:00,000 --> 00:00:02,640 2 00:00:02,640 --> 00:00:05,700 In this problem, we are looking at a student whose 3 00:00:05,700 --> 00:00:09,070 performance from day to day sort of oscillates according 4 00:00:09,070 --> 00:00:10,610 to a Markov chain. 5 00:00:10,610 --> 00:00:15,360 In particular, the student can either be in state 1, which is 6 00:00:15,360 --> 00:00:23,740 a state of being up to date, or in state 2, which is a 7 00:00:23,740 --> 00:00:28,120 state of being kind of fallen behind. 8 00:00:28,120 --> 00:00:30,370 Now, the transition probabilities between these 9 00:00:30,370 --> 00:00:37,000 two states are given by the numbers here, which is 0.2 10 00:00:37,000 --> 00:00:43,977 from state 1 to 2, 0.6 from 2 to 1, 0.4 from 2 back to 2, 11 00:00:43,977 --> 00:00:49,840 and 0.8 from 1 back to state 1. 12 00:00:49,840 --> 00:00:53,140 The quantity we're interesting calculating is this notion of 13 00:00:53,140 --> 00:00:54,810 first passage time. 14 00:00:54,810 --> 00:00:57,090 Let me define what that means. 15 00:00:57,090 --> 00:00:59,810 Suppose we are looking at a time horizon of 16 00:00:59,810 --> 00:01:03,870 time 0, 1, 2, 3. 17 00:01:03,870 --> 00:01:07,970 And let's call the state of the Markov chain x of t. 18 00:01:07,970 --> 00:01:13,000 Suppose we start from the chain being in state 2 here. 19 00:01:13,000 --> 00:01:15,490 Now, if we look at a particular sample path, let's 20 00:01:15,490 --> 00:01:23,010 say 2 and 2 again on day 1, and 2 again on day 2, and on 21 00:01:23,010 --> 00:01:27,330 day 3, the student enters state 1. 22 00:01:27,330 --> 00:01:32,790 So in this sample path, we start from time 0 and time 3 23 00:01:32,790 --> 00:01:36,010 is the first time we enter state 1. 24 00:01:36,010 --> 00:01:39,120 And we'll say that the first passage time, namely, the 25 00:01:39,120 --> 00:01:44,312 first time we enter state 1 in this case, is equal to 3. 26 00:01:44,312 --> 00:01:53,010 More formally, we'll define tj as the first pass the time to 27 00:01:53,010 --> 00:02:03,150 state 1 conditional on that we start from state j at time 0. 28 00:02:03,150 --> 00:02:14,620 29 00:02:14,620 --> 00:02:16,900 Now, this quantity, of course, is random. 30 00:02:16,900 --> 00:02:19,740 Depending on the realization, we have different numbers. 31 00:02:19,740 --> 00:02:22,380 And we are interested in calculating the 32 00:02:22,380 --> 00:02:26,650 expected value of t2. 33 00:02:26,650 --> 00:02:30,160 That is, on average, if we start from state 2 here, how 34 00:02:30,160 --> 00:02:34,810 long would it take for us to enter state 1? 35 00:02:34,810 --> 00:02:37,750 Now to calculate this quantity, in the following 36 00:02:37,750 --> 00:02:39,910 recursion will be very important. 37 00:02:39,910 --> 00:02:43,900 The idea is we don't know exactly what t2 is. 38 00:02:43,900 --> 00:02:48,300 But t2 has to satisfy a certain recurrent equation, 39 00:02:48,300 --> 00:02:57,070 namely, t2 must be equal to 1 plus summation j 40 00:02:57,070 --> 00:03:00,226 equal to 1 to 2 P2jtj. 41 00:03:00,226 --> 00:03:04,815 42 00:03:04,815 --> 00:03:08,390 Now let me explain what this equation means. 43 00:03:08,390 --> 00:03:12,840 Let's say we are at state 2. 44 00:03:12,840 --> 00:03:16,100 Well, we don't actually know how long it's going to take 45 00:03:16,100 --> 00:03:17,690 for us to enter state 1. 46 00:03:17,690 --> 00:03:20,800 But we do know after one step, I will be go 47 00:03:20,800 --> 00:03:22,530 into some other state. 48 00:03:22,530 --> 00:03:24,660 Let's call it state j. 49 00:03:24,660 --> 00:03:28,910 And from state j, it's going to take some time to enter 50 00:03:28,910 --> 00:03:31,330 state 1 finally. 51 00:03:31,330 --> 00:03:34,710 So this equation essentially says the time for us to first 52 00:03:34,710 --> 00:03:37,680 enter state 1 from 2 is 1-- 53 00:03:37,680 --> 00:03:39,490 which is the next step-- 54 00:03:39,490 --> 00:03:44,670 plus the expected time from that point on to enter 1. 55 00:03:44,670 --> 00:03:47,100 So that constitutes our [? recurrent ?] 56 00:03:47,100 --> 00:03:48,490 relationship. 57 00:03:48,490 --> 00:03:53,480 Now, by this definition, we can see that this is simply 1 58 00:03:53,480 --> 00:04:03,030 plus P21 times t1 plus P22 times t2. 59 00:04:03,030 --> 00:04:09,450 Now, the definition of tj says t1 must be 0 because, by 60 00:04:09,450 --> 00:04:12,700 definition, if we start from state 1, we are 61 00:04:12,700 --> 00:04:13,930 already in state 1. 62 00:04:13,930 --> 00:04:16,360 So the time to reach state 1 is simply 0. 63 00:04:16,360 --> 00:04:18,709 So this term disappears. 64 00:04:18,709 --> 00:04:24,740 And we end up with 1 plus P22 t2. 65 00:04:24,740 --> 00:04:29,150 If we plug in a number of P22-- 66 00:04:29,150 --> 00:04:32,030 which is 0.4 right here-- 67 00:04:32,030 --> 00:04:36,840 we get 1 plus 0.4 t2. 68 00:04:36,840 --> 00:04:41,960 Now we started from t2 and we ended up with another 69 00:04:41,960 --> 00:04:45,420 expression involving numbers and only one 70 00:04:45,420 --> 00:04:47,570 unknown, which is t2. 71 00:04:47,570 --> 00:04:54,220 Combining this together and solving for t2, we get t2 72 00:04:54,220 --> 00:05:03,400 equals 1 divided by 1 minus 0.4, which is 5/3. 73 00:05:03,400 --> 00:05:08,920 And that is the answer for the first part of the problem. 74 00:05:08,920 --> 00:05:11,550 In the second part of the problem, we are asked to do 75 00:05:11,550 --> 00:05:15,510 something similar as before but with a slight twist. 76 00:05:15,510 --> 00:05:18,050 Here, I copied over the definition for tj, which is 77 00:05:18,050 --> 00:05:22,460 the first time to visit state 1 starting from state j at 78 00:05:22,460 --> 00:05:23,960 time t equals 0. 79 00:05:23,960 --> 00:05:26,560 And the little tj is this expectation. 80 00:05:26,560 --> 00:05:30,880 And here we're going to define a similar quantity, which is 81 00:05:30,880 --> 00:05:43,320 t1, let's say, star, defined as the first time to visit 82 00:05:43,320 --> 00:05:50,520 state 1 again. 83 00:05:50,520 --> 00:06:01,154 So that's the recurrence part starting from state 1, 84 00:06:01,154 --> 00:06:06,730 1 at t equals 0. 85 00:06:06,730 --> 00:06:10,000 So this is the recurrence time from state 1 86 00:06:10,000 --> 00:06:12,270 back to state 1 again. 87 00:06:12,270 --> 00:06:17,970 As an example, again, we look at t equals 0, 1, 2, 3, 4. 88 00:06:17,970 --> 00:06:23,550 And here, if we start from state 1 on time 0, we went to 89 00:06:23,550 --> 00:06:28,020 state 2, 2, 1, 1 again. 90 00:06:28,020 --> 00:06:32,910 Now here, again, time 3 will be the first time to visit 91 00:06:32,910 --> 00:06:37,200 state 1 after time 0. 92 00:06:37,200 --> 00:06:39,120 And we don't count the very first 0. 93 00:06:39,120 --> 00:06:43,120 And that will be our t1 star. 94 00:06:43,120 --> 00:06:47,150 So t1 star in this particular case is equal to 3. 95 00:06:47,150 --> 00:06:50,370 96 00:06:50,370 --> 00:06:51,620 OK. 97 00:06:51,620 --> 00:06:53,660 98 00:06:53,660 --> 00:06:58,480 Same as before, we like to calculate the expected time to 99 00:06:58,480 --> 00:07:00,840 revisit state 1. 100 00:07:00,840 --> 00:07:07,760 Define little t1 star expected value of t1 star. 101 00:07:07,760 --> 00:07:11,070 And we'll be using the same recurrence trick through the 102 00:07:11,070 --> 00:07:12,510 following equation. 103 00:07:12,510 --> 00:07:22,020 We say that t1 star is equal to 1 plus j from 1 to 2. 104 00:07:22,020 --> 00:07:25,750 Now, since we started from state 1, this goes from 1 to 105 00:07:25,750 --> 00:07:31,660 state 1j and tj. 106 00:07:31,660 --> 00:07:35,680 Again, the interpretation is we started at state 1 at time 107 00:07:35,680 --> 00:07:39,110 t equals 0, we went to some other state-- 108 00:07:39,110 --> 00:07:42,160 we call it j-- 109 00:07:42,160 --> 00:07:47,500 and front of state j, it goes around, and after time 110 00:07:47,500 --> 00:07:54,560 expected value tj, we came back to state 1. 111 00:07:54,560 --> 00:07:58,290 Here, and as before, this equation works because we are 112 00:07:58,290 --> 00:08:01,990 working with a Markov chain whereby the time to reach some 113 00:08:01,990 --> 00:08:04,090 other state only depends on the current state. 114 00:08:04,090 --> 00:08:06,680 And that's why we're able to break down the 115 00:08:06,680 --> 00:08:08,871 recursion as follows. 116 00:08:08,871 --> 00:08:12,840 If we write out the recursion, we get 1 plus 117 00:08:12,840 --> 00:08:18,267 P11 t1 plus P12 t2. 118 00:08:18,267 --> 00:08:20,830 119 00:08:20,830 --> 00:08:26,370 As before, t1 now is just the expected first passage time 120 00:08:26,370 --> 00:08:27,280 from state 1. 121 00:08:27,280 --> 00:08:28,990 And by definition, it is 0. 122 00:08:28,990 --> 00:08:33,970 Because if we start from state 1, it's already in state 1 and 123 00:08:33,970 --> 00:08:35,289 takes 0 time to get there. 124 00:08:35,289 --> 00:08:38,520 So again, like before, this term goes out. 125 00:08:38,520 --> 00:08:46,510 And we have 1 plus 0.2 times 5/3. 126 00:08:46,510 --> 00:08:50,760 And this number came from the previous calculation of t2. 127 00:08:50,760 --> 00:08:54,250 And this gives us 4/3. 128 00:08:54,250 --> 00:08:55,830 So this completes the problem. 129 00:08:55,830 --> 00:09:00,470 And just to remind ourselves, the kind of crux of the 130 00:09:00,470 --> 00:09:04,250 problem is this type of recursion which expresses a 131 00:09:04,250 --> 00:09:08,890 certain quantity in the one incremental step followed by 132 00:09:08,890 --> 00:09:12,750 the expected time to reach a certain destination 133 00:09:12,750 --> 00:09:14,310 after that one step. 134 00:09:14,310 --> 00:09:17,700 And we can do so because the dynamics is modeled by a 135 00:09:17,700 --> 00:09:18,750 Markov chain. 136 00:09:18,750 --> 00:09:21,530 And hence, the time to reach a certain destination after this 137 00:09:21,530 --> 00:09:24,730 first step only depends on where you start again, in this 138 00:09:24,730 --> 00:09:25,980 case, state j. 139 00:09:25,980 --> 00:09:36,200