1
00:00:00,000 --> 00:00:02,640
2
00:00:02,640 --> 00:00:05,700
In this problem, we are looking
at a student whose
3
00:00:05,700 --> 00:00:09,070
performance from day to day sort
of oscillates according
4
00:00:09,070 --> 00:00:10,610
to a Markov chain.
5
00:00:10,610 --> 00:00:15,360
In particular, the student can
either be in state 1, which is
6
00:00:15,360 --> 00:00:23,740
a state of being up to date,
or in state 2, which is a
7
00:00:23,740 --> 00:00:28,120
state of being kind
of fallen behind.
8
00:00:28,120 --> 00:00:30,370
Now, the transition
probabilities between these
9
00:00:30,370 --> 00:00:37,000
two states are given by the
numbers here, which is 0.2
10
00:00:37,000 --> 00:00:43,977
from state 1 to 2, 0.6 from 2
to 1, 0.4 from 2 back to 2,
11
00:00:43,977 --> 00:00:49,840
and 0.8 from 1 back
to state 1.
12
00:00:49,840 --> 00:00:53,140
The quantity we're interesting
calculating is this notion of
13
00:00:53,140 --> 00:00:54,810
first passage time.
14
00:00:54,810 --> 00:00:57,090
Let me define what that means.
15
00:00:57,090 --> 00:00:59,810
Suppose we are looking
at a time horizon of
16
00:00:59,810 --> 00:01:03,870
time 0, 1, 2, 3.
17
00:01:03,870 --> 00:01:07,970
And let's call the state of
the Markov chain x of t.
18
00:01:07,970 --> 00:01:13,000
Suppose we start from the chain
being in state 2 here.
19
00:01:13,000 --> 00:01:15,490
Now, if we look at a particular
sample path, let's
20
00:01:15,490 --> 00:01:23,010
say 2 and 2 again on day 1, and
2 again on day 2, and on
21
00:01:23,010 --> 00:01:27,330
day 3, the student
enters state 1.
22
00:01:27,330 --> 00:01:32,790
So in this sample path, we start
from time 0 and time 3
23
00:01:32,790 --> 00:01:36,010
is the first time we
enter state 1.
24
00:01:36,010 --> 00:01:39,120
And we'll say that the first
passage time, namely, the
25
00:01:39,120 --> 00:01:44,312
first time we enter state 1 in
this case, is equal to 3.
26
00:01:44,312 --> 00:01:53,010
More formally, we'll define tj
as the first pass the time to
27
00:01:53,010 --> 00:02:03,150
state 1 conditional on that we
start from state j at time 0.
28
00:02:03,150 --> 00:02:14,620
29
00:02:14,620 --> 00:02:16,900
Now, this quantity, of
course, is random.
30
00:02:16,900 --> 00:02:19,740
Depending on the realization,
we have different numbers.
31
00:02:19,740 --> 00:02:22,380
And we are interested
in calculating the
32
00:02:22,380 --> 00:02:26,650
expected value of t2.
33
00:02:26,650 --> 00:02:30,160
That is, on average, if we start
from state 2 here, how
34
00:02:30,160 --> 00:02:34,810
long would it take for
us to enter state 1?
35
00:02:34,810 --> 00:02:37,750
Now to calculate this quantity,
in the following
36
00:02:37,750 --> 00:02:39,910
recursion will be
very important.
37
00:02:39,910 --> 00:02:43,900
The idea is we don't know
exactly what t2 is.
38
00:02:43,900 --> 00:02:48,300
But t2 has to satisfy a certain
recurrent equation,
39
00:02:48,300 --> 00:02:57,070
namely, t2 must be equal
to 1 plus summation j
40
00:02:57,070 --> 00:03:00,226
equal to 1 to 2 P2jtj.
41
00:03:00,226 --> 00:03:04,815
42
00:03:04,815 --> 00:03:08,390
Now let me explain what
this equation means.
43
00:03:08,390 --> 00:03:12,840
Let's say we are at state 2.
44
00:03:12,840 --> 00:03:16,100
Well, we don't actually know
how long it's going to take
45
00:03:16,100 --> 00:03:17,690
for us to enter state 1.
46
00:03:17,690 --> 00:03:20,800
But we do know after one
step, I will be go
47
00:03:20,800 --> 00:03:22,530
into some other state.
48
00:03:22,530 --> 00:03:24,660
Let's call it state j.
49
00:03:24,660 --> 00:03:28,910
And from state j, it's going
to take some time to enter
50
00:03:28,910 --> 00:03:31,330
state 1 finally.
51
00:03:31,330 --> 00:03:34,710
So this equation essentially
says the time for us to first
52
00:03:34,710 --> 00:03:37,680
enter state 1 from 2 is 1--
53
00:03:37,680 --> 00:03:39,490
which is the next step--
54
00:03:39,490 --> 00:03:44,670
plus the expected time from
that point on to enter 1.
55
00:03:44,670 --> 00:03:47,100
So that constitutes our
[? recurrent ?]
56
00:03:47,100 --> 00:03:48,490
relationship.
57
00:03:48,490 --> 00:03:53,480
Now, by this definition, we can
see that this is simply 1
58
00:03:53,480 --> 00:04:03,030
plus P21 times t1 plus
P22 times t2.
59
00:04:03,030 --> 00:04:09,450
Now, the definition of tj says
t1 must be 0 because, by
60
00:04:09,450 --> 00:04:12,700
definition, if we start
from state 1, we are
61
00:04:12,700 --> 00:04:13,930
already in state 1.
62
00:04:13,930 --> 00:04:16,360
So the time to reach state
1 is simply 0.
63
00:04:16,360 --> 00:04:18,709
So this term disappears.
64
00:04:18,709 --> 00:04:24,740
And we end up with
1 plus P22 t2.
65
00:04:24,740 --> 00:04:29,150
If we plug in a number
of P22--
66
00:04:29,150 --> 00:04:32,030
which is 0.4 right here--
67
00:04:32,030 --> 00:04:36,840
we get 1 plus 0.4 t2.
68
00:04:36,840 --> 00:04:41,960
Now we started from t2 and
we ended up with another
69
00:04:41,960 --> 00:04:45,420
expression involving numbers
and only one
70
00:04:45,420 --> 00:04:47,570
unknown, which is t2.
71
00:04:47,570 --> 00:04:54,220
Combining this together and
solving for t2, we get t2
72
00:04:54,220 --> 00:05:03,400
equals 1 divided by 1 minus
0.4, which is 5/3.
73
00:05:03,400 --> 00:05:08,920
And that is the answer for the
first part of the problem.
74
00:05:08,920 --> 00:05:11,550
In the second part of the
problem, we are asked to do
75
00:05:11,550 --> 00:05:15,510
something similar as before
but with a slight twist.
76
00:05:15,510 --> 00:05:18,050
Here, I copied over the
definition for tj, which is
77
00:05:18,050 --> 00:05:22,460
the first time to visit state
1 starting from state j at
78
00:05:22,460 --> 00:05:23,960
time t equals 0.
79
00:05:23,960 --> 00:05:26,560
And the little tj is
this expectation.
80
00:05:26,560 --> 00:05:30,880
And here we're going to define
a similar quantity, which is
81
00:05:30,880 --> 00:05:43,320
t1, let's say, star, defined
as the first time to visit
82
00:05:43,320 --> 00:05:50,520
state 1 again.
83
00:05:50,520 --> 00:06:01,154
So that's the recurrence part
starting from state 1,
84
00:06:01,154 --> 00:06:06,730
1 at t equals 0.
85
00:06:06,730 --> 00:06:10,000
So this is the recurrence
time from state 1
86
00:06:10,000 --> 00:06:12,270
back to state 1 again.
87
00:06:12,270 --> 00:06:17,970
As an example, again, we look
at t equals 0, 1, 2, 3, 4.
88
00:06:17,970 --> 00:06:23,550
And here, if we start from state
1 on time 0, we went to
89
00:06:23,550 --> 00:06:28,020
state 2, 2, 1, 1 again.
90
00:06:28,020 --> 00:06:32,910
Now here, again, time 3 will
be the first time to visit
91
00:06:32,910 --> 00:06:37,200
state 1 after time 0.
92
00:06:37,200 --> 00:06:39,120
And we don't count
the very first 0.
93
00:06:39,120 --> 00:06:43,120
And that will be our t1 star.
94
00:06:43,120 --> 00:06:47,150
So t1 star in this particular
case is equal to 3.
95
00:06:47,150 --> 00:06:50,370
96
00:06:50,370 --> 00:06:51,620
OK.
97
00:06:51,620 --> 00:06:53,660
98
00:06:53,660 --> 00:06:58,480
Same as before, we like to
calculate the expected time to
99
00:06:58,480 --> 00:07:00,840
revisit state 1.
100
00:07:00,840 --> 00:07:07,760
Define little t1 star expected
value of t1 star.
101
00:07:07,760 --> 00:07:11,070
And we'll be using the same
recurrence trick through the
102
00:07:11,070 --> 00:07:12,510
following equation.
103
00:07:12,510 --> 00:07:22,020
We say that t1 star is equal
to 1 plus j from 1 to 2.
104
00:07:22,020 --> 00:07:25,750
Now, since we started from state
1, this goes from 1 to
105
00:07:25,750 --> 00:07:31,660
state 1j and tj.
106
00:07:31,660 --> 00:07:35,680
Again, the interpretation is we
started at state 1 at time
107
00:07:35,680 --> 00:07:39,110
t equals 0, we went to
some other state--
108
00:07:39,110 --> 00:07:42,160
we call it j--
109
00:07:42,160 --> 00:07:47,500
and front of state j, it goes
around, and after time
110
00:07:47,500 --> 00:07:54,560
expected value tj, we came
back to state 1.
111
00:07:54,560 --> 00:07:58,290
Here, and as before, this
equation works because we are
112
00:07:58,290 --> 00:08:01,990
working with a Markov chain
whereby the time to reach some
113
00:08:01,990 --> 00:08:04,090
other state only depends
on the current state.
114
00:08:04,090 --> 00:08:06,680
And that's why we're able
to break down the
115
00:08:06,680 --> 00:08:08,871
recursion as follows.
116
00:08:08,871 --> 00:08:12,840
If we write out the recursion,
we get 1 plus
117
00:08:12,840 --> 00:08:18,267
P11 t1 plus P12 t2.
118
00:08:18,267 --> 00:08:20,830
119
00:08:20,830 --> 00:08:26,370
As before, t1 now is just the
expected first passage time
120
00:08:26,370 --> 00:08:27,280
from state 1.
121
00:08:27,280 --> 00:08:28,990
And by definition, it is 0.
122
00:08:28,990 --> 00:08:33,970
Because if we start from state
1, it's already in state 1 and
123
00:08:33,970 --> 00:08:35,289
takes 0 time to get there.
124
00:08:35,289 --> 00:08:38,520
So again, like before,
this term goes out.
125
00:08:38,520 --> 00:08:46,510
And we have 1 plus
0.2 times 5/3.
126
00:08:46,510 --> 00:08:50,760
And this number came from the
previous calculation of t2.
127
00:08:50,760 --> 00:08:54,250
And this gives us 4/3.
128
00:08:54,250 --> 00:08:55,830
So this completes the problem.
129
00:08:55,830 --> 00:09:00,470
And just to remind ourselves,
the kind of crux of the
130
00:09:00,470 --> 00:09:04,250
problem is this type of
recursion which expresses a
131
00:09:04,250 --> 00:09:08,890
certain quantity in the one
incremental step followed by
132
00:09:08,890 --> 00:09:12,750
the expected time to reach
a certain destination
133
00:09:12,750 --> 00:09:14,310
after that one step.
134
00:09:14,310 --> 00:09:17,700
And we can do so because the
dynamics is modeled by a
135
00:09:17,700 --> 00:09:18,750
Markov chain.
136
00:09:18,750 --> 00:09:21,530
And hence, the time to reach a
certain destination after this
137
00:09:21,530 --> 00:09:24,730
first step only depends on where
you start again, in this
138
00:09:24,730 --> 00:09:25,980
case, state j.
139
00:09:25,980 --> 00:09:36,200