WEBVTT

00:00:02.640 --> 00:00:05.700
In this problem, we are looking
at a student whose

00:00:05.700 --> 00:00:09.070
performance from day to day sort
of oscillates according

00:00:09.070 --> 00:00:10.610
to a Markov chain.

00:00:10.610 --> 00:00:15.360
In particular, the student can
either be in state 1, which is

00:00:15.360 --> 00:00:23.740
a state of being up to date,
or in state 2, which is a

00:00:23.740 --> 00:00:28.120
state of being kind
of fallen behind.

00:00:28.120 --> 00:00:30.370
Now, the transition
probabilities between these

00:00:30.370 --> 00:00:37.000
two states are given by the
numbers here, which is 0.2

00:00:37.000 --> 00:00:43.977
from state 1 to 2, 0.6 from 2
to 1, 0.4 from 2 back to 2,

00:00:43.977 --> 00:00:49.840
and 0.8 from 1 back
to state 1.

00:00:49.840 --> 00:00:53.140
The quantity we're interesting
calculating is this notion of

00:00:53.140 --> 00:00:54.810
first passage time.

00:00:54.810 --> 00:00:57.090
Let me define what that means.

00:00:57.090 --> 00:00:59.810
Suppose we are looking
at a time horizon of

00:00:59.810 --> 00:01:03.870
time 0, 1, 2, 3.

00:01:03.870 --> 00:01:07.970
And let's call the state of
the Markov chain x of t.

00:01:07.970 --> 00:01:13.000
Suppose we start from the chain
being in state 2 here.

00:01:13.000 --> 00:01:15.490
Now, if we look at a particular
sample path, let's

00:01:15.490 --> 00:01:23.010
say 2 and 2 again on day 1, and
2 again on day 2, and on

00:01:23.010 --> 00:01:27.330
day 3, the student
enters state 1.

00:01:27.330 --> 00:01:32.790
So in this sample path, we start
from time 0 and time 3

00:01:32.790 --> 00:01:36.010
is the first time we
enter state 1.

00:01:36.010 --> 00:01:39.120
And we'll say that the first
passage time, namely, the

00:01:39.120 --> 00:01:44.312
first time we enter state 1 in
this case, is equal to 3.

00:01:44.312 --> 00:01:53.010
More formally, we'll define tj
as the first pass the time to

00:01:53.010 --> 00:02:03.150
state 1 conditional on that we
start from state j at time 0.

00:02:14.620 --> 00:02:16.900
Now, this quantity, of
course, is random.

00:02:16.900 --> 00:02:19.740
Depending on the realization,
we have different numbers.

00:02:19.740 --> 00:02:22.380
And we are interested
in calculating the

00:02:22.380 --> 00:02:26.650
expected value of t2.

00:02:26.650 --> 00:02:30.160
That is, on average, if we start
from state 2 here, how

00:02:30.160 --> 00:02:34.810
long would it take for
us to enter state 1?

00:02:34.810 --> 00:02:37.750
Now to calculate this quantity,
in the following

00:02:37.750 --> 00:02:39.910
recursion will be
very important.

00:02:39.910 --> 00:02:43.900
The idea is we don't know
exactly what t2 is.

00:02:43.900 --> 00:02:48.300
But t2 has to satisfy a certain
recurrent equation,

00:02:48.300 --> 00:02:57.070
namely, t2 must be equal
to 1 plus summation j

00:02:57.070 --> 00:03:00.226
equal to 1 to 2 P2jtj.

00:03:04.815 --> 00:03:08.390
Now let me explain what
this equation means.

00:03:08.390 --> 00:03:12.840
Let's say we are at state 2.

00:03:12.840 --> 00:03:16.100
Well, we don't actually know
how long it's going to take

00:03:16.100 --> 00:03:17.690
for us to enter state 1.

00:03:17.690 --> 00:03:20.800
But we do know after one
step, I will be go

00:03:20.800 --> 00:03:22.530
into some other state.

00:03:22.530 --> 00:03:24.660
Let's call it state j.

00:03:24.660 --> 00:03:28.910
And from state j, it's going
to take some time to enter

00:03:28.910 --> 00:03:31.330
state 1 finally.

00:03:31.330 --> 00:03:34.710
So this equation essentially
says the time for us to first

00:03:34.710 --> 00:03:37.680
enter state 1 from 2 is 1--

00:03:37.680 --> 00:03:39.490
which is the next step--

00:03:39.490 --> 00:03:44.670
plus the expected time from
that point on to enter 1.

00:03:44.670 --> 00:03:47.100
So that constitutes our
[? recurrent ?]

00:03:47.100 --> 00:03:48.490
relationship.

00:03:48.490 --> 00:03:53.480
Now, by this definition, we can
see that this is simply 1

00:03:53.480 --> 00:04:03.030
plus P21 times t1 plus
P22 times t2.

00:04:03.030 --> 00:04:09.450
Now, the definition of tj says
t1 must be 0 because, by

00:04:09.450 --> 00:04:12.700
definition, if we start
from state 1, we are

00:04:12.700 --> 00:04:13.930
already in state 1.

00:04:13.930 --> 00:04:16.360
So the time to reach state
1 is simply 0.

00:04:16.360 --> 00:04:18.709
So this term disappears.

00:04:18.709 --> 00:04:24.740
And we end up with
1 plus P22 t2.

00:04:24.740 --> 00:04:29.150
If we plug in a number
of P22--

00:04:29.150 --> 00:04:32.030
which is 0.4 right here--

00:04:32.030 --> 00:04:36.840
we get 1 plus 0.4 t2.

00:04:36.840 --> 00:04:41.960
Now we started from t2 and
we ended up with another

00:04:41.960 --> 00:04:45.420
expression involving numbers
and only one

00:04:45.420 --> 00:04:47.570
unknown, which is t2.

00:04:47.570 --> 00:04:54.220
Combining this together and
solving for t2, we get t2

00:04:54.220 --> 00:05:03.400
equals 1 divided by 1 minus
0.4, which is 5/3.

00:05:03.400 --> 00:05:08.920
And that is the answer for the
first part of the problem.

00:05:08.920 --> 00:05:11.550
In the second part of the
problem, we are asked to do

00:05:11.550 --> 00:05:15.510
something similar as before
but with a slight twist.

00:05:15.510 --> 00:05:18.050
Here, I copied over the
definition for tj, which is

00:05:18.050 --> 00:05:22.460
the first time to visit state
1 starting from state j at

00:05:22.460 --> 00:05:23.960
time t equals 0.

00:05:23.960 --> 00:05:26.560
And the little tj is
this expectation.

00:05:26.560 --> 00:05:30.880
And here we're going to define
a similar quantity, which is

00:05:30.880 --> 00:05:43.320
t1, let's say, star, defined
as the first time to visit

00:05:43.320 --> 00:05:50.520
state 1 again.

00:05:50.520 --> 00:06:01.154
So that's the recurrence part
starting from state 1,

00:06:01.154 --> 00:06:06.730
1 at t equals 0.

00:06:06.730 --> 00:06:10.000
So this is the recurrence
time from state 1

00:06:10.000 --> 00:06:12.270
back to state 1 again.

00:06:12.270 --> 00:06:17.970
As an example, again, we look
at t equals 0, 1, 2, 3, 4.

00:06:17.970 --> 00:06:23.550
And here, if we start from state
1 on time 0, we went to

00:06:23.550 --> 00:06:28.020
state 2, 2, 1, 1 again.

00:06:28.020 --> 00:06:32.910
Now here, again, time 3 will
be the first time to visit

00:06:32.910 --> 00:06:37.200
state 1 after time 0.

00:06:37.200 --> 00:06:39.120
And we don't count
the very first 0.

00:06:39.120 --> 00:06:43.120
And that will be our t1 star.

00:06:43.120 --> 00:06:47.150
So t1 star in this particular
case is equal to 3.

00:06:50.370 --> 00:06:51.620
OK.

00:06:53.660 --> 00:06:58.480
Same as before, we like to
calculate the expected time to

00:06:58.480 --> 00:07:00.840
revisit state 1.

00:07:00.840 --> 00:07:07.760
Define little t1 star expected
value of t1 star.

00:07:07.760 --> 00:07:11.070
And we'll be using the same
recurrence trick through the

00:07:11.070 --> 00:07:12.510
following equation.

00:07:12.510 --> 00:07:22.020
We say that t1 star is equal
to 1 plus j from 1 to 2.

00:07:22.020 --> 00:07:25.750
Now, since we started from state
1, this goes from 1 to

00:07:25.750 --> 00:07:31.660
state 1j and tj.

00:07:31.660 --> 00:07:35.680
Again, the interpretation is we
started at state 1 at time

00:07:35.680 --> 00:07:39.110
t equals 0, we went to
some other state--

00:07:39.110 --> 00:07:42.160
we call it j--

00:07:42.160 --> 00:07:47.500
and front of state j, it goes
around, and after time

00:07:47.500 --> 00:07:54.560
expected value tj, we came
back to state 1.

00:07:54.560 --> 00:07:58.290
Here, and as before, this
equation works because we are

00:07:58.290 --> 00:08:01.990
working with a Markov chain
whereby the time to reach some

00:08:01.990 --> 00:08:04.090
other state only depends
on the current state.

00:08:04.090 --> 00:08:06.680
And that's why we're able
to break down the

00:08:06.680 --> 00:08:08.871
recursion as follows.

00:08:08.871 --> 00:08:12.840
If we write out the recursion,
we get 1 plus

00:08:12.840 --> 00:08:18.267
P11 t1 plus P12 t2.

00:08:20.830 --> 00:08:26.370
As before, t1 now is just the
expected first passage time

00:08:26.370 --> 00:08:27.280
from state 1.

00:08:27.280 --> 00:08:28.990
And by definition, it is 0.

00:08:28.990 --> 00:08:33.970
Because if we start from state
1, it's already in state 1 and

00:08:33.970 --> 00:08:35.289
takes 0 time to get there.

00:08:35.289 --> 00:08:38.520
So again, like before,
this term goes out.

00:08:38.520 --> 00:08:46.510
And we have 1 plus
0.2 times 5/3.

00:08:46.510 --> 00:08:50.760
And this number came from the
previous calculation of t2.

00:08:50.760 --> 00:08:54.250
And this gives us 4/3.

00:08:54.250 --> 00:08:55.830
So this completes the problem.

00:08:55.830 --> 00:09:00.470
And just to remind ourselves,
the kind of crux of the

00:09:00.470 --> 00:09:04.250
problem is this type of
recursion which expresses a

00:09:04.250 --> 00:09:08.890
certain quantity in the one
incremental step followed by

00:09:08.890 --> 00:09:12.750
the expected time to reach
a certain destination

00:09:12.750 --> 00:09:14.310
after that one step.

00:09:14.310 --> 00:09:17.700
And we can do so because the
dynamics is modeled by a

00:09:17.700 --> 00:09:18.750
Markov chain.

00:09:18.750 --> 00:09:21.530
And hence, the time to reach a
certain destination after this

00:09:21.530 --> 00:09:24.730
first step only depends on where
you start again, in this

00:09:24.730 --> 00:09:25.980
case, state j.