WEBVTT
00:00:00.980 --> 00:00:03.360
In this video, we are
going to calculate
00:00:03.360 --> 00:00:05.270
interesting quantities
that have to do
00:00:05.270 --> 00:00:08.700
with the short-term behavior
of Markov chains as opposed
00:00:08.700 --> 00:00:12.650
to those dealing with long-term
steady-state behaviors.
00:00:12.650 --> 00:00:16.690
But first, let us introduce
the notion of absorbing state.
00:00:16.690 --> 00:00:20.570
As indicated in this definition,
an absorbing state is
00:00:20.570 --> 00:00:25.190
a recurrent state from which
you cannot escape once you get
00:00:25.190 --> 00:00:25.980
to it.
00:00:25.980 --> 00:00:29.450
The transition probabilities
from k to k is 1.
00:00:29.450 --> 00:00:34.870
So in some sense, you get
absorbed by the state.
00:00:34.870 --> 00:00:38.560
For example, consider this
transition probability graph.
00:00:38.560 --> 00:00:42.960
States 4 and 5 are
both absorbing states.
00:00:42.960 --> 00:00:46.990
Indeed, when you get
to 4, you stay in 4.
00:00:46.990 --> 00:00:49.910
Or when you get to
5, you stay in 5.
00:00:49.910 --> 00:00:55.800
State 1, 2, and 3
are transient states.
00:00:55.800 --> 00:01:00.010
So if the Markov chain
initially started in 4, then
00:01:00.010 --> 00:01:01.900
it will stay in 4 forever.
00:01:01.900 --> 00:01:05.830
If it started in 5, it's
going to stay in 5 forever.
00:01:05.830 --> 00:01:10.770
But what if the Markov chain
started in either 1, 2, or 3?
00:01:10.770 --> 00:01:13.420
Eventually, after
some moving around,
00:01:13.420 --> 00:01:17.370
you will either make that
transition to state 4
00:01:17.370 --> 00:01:19.740
and get absorbed
by it, or you're
00:01:19.740 --> 00:01:22.990
going to do that
transition and get to 5
00:01:22.990 --> 00:01:25.870
and get absorbed by the state 5.
00:01:25.870 --> 00:01:29.460
So the question is, are
you going to end up in 4,
00:01:29.460 --> 00:01:31.800
or are you going to end up in 5?
00:01:31.800 --> 00:01:33.610
Well, we don't know for sure.
00:01:33.610 --> 00:01:36.090
These correspond
to random events.
00:01:36.090 --> 00:01:39.070
But can we say anything
about their probabilities?
00:01:39.070 --> 00:01:42.729
Well, let us try to calculate
the probability that you end up
00:01:42.729 --> 00:01:44.590
in 4 as an example.
00:01:44.590 --> 00:01:50.050
First, it seems plausible that
this probability of ending in 4
00:01:50.050 --> 00:01:51.950
will depend on
where you started.
00:01:51.950 --> 00:01:55.289
If you start in 2, you
probably have more chances
00:01:55.289 --> 00:01:57.850
of getting to 4 than
if you started from 3.
00:01:57.850 --> 00:02:00.770
Because if you start
from 2, at the next step
00:02:00.770 --> 00:02:04.250
you have immediately the
chance of getting to 4.
00:02:04.250 --> 00:02:06.890
But if you start from
3, there is some chance
00:02:06.890 --> 00:02:09.340
that you will go to
5 and never go to 4,
00:02:09.340 --> 00:02:14.310
or you will have to go through
2 in order to get to 4 anyway.
00:02:14.310 --> 00:02:17.010
So it looks like the
probability of reaching 4,
00:02:17.010 --> 00:02:19.040
given you started
from 2, will be bigger
00:02:19.040 --> 00:02:20.940
than if you started from 3.
00:02:20.940 --> 00:02:23.930
Now, from 1, it's unclear.
00:02:23.930 --> 00:02:25.960
Let us be systematic then.
00:02:25.960 --> 00:02:28.829
Let us consider all possible
probabilities to end up
00:02:28.829 --> 00:02:31.800
in 4 depending on
the initial state.
00:02:31.800 --> 00:02:34.290
So let us ask this
question, what
00:02:34.290 --> 00:02:37.829
is the probability, a of i,
that the chain eventually
00:02:37.829 --> 00:02:41.470
settles in 4 given
that it started in i?
00:02:41.470 --> 00:02:44.780
So in other words, a
of i is the probability
00:02:44.780 --> 00:02:49.170
that you end up in 4 given
that you started in i.
00:02:49.170 --> 00:02:53.630
Now, the answer to that question
is very easy for some cases.
00:02:53.630 --> 00:02:56.329
If you start in
4, the probability
00:02:56.329 --> 00:02:58.740
that you end up in 4 is 1.
00:02:58.740 --> 00:03:02.030
And if you start in
5, the probability
00:03:02.030 --> 00:03:04.040
that you end up in 4 is 0.
00:03:04.040 --> 00:03:07.600
There is no way that
you can go from 5 to 4.
00:03:07.600 --> 00:03:09.120
What about the other cases?
00:03:09.120 --> 00:03:12.230
Well, it's not so clear.
00:03:12.230 --> 00:03:15.600
Let us consider, for example,
that you started from 2.
00:03:15.600 --> 00:03:17.270
What could happen next?
00:03:17.270 --> 00:03:19.780
Well, from state 2,
let's build a tree.
00:03:19.780 --> 00:03:25.340
You can either, with a
probability 0.2, go to 4.
00:03:25.340 --> 00:03:31.210
Or with a probability
0.8, you will go to 1.
00:03:31.210 --> 00:03:33.380
Now, in the first
case, you're done.
00:03:33.380 --> 00:03:35.280
You reach 4.
00:03:35.280 --> 00:03:37.900
But in the second
case, you arrive in 1.
00:03:37.900 --> 00:03:39.450
And what happens next?
00:03:39.450 --> 00:03:40.590
You don't know.
00:03:40.590 --> 00:03:43.360
But what you know is
that from that state,
00:03:43.360 --> 00:03:48.070
either eventually you
go and get trapped in 5,
00:03:48.070 --> 00:03:52.150
or you go and eventually
get trapped in 4.
00:03:52.150 --> 00:03:54.740
What are the probabilities
of these events?
00:03:54.740 --> 00:03:56.560
Well, we don't know.
00:03:56.560 --> 00:03:59.390
But one of them has
been defined before.
00:03:59.390 --> 00:04:01.900
This represents the
probability of ending up
00:04:01.900 --> 00:04:06.420
in 4 given that you start in 1,
and that has been defined here.
00:04:06.420 --> 00:04:10.030
This is nothing else than a1.
00:04:10.030 --> 00:04:12.870
So the overall probability
of interest for us,
00:04:12.870 --> 00:04:17.070
which is a of 2, using the
total probability theorem,
00:04:17.070 --> 00:04:19.410
you can enumerate all options.
00:04:19.410 --> 00:04:22.940
It's with probability
0.2 you will go to 4.
00:04:22.940 --> 00:04:24.450
And then the
probability of going
00:04:24.450 --> 00:04:32.835
to 4 given that you started
in 4 is a4 plus 0.8 times a1.
00:04:32.835 --> 00:04:37.310
Now, a of 4 is, of course, 1,
as we have discussed before.
00:04:37.310 --> 00:04:42.390
So we get the relation
between a2 and a of 1.
00:04:42.390 --> 00:04:45.780
Now, of course, you can do the
same thing for the other state.
00:04:45.780 --> 00:04:51.000
For example, if you started
from 1, what can happen next?
00:04:51.000 --> 00:04:54.909
Well, you can go to 2
with a probability 0.6.
00:04:54.909 --> 00:04:57.190
Once you're in 2,
you're asking yourself,
00:04:57.190 --> 00:04:59.030
what is the probability
of reaching 4?
00:04:59.030 --> 00:05:02.400
Well, by definition, it's a2.
00:05:02.400 --> 00:05:07.940
Or from 1, you go to 3
with a probability 0.4.
00:05:07.940 --> 00:05:09.690
And given that you
have done that,
00:05:09.690 --> 00:05:12.140
what is the probability
that eventually you reach 4?
00:05:12.140 --> 00:05:15.140
It's a3.
00:05:15.140 --> 00:05:17.660
If initially you start with
a3, what can happen next?
00:05:17.660 --> 00:05:22.960
Again, with probability 0.3,
you will end up in state 2.
00:05:22.960 --> 00:05:26.740
And there, a of 2 is the
probability of interest.
00:05:26.740 --> 00:05:30.830
Or with a probability
0.5, you go to state 1.
00:05:30.830 --> 00:05:34.070
And in that case,
you get a of 1.
00:05:34.070 --> 00:05:38.505
And finally, with a probability
0.2, you get trapped in 5.
00:05:38.505 --> 00:05:39.090
All right?
00:05:39.090 --> 00:05:43.420
You can write if you want, but
0.2, and you get trapped in 5.
00:05:43.420 --> 00:05:48.860
But we know that a of 5 is 0,
so this term will disappear.
00:05:48.860 --> 00:05:51.930
So in the end, you
get a system here.
00:05:51.930 --> 00:05:56.690
After you replace
a4 by 1 and a5 by 0,
00:05:56.690 --> 00:06:02.020
you get a system of three linear
equations with three unknown.
00:06:02.020 --> 00:06:03.340
And it is easy to solve.
00:06:03.340 --> 00:06:04.310
I will not do that.
00:06:04.310 --> 00:06:05.760
You can do it yourself.
00:06:05.760 --> 00:06:08.550
But here are the results.
00:06:08.550 --> 00:06:17.950
You will get a of 1 equals
18/28, a of 2 will be 20/28,
00:06:17.950 --> 00:06:19.835
and a of 3 will be 15/28.
00:06:23.390 --> 00:06:26.780
Now I expressed them so that
we can compare them easily.
00:06:26.780 --> 00:06:29.210
And as a sanity
check, you can verify
00:06:29.210 --> 00:06:32.560
that indeed the
probability starting from 2
00:06:32.560 --> 00:06:35.500
will be larger than the
probability starting from 3.
00:06:35.500 --> 00:06:38.090
And it turns out
that a of 1 will
00:06:38.090 --> 00:06:40.100
be in between the other two.
00:06:40.100 --> 00:06:42.860
So these probabilities
are consistent
00:06:42.860 --> 00:06:45.990
with our previous intuitions.
00:06:45.990 --> 00:06:50.080
As an aside, note that you could
have written a system of five
00:06:50.080 --> 00:06:53.320
linear equations with
five unknown, the five
00:06:53.320 --> 00:06:56.590
unknown corresponding to
the five states possible.
00:06:56.590 --> 00:06:59.290
In fact, we had our
five equations here.
00:06:59.290 --> 00:07:05.530
Here was one, another one here,
and 1, 2, 3, so 3 plus 2, 5.
00:07:05.530 --> 00:07:07.030
Of course, this one was easy.
00:07:07.030 --> 00:07:12.470
It was a4 equals 1 and a5 equals
0 that you can replace then
00:07:12.470 --> 00:07:16.490
there, and you get a limited
or restricted or smaller system
00:07:16.490 --> 00:07:19.550
of three equations
with three unknown.
00:07:19.550 --> 00:07:23.180
Now, what if you had been
interested instead in finding
00:07:23.180 --> 00:07:27.330
the probability b of i
that the chain eventually
00:07:27.330 --> 00:07:31.250
settles in 5 given
you started in i.
00:07:31.250 --> 00:07:32.210
How to do that?
00:07:32.210 --> 00:07:35.540
Well, you can repeat
exactly all this calculation
00:07:35.540 --> 00:07:40.990
that we have done, but looking
at 5 as the state of interest.
00:07:40.990 --> 00:07:43.580
But of course, you
don't have to do this.
00:07:43.580 --> 00:07:47.950
For any state i, given
that you started in i,
00:07:47.950 --> 00:07:50.300
you will eventually
with probability 1
00:07:50.300 --> 00:07:52.860
end up in either 4 or 5.
00:07:52.860 --> 00:08:00.660
So you have a of i plus b of
i equals 1 for all possible i.
00:08:00.660 --> 00:08:05.930
So once you have calculated
a1, a2, a3, a4, and a5,
00:08:05.930 --> 00:08:13.020
you get b1, b2, b3, b4, and
b5 by using this formula.
00:08:13.020 --> 00:08:16.200
To sum up, in general,
the calculation
00:08:16.200 --> 00:08:21.560
of the probabilities to
reach a given absorbing state
00:08:21.560 --> 00:08:26.010
s starting from any state
i of a general Markov
00:08:26.010 --> 00:08:28.550
chain with m states--
so let's assume
00:08:28.550 --> 00:08:33.090
that you have m states-- will be
the unique solution of a system
00:08:33.090 --> 00:08:42.730
of m equations with m unknowns,
with the additional conditions
00:08:42.730 --> 00:08:50.000
that a of s equals
1 and a of s prime
00:08:50.000 --> 00:08:55.970
equals 0 for the other
absorbing states.
00:09:02.630 --> 00:09:04.730
Now, going back to
the following question
00:09:04.730 --> 00:09:06.930
that we posed at the
end of the review
00:09:06.930 --> 00:09:10.520
on steady-state behavior,
we had this diagram,
00:09:10.520 --> 00:09:15.020
and we wanted to know which
recurrent class this chain
00:09:15.020 --> 00:09:19.170
would end up if it started in
one of these transient states.
00:09:19.170 --> 00:09:21.650
Well, we can now
answer this question.
00:09:21.650 --> 00:09:23.830
For the purpose of
this calculation,
00:09:23.830 --> 00:09:28.000
the trick is simply to think
of a recurrent class as one
00:09:28.000 --> 00:09:31.420
big absorbing state and
go through the calculation
00:09:31.420 --> 00:09:33.320
as we have done here.
00:09:33.320 --> 00:09:37.200
So think about this class,
for example, as being one
00:09:37.200 --> 00:09:41.100
big state, an absorbing state.
00:09:41.100 --> 00:09:45.070
And now forget about the inside
and calculate the probability
00:09:45.070 --> 00:09:48.730
that you end up in this class
as the probability of reaching
00:09:48.730 --> 00:09:53.820
this absorbing state given that
you started in 1, 2, and 4,
00:09:53.820 --> 00:09:56.577
and you do the same
kind of calculation.