WEBVTT
00:00:00.040 --> 00:00:02.460
The following content is
provided under a Creative
00:00:02.460 --> 00:00:03.870
Commons license.
00:00:03.870 --> 00:00:06.910
Your support will help MIT
OpenCourseWare continue to
00:00:06.910 --> 00:00:10.560
offer high quality educational
resources for free.
00:00:10.560 --> 00:00:13.460
To make a donation or view
additional materials from
00:00:13.460 --> 00:00:17.390
hundreds of MIT courses, visit
MIT OpenCourseWare at
00:00:17.390 --> 00:00:18.640
ocw.mit.edu.
00:00:23.440 --> 00:00:25.930
JOHN TSITSIKLIS: So what we're
going to do is to review what
00:00:25.930 --> 00:00:29.390
we have discussed last time.
00:00:29.390 --> 00:00:32.380
Then we're going to talk about
the classic application of
00:00:32.380 --> 00:00:35.950
Markov chains to analyze
how do you
00:00:35.950 --> 00:00:39.130
dimension a phone system.
00:00:39.130 --> 00:00:42.400
And finally, there will be
two new things today.
00:00:42.400 --> 00:00:44.940
We will see how we can calculate
certain interesting
00:00:44.940 --> 00:00:48.950
quantities that have to
do with Markov chains.
00:00:48.950 --> 00:00:50.200
So let us start.
00:00:52.860 --> 00:00:56.680
We've got our Markov chain and
let's make the assumption that
00:00:56.680 --> 00:00:58.800
our chain is kind of nice.
00:00:58.800 --> 00:01:01.550
And by nice we mean that
we've got maybe
00:01:01.550 --> 00:01:02.800
some transient states.
00:01:06.380 --> 00:01:11.580
And then we've got a single
recurrent class
00:01:11.580 --> 00:01:15.450
of recurrent states.
00:01:15.450 --> 00:01:18.600
So this is a single recurrent
class in the sense that from
00:01:18.600 --> 00:01:22.080
any state in that class you can
get to any other state.
00:01:22.080 --> 00:01:24.790
So once you're in here you're
going to circulate and keep
00:01:24.790 --> 00:01:26.650
visiting all of those states.
00:01:26.650 --> 00:01:28.340
Those states appear transient.
00:01:28.340 --> 00:01:32.020
The trajectory may move around
here, but eventually one of
00:01:32.020 --> 00:01:34.320
these transitions will happen
and you're going to
00:01:34.320 --> 00:01:36.040
end up in this lump.
00:01:38.640 --> 00:01:42.680
Let's make the assumption that
the single recurrent class is
00:01:42.680 --> 00:01:43.930
not periodic.
00:01:48.170 --> 00:01:51.740
These are the nicest kind
of Markov chains.
00:01:51.740 --> 00:01:54.500
And they're nicest because
they have the following
00:01:54.500 --> 00:01:58.900
property, the probability that
you find yourself at some
00:01:58.900 --> 00:02:03.100
particular state j at
the time n when that
00:02:03.100 --> 00:02:04.800
time is very large.
00:02:04.800 --> 00:02:08.130
That probability settles to a
steady state value that we
00:02:08.130 --> 00:02:10.139
denote by pi sub j.
00:02:10.139 --> 00:02:12.630
And there are two parts
in the statement.
00:02:12.630 --> 00:02:15.470
One part is that this
limit exists.
00:02:15.470 --> 00:02:19.010
So the probability of state j
settles to something, and
00:02:19.010 --> 00:02:23.080
furthermore that probability
is not affected by i.
00:02:23.080 --> 00:02:25.550
It doesn't matter where you
started, no matter where you
00:02:25.550 --> 00:02:28.310
started, the probability of
state j is going to be the
00:02:28.310 --> 00:02:30.720
same in the long run.
00:02:30.720 --> 00:02:34.990
Maybe a clearer notation
could be of this form.
00:02:34.990 --> 00:02:37.650
The probability of being at
state j given the initial
00:02:37.650 --> 00:02:42.570
state being i is equal to
pi(j) in the limit.
00:02:42.570 --> 00:02:47.380
Now, if I don't tell you where
you started and you look at
00:02:47.380 --> 00:02:52.520
the unconditional probability
of being at state i, you can
00:02:52.520 --> 00:02:56.260
average over the initial
states, use the total
00:02:56.260 --> 00:03:01.560
expectation theorem and you're
going to get the same answer
00:03:01.560 --> 00:03:05.330
pi(j) in the limit.
00:03:05.330 --> 00:03:09.400
So this tells you that to the
conditional probability given
00:03:09.400 --> 00:03:11.820
the initial state in the limit
is the same as the
00:03:11.820 --> 00:03:13.780
unconditional probability.
00:03:13.780 --> 00:03:17.200
And that's a situation that we
recognize as being one where
00:03:17.200 --> 00:03:18.890
we have independence.
00:03:18.890 --> 00:03:24.970
So what this result tells
us is that Xn and Xi are
00:03:24.970 --> 00:03:27.410
approximately independent.
00:03:27.410 --> 00:03:32.690
They become independent in the
limit as n goes to infinity.
00:03:32.690 --> 00:03:35.130
So that's what the steady
state theorem tells us.
00:03:35.130 --> 00:03:38.860
The initial conditions don't
matter, so your state at some
00:03:38.860 --> 00:03:43.440
large time n has nothing to do,
is not affected by what
00:03:43.440 --> 00:03:45.350
your initial state was.
00:03:45.350 --> 00:03:47.790
Knowing the initial state
doesn't tell you anything
00:03:47.790 --> 00:03:50.830
about your state at time
n, therefore the
00:03:50.830 --> 00:03:52.890
states at the times--
00:03:52.890 --> 00:03:56.580
sorry that should be a 1,
or it should be a 0 --
00:03:56.580 --> 00:04:02.350
so the state is not affected by
where the process started.
00:04:02.350 --> 00:04:05.450
So if the Markov chain is to
operate for a long time and
00:04:05.450 --> 00:04:08.520
we're interested in the question
where is the state,
00:04:08.520 --> 00:04:11.470
then your answer would be, I
don't know, it's random.
00:04:11.470 --> 00:04:14.020
But it's going to be a
particular j with this
00:04:14.020 --> 00:04:15.500
particular probability.
00:04:15.500 --> 00:04:18.550
So the steady state
probabilities are interesting
00:04:18.550 --> 00:04:21.060
to us and that raises
the question of how
00:04:21.060 --> 00:04:22.470
do we compute them.
00:04:22.470 --> 00:04:25.460
The way we compute them is by
solving a linear system of
00:04:25.460 --> 00:04:28.580
equations, which are called
the balance equations,
00:04:28.580 --> 00:04:31.510
together with an extra equation,
the normalization
00:04:31.510 --> 00:04:33.690
equation that has to be
satisfied by probability,
00:04:33.690 --> 00:04:37.940
because probabilities must
always add up to 1.
00:04:37.940 --> 00:04:40.660
We talked about the
interpretation of this
00:04:40.660 --> 00:04:42.280
equation last time.
00:04:42.280 --> 00:04:45.500
It's basically a conservation
of probability
00:04:45.500 --> 00:04:47.400
flow in some sense.
00:04:47.400 --> 00:04:50.140
What comes in must get out.
00:04:50.140 --> 00:04:53.190
The probability of finding
yourself at state j at a
00:04:53.190 --> 00:04:58.150
particular time is the total
probability of the last
00:04:58.150 --> 00:05:01.390
transition taking
me into state j.
00:05:01.390 --> 00:05:04.860
The last transition takes me
into state j in various ways.
00:05:04.860 --> 00:05:07.890
It could be that the previous
time I was at the particular
00:05:07.890 --> 00:05:12.550
state, j and i made a transition
from k into j.
00:05:12.550 --> 00:05:15.610
So this number here, we
interpret as the frequency
00:05:15.610 --> 00:05:18.190
with which transitions
of these particular
00:05:18.190 --> 00:05:21.900
type k to j, occur.
00:05:21.900 --> 00:05:25.840
And then by adding over all k's
we consider transitions of
00:05:25.840 --> 00:05:29.770
all types that lead
us inside state j.
00:05:29.770 --> 00:05:34.240
So the probability of being at
the j is the sum total of the
00:05:34.240 --> 00:05:38.690
probabilities of
getting into j.
00:05:38.690 --> 00:05:42.800
What if we had multiple
recurrent classes?
00:05:42.800 --> 00:05:48.645
So if we take this picture
and change it to this.
00:05:53.420 --> 00:05:55.920
So here we got a secondary
recurrent class.
00:05:55.920 --> 00:05:57.750
If you're here, you
cannot get there.
00:05:57.750 --> 00:06:00.050
If you are here, you
cannot get there.
00:06:00.050 --> 00:06:02.370
What happens in the long run?
00:06:02.370 --> 00:06:05.990
Well, in the long run, if you
start from here you're going
00:06:05.990 --> 00:06:09.360
to make a transition eventually,
either of this
00:06:09.360 --> 00:06:12.540
type and you would end up
here, or you will make a
00:06:12.540 --> 00:06:16.210
transitional of that type and
you will end up there.
00:06:16.210 --> 00:06:21.740
If you end up here, the long
term statistics of your chain,
00:06:21.740 --> 00:06:24.860
that is, the probabilities of
the different states, will be
00:06:24.860 --> 00:06:27.080
the steady state probabilities
of this
00:06:27.080 --> 00:06:30.230
chain regarded in isolation.
00:06:30.230 --> 00:06:34.440
So you go ahead and you solve
this system of equations just
00:06:34.440 --> 00:06:37.270
for this chain, and these will
be your steady state
00:06:37.270 --> 00:06:38.560
probabilities.
00:06:38.560 --> 00:06:41.660
If you happened to
get in here.
00:06:41.660 --> 00:06:45.650
If, on the other hand, it
happens that you went there,
00:06:45.650 --> 00:06:49.710
given that event, then what
happens in the long run has to
00:06:49.710 --> 00:06:53.390
do with just this chain
running by itself.
00:06:53.390 --> 00:06:55.610
So you find the steady
state probabilities
00:06:55.610 --> 00:06:57.340
inside that sub chain.
00:06:57.340 --> 00:06:59.860
So you solve the linear system,
the steady state
00:06:59.860 --> 00:07:03.520
equations, for this chain
separately and for that chain
00:07:03.520 --> 00:07:04.590
separately.
00:07:04.590 --> 00:07:08.160
If you happen to start inside
here then the steady state
00:07:08.160 --> 00:07:12.650
probabilities for this sub
chain are going to apply.
00:07:12.650 --> 00:07:16.420
Now of course this raises the
question, if I start here, how
00:07:16.420 --> 00:07:19.500
do I know whether I'm going
to get here or there?
00:07:19.500 --> 00:07:21.360
Well, you don't know,
it's random.
00:07:21.360 --> 00:07:23.620
It may turn out that you get to
here, it may turn out that
00:07:23.620 --> 00:07:24.760
you get there.
00:07:24.760 --> 00:07:27.790
So we will be interested in
calculating the probabilities
00:07:27.790 --> 00:07:31.240
that eventually you end up here
versus the probability
00:07:31.240 --> 00:07:33.570
that eventually you
end up there.
00:07:33.570 --> 00:07:36.590
This is something that we're
going to do towards the end of
00:07:36.590 --> 00:07:37.840
today's lecture.
00:07:43.730 --> 00:07:48.420
So, as a warm up, just to see
how we interpret those steady
00:07:48.420 --> 00:07:52.710
state probabilities, let us look
at our familiar example.
00:07:52.710 --> 00:07:54.680
This is a 2-state
Markov chain.
00:07:54.680 --> 00:07:57.860
Last time we did write down the
balance equations for this
00:07:57.860 --> 00:08:01.760
chain and we found the steady
state probabilities to be 2/7
00:08:01.760 --> 00:08:04.250
and 5/7 respectively.
00:08:04.250 --> 00:08:07.040
So let us try to calculate
some quantities.
00:08:07.040 --> 00:08:10.520
Suppose that you start at
state 1, and you want to
00:08:10.520 --> 00:08:12.940
calculate to this particular
probability.
00:08:12.940 --> 00:08:15.620
So since we're assuming that
we're starting at state 1,
00:08:15.620 --> 00:08:19.010
essentially here we are
conditioning on the initial
00:08:19.010 --> 00:08:21.790
state being equal to 1.
00:08:21.790 --> 00:08:23.920
Now the conditional probability
of two things
00:08:23.920 --> 00:08:30.370
happening is the probability
that the first thing happens.
00:08:30.370 --> 00:08:34.090
But we're living in the world
where we said that the initial
00:08:34.090 --> 00:08:35.720
state was 1.
00:08:35.720 --> 00:08:40.000
And then given that this thing
happened, the probability that
00:08:40.000 --> 00:08:43.530
the second thing happens.
00:08:43.530 --> 00:08:46.200
But again, we're talking about
conditional probabilities
00:08:46.200 --> 00:08:50.150
given that the initial
state was 1.
00:08:50.150 --> 00:08:53.080
So what is this quantity?
00:08:53.080 --> 00:08:57.040
This one is the transition
probability from state 1 to
00:08:57.040 --> 00:09:00.800
state 1, so it's P11.
00:09:00.800 --> 00:09:04.730
How about the second
probability?
00:09:04.730 --> 00:09:08.130
So given that you started at 1
and the next time you were at
00:09:08.130 --> 00:09:11.980
1, what's the probability that
at the time 100 you are at 1?
00:09:11.980 --> 00:09:15.160
Now because of the Markov
property, if I tell you that
00:09:15.160 --> 00:09:17.520
at this time you are
at 1, it doesn't
00:09:17.520 --> 00:09:19.190
matter how you get there.
00:09:19.190 --> 00:09:22.560
So this part of the conditioning
doesn't matter.
00:09:22.560 --> 00:09:29.650
And what we have is the 99 step
transition probability
00:09:29.650 --> 00:09:34.820
from state 1 to state 1.
00:09:34.820 --> 00:09:38.910
So the probability that you
get to 1 and then 99 steps
00:09:38.910 --> 00:09:42.340
later you find yourself again at
one is the probability that
00:09:42.340 --> 00:09:45.350
the first transition takes you
to 1 times the probability
00:09:45.350 --> 00:09:51.810
that over the next 99
transitions starting from 1,
00:09:51.810 --> 00:09:55.930
after 99 steps you end
up again at state 1.
00:09:55.930 --> 00:10:00.720
Now, 99 is possibly a big
number, and so we approximate
00:10:00.720 --> 00:10:01.730
this quantity.
00:10:01.730 --> 00:10:05.810
We're using the steady state
probability of state 1.
00:10:05.810 --> 00:10:08.110
And that gives us an
approximation for this
00:10:08.110 --> 00:10:10.510
particular expression.
00:10:10.510 --> 00:10:14.790
We can do the same thing
to calculate
00:10:14.790 --> 00:10:16.490
something of the same kind.
00:10:16.490 --> 00:10:19.260
So you start at state 1.
00:10:19.260 --> 00:10:23.230
What's the probability that 100
steps later you are again
00:10:23.230 --> 00:10:24.560
at state 1?
00:10:24.560 --> 00:10:26.000
So that's going to be P11--
00:10:28.970 --> 00:10:29.890
not P --
00:10:29.890 --> 00:10:31.140
R11.
00:10:34.830 --> 00:10:38.560
The 100 step transition
probability that starting from
00:10:38.560 --> 00:10:43.980
1 you get to 1, and then after
you get to 1 at time 100
00:10:43.980 --> 00:10:46.120
what's the probability that
the next time you find
00:10:46.120 --> 00:10:47.870
yourself at state 2?
00:10:47.870 --> 00:10:50.580
This is going to be the
probability P12.
00:10:50.580 --> 00:10:55.320
And approximately, since 100
is a large number, this is
00:10:55.320 --> 00:10:57.890
approximately pi(1) times P12.
00:11:06.230 --> 00:11:07.070
OK.
00:11:07.070 --> 00:11:12.180
So that's how we can use steady
state probabilities to
00:11:12.180 --> 00:11:14.190
make approximations.
00:11:14.190 --> 00:11:19.090
Or you could, for example, if
you continue doing examples of
00:11:19.090 --> 00:11:23.250
this kind, you could ask for
what's the probability that X
00:11:23.250 --> 00:11:32.170
at time 100 is 1, and also X
at time 200 is equal to 1.
00:11:32.170 --> 00:11:36.200
Then this is going to be the
transition probability from 1
00:11:36.200 --> 00:11:43.140
to 1 in 100 steps, and then over
the next 100 steps from 1
00:11:43.140 --> 00:11:46.510
you get again to 1.
00:11:46.510 --> 00:11:48.750
And this is going to be
00:11:48.750 --> 00:11:51.400
approximately pi(1) times pi(1).
00:11:57.550 --> 00:12:01.090
So we approximate multi-step
transition probabilities by
00:12:01.090 --> 00:12:05.120
the steady state probabilities
when the number n that's
00:12:05.120 --> 00:12:07.510
involved in here is big.
00:12:07.510 --> 00:12:11.270
Now I said that's 99
or 100 is big.
00:12:11.270 --> 00:12:15.180
How do we know that it's big
enough so that the limit has
00:12:15.180 --> 00:12:19.560
taken effect, and that our
approximation is good?
00:12:19.560 --> 00:12:23.640
This has something to do with
the time scale of our Markov
00:12:23.640 --> 00:12:27.460
chain, and by time scale, I mean
how long does it take for
00:12:27.460 --> 00:12:29.740
the initial states
to be forgotten.
00:12:29.740 --> 00:12:35.080
How long does it take for there
to be enough randomness
00:12:35.080 --> 00:12:37.700
so that things sort of
mix and it doesn't
00:12:37.700 --> 00:12:39.330
matter where you started?
00:12:39.330 --> 00:12:43.720
So if you look at this chain, it
takes on the average, let's
00:12:43.720 --> 00:12:47.660
say 5 tries to make a transition
of this kind.
00:12:47.660 --> 00:12:51.970
It takes on the average 2 tries
for a transition of that
00:12:51.970 --> 00:12:53.990
kind to take place.
00:12:53.990 --> 00:12:58.840
So every 10 time steps or so
there's a little bit of
00:12:58.840 --> 00:12:59.660
randomness.
00:12:59.660 --> 00:13:03.000
Over 100 times steps there's
a lot of randomness, so you
00:13:03.000 --> 00:13:06.420
expect that the initial state
will have been forgotten.
00:13:06.420 --> 00:13:07.820
It doesn't matter.
00:13:07.820 --> 00:13:10.540
There's enough mixing and
randomness that happens over
00:13:10.540 --> 00:13:11.870
100 time steps.
00:13:11.870 --> 00:13:16.100
And so this approximation
is good.
00:13:16.100 --> 00:13:19.080
On the other hand, if the
numbers were different, the
00:13:19.080 --> 00:13:20.880
story would have
been different.
00:13:20.880 --> 00:13:26.620
Suppose that this number is
0.999 and that number is
00:13:26.620 --> 00:13:33.570
something like 0.998, so that
this number becomes 0.002, and
00:13:33.570 --> 00:13:37.910
that number becomes 0.001.
00:13:37.910 --> 00:13:40.620
Suppose that the numbers
were of this kind.
00:13:40.620 --> 00:13:44.760
How long does it take to forget
the initial state?
00:13:44.760 --> 00:13:48.840
If I start here, there's a
probability of 1 in 1,000 that
00:13:48.840 --> 00:13:50.770
next time I'm going
to be there.
00:13:50.770 --> 00:13:53.840
So on the average it's going
to take me about a thousand
00:13:53.840 --> 00:13:58.330
tries just to leave
that state.
00:13:58.330 --> 00:14:03.390
So, over roughly a thousand time
steps my initial state
00:14:03.390 --> 00:14:05.210
really does matter.
00:14:05.210 --> 00:14:08.550
If I tell you that you started
here, you're pretty certain
00:14:08.550 --> 00:14:11.320
that, let's say over the next
100 time steps, you
00:14:11.320 --> 00:14:12.710
will still be here.
00:14:12.710 --> 00:14:15.580
So the initial state
has a big effect.
00:14:15.580 --> 00:14:19.600
In this case we say that this
Markov chain has a much slower
00:14:19.600 --> 00:14:21.030
time scale.
00:14:21.030 --> 00:14:25.350
It takes a much longer time to
mix, it takes a much longer
00:14:25.350 --> 00:14:29.150
time for the initial state to
be forgotten, and this means
00:14:29.150 --> 00:14:32.670
that we cannot do this kind of
approximation if the number of
00:14:32.670 --> 00:14:34.870
steps is just 99.
00:14:34.870 --> 00:14:39.620
Here we might need n to be as
large as, let's say, 10,000 or
00:14:39.620 --> 00:14:43.320
so before we can start using
the approximation.
00:14:43.320 --> 00:14:46.210
So when one uses that
approximation, one needs to
00:14:46.210 --> 00:14:50.830
have some sense of how quickly
does the state move around and
00:14:50.830 --> 00:14:52.210
take that into account.
00:14:52.210 --> 00:14:56.100
So there's a whole sub-field
that deals with estimating or
00:14:56.100 --> 00:15:00.030
figuring out how quickly
different Markov chains mix,
00:15:00.030 --> 00:15:02.460
and that's the question of
when can you apply those
00:15:02.460 --> 00:15:03.710
steady state approximations.
00:15:06.860 --> 00:15:12.180
So now let's get a little closer
to the real world.
00:15:12.180 --> 00:15:15.590
We're going to talk about a
famous problem that was posed,
00:15:15.590 --> 00:15:19.220
started, and solved by
a Danish engineer
00:15:19.220 --> 00:15:21.110
by the name of Erlang.
00:15:21.110 --> 00:15:25.180
This is the same person whose
name is given to the Erlang
00:15:25.180 --> 00:15:27.080
distribution that we
saw in the context
00:15:27.080 --> 00:15:28.940
of the Poisson processes.
00:15:28.940 --> 00:15:33.950
So this was more than 100 years
ago, when phones had
00:15:33.950 --> 00:15:36.060
just started existing.
00:15:36.060 --> 00:15:42.900
And he was trying to figure out
what it would take to set
00:15:42.900 --> 00:15:47.260
up a phone system that how many
lines should you set up
00:15:47.260 --> 00:15:49.920
for a community to be
able to communicate
00:15:49.920 --> 00:15:53.020
to the outside world.
00:15:53.020 --> 00:15:54.350
So here's the story.
00:15:54.350 --> 00:15:57.730
You've got a village, and that
village has a certain
00:15:57.730 --> 00:16:03.705
population, and you want
to set up phone lines.
00:16:06.290 --> 00:16:09.720
So you want to set up a number
of phone lines, let's say that
00:16:09.720 --> 00:16:13.520
number is B, to the
outside world.
00:16:17.710 --> 00:16:19.690
And how do you want
to do that?
00:16:19.690 --> 00:16:22.000
Well, you want B to
be kind of small.
00:16:22.000 --> 00:16:24.130
You don't want to set
up too many wires
00:16:24.130 --> 00:16:25.680
because that's expensive.
00:16:25.680 --> 00:16:30.300
On the other hand, you want to
have enough wires so that if a
00:16:30.300 --> 00:16:33.000
reasonable number of people
place phone calls
00:16:33.000 --> 00:16:38.000
simultaneously, they will all
get a line and they will be
00:16:38.000 --> 00:16:39.600
able to talk.
00:16:39.600 --> 00:16:45.220
So if B is 10 and 12 people want
to talk at the same time,
00:16:45.220 --> 00:16:48.490
then 2 of these people would get
a busy signal, and that's
00:16:48.490 --> 00:16:50.220
not something that we like.
00:16:50.220 --> 00:16:53.010
We would like B to be large
enough so that there's a
00:16:53.010 --> 00:16:57.240
substantial probability, that
there's almost certainty that,
00:16:57.240 --> 00:17:00.120
under reasonable conditions,
no one is going
00:17:00.120 --> 00:17:02.230
to get a busy signal.
00:17:02.230 --> 00:17:06.720
So how do we go about modeling
a situation like this?
00:17:06.720 --> 00:17:09.890
Well, to set up a model you
need two pieces, one is to
00:17:09.890 --> 00:17:17.000
describe how do phone calls
get initiated, and once a
00:17:17.000 --> 00:17:21.819
phone call gets started, how
long does it take until the
00:17:21.819 --> 00:17:24.530
phone call is terminated?
00:17:24.530 --> 00:17:27.530
So we're going to make the
simplest assumptions possible.
00:17:27.530 --> 00:17:29.890
Let's assume that phone
calls originate
00:17:29.890 --> 00:17:31.920
as a Poisson process.
00:17:31.920 --> 00:17:34.210
That is, out of that population
people do not
00:17:34.210 --> 00:17:35.290
really coordinate.
00:17:35.290 --> 00:17:38.160
At completely random times,
different people with decide
00:17:38.160 --> 00:17:39.740
to pick up the phone.
00:17:39.740 --> 00:17:42.320
There's no dependencies between
different people,
00:17:42.320 --> 00:17:44.790
there's nothing special about
different times, different
00:17:44.790 --> 00:17:46.340
times are independent.
00:17:46.340 --> 00:17:50.550
So a Poisson model is a
reasonable way of modeling
00:17:50.550 --> 00:17:51.410
this situation.
00:17:51.410 --> 00:17:55.870
And it's going to be a Poisson
process with some rate lambda.
00:17:55.870 --> 00:17:59.870
Now, the rate lambda would be
easy to estimate in practice.
00:17:59.870 --> 00:18:02.760
You observe what happens in
that village just over a
00:18:02.760 --> 00:18:06.050
couple of days, and you figure
out what's the rate at which
00:18:06.050 --> 00:18:09.550
people attempt to place
phone calls.
00:18:09.550 --> 00:18:11.930
Now, about phone calls
themselves, we're going to
00:18:11.930 --> 00:18:15.120
make the assumption that the
duration of a phone call is a
00:18:15.120 --> 00:18:18.580
random variable that has an
exponential distribution with
00:18:18.580 --> 00:18:20.630
a certain parameter mu.
00:18:20.630 --> 00:18:24.400
So 1/mu is the mean duration
of a phone call.
00:18:24.400 --> 00:18:27.570
So the mean duration, , again,
is easy to estimate.
00:18:27.570 --> 00:18:31.240
You just observe what's
happening, see on the average
00:18:31.240 --> 00:18:33.680
how long these phone
calls are.
00:18:33.680 --> 00:18:36.590
Is the exponential assumption
a good assumption?
00:18:36.590 --> 00:18:39.990
Well, it's means that most phone
calls will be kind of
00:18:39.990 --> 00:18:43.180
short, but there's going to be a
fraction of phone calls that
00:18:43.180 --> 00:18:45.640
are going to be larger, and
then a very small fraction
00:18:45.640 --> 00:18:47.900
that are going to
be even larger.
00:18:47.900 --> 00:18:49.880
So it sounds plausible.
00:18:49.880 --> 00:18:57.960
It's not exactly realistic, that
is, phone calls that last
00:18:57.960 --> 00:19:02.250
short of 15 seconds are
not that common.
00:19:02.250 --> 00:19:04.890
So either nothing happens
or you have to say a few
00:19:04.890 --> 00:19:06.630
sentences and so on.
00:19:06.630 --> 00:19:10.200
Also, back into the days when
people used to connect to the
00:19:10.200 --> 00:19:15.430
internet using dial up modems,
that assumption was completely
00:19:15.430 --> 00:19:20.390
destroyed, because people would
dial up and then keep
00:19:20.390 --> 00:19:25.730
their phone line busy for a few
hours, if the phone call
00:19:25.730 --> 00:19:26.930
was a free one.
00:19:26.930 --> 00:19:30.390
So at those times the
exponential assumption for the
00:19:30.390 --> 00:19:33.260
phone call duration was
completely destroyed.
00:19:33.260 --> 00:19:36.430
But leaving that detail aside,
it's sort of a reasonable
00:19:36.430 --> 00:19:41.000
assumption to just get started
with this problem.
00:19:41.000 --> 00:19:43.790
All right, so now that we have
those assumptions, let's try
00:19:43.790 --> 00:19:46.510
to come up with the model.
00:19:46.510 --> 00:19:49.470
And we're going to set up
a Markov process model.
00:19:49.470 --> 00:19:52.990
Now the Poisson process runs in
continuous time, and call
00:19:52.990 --> 00:19:56.280
durations being exponential
random variables also are
00:19:56.280 --> 00:19:59.330
continuous random variables, so
it seems that we are in a
00:19:59.330 --> 00:20:01.020
continuous time universe.
00:20:01.020 --> 00:20:03.610
But we have only started
Markov chains for the
00:20:03.610 --> 00:20:05.310
discrete time case.
00:20:05.310 --> 00:20:07.370
What are we going to do?
00:20:07.370 --> 00:20:10.470
We can either develop the theory
of continuous time
00:20:10.470 --> 00:20:13.600
Markov chains, which
is possible.
00:20:13.600 --> 00:20:16.300
But we are not going to
do that in this class.
00:20:16.300 --> 00:20:20.460
Or we can discretize time
and work with a
00:20:20.460 --> 00:20:22.070
discrete time model.
00:20:22.070 --> 00:20:24.850
So we're going to discretize
time in the familiar way, the
00:20:24.850 --> 00:20:27.350
way we did it when we started
the Poisson process.
00:20:27.350 --> 00:20:31.270
We're going to take the time
axis and split it into little
00:20:31.270 --> 00:20:35.130
discrete mini slots, where
every mini slot
00:20:35.130 --> 00:20:36.880
has a duration delta.
00:20:36.880 --> 00:20:41.520
So this delta is supposed to
be a very small number.
00:20:41.520 --> 00:20:44.760
So what is the state
of the system?
00:20:44.760 --> 00:20:47.290
So, you look at the situation
in the system at some
00:20:47.290 --> 00:20:51.340
particular time and I ask you
what is going on right now,
00:20:51.340 --> 00:20:53.680
what's the information
you would tell me?
00:20:53.680 --> 00:20:56.990
Well, you would tell me that
right now out of these capital
00:20:56.990 --> 00:21:02.160
B lines, 10 of them are busy,
or 12 of them are busy.
00:21:02.160 --> 00:21:04.740
That describes the state of
the system, that tells me
00:21:04.740 --> 00:21:06.560
what's happening
at this point.
00:21:06.560 --> 00:21:10.800
So we set up our states base by
being the numbers from 0 to
00:21:10.800 --> 00:21:15.510
B. 0 corresponds to a state in
which all the phone lines are
00:21:15.510 --> 00:21:17.260
free, no one is talking.
00:21:17.260 --> 00:21:19.590
Capital B corresponds to
a case where all the
00:21:19.590 --> 00:21:21.580
phone lines are busy.
00:21:21.580 --> 00:21:24.020
And then you've got
states in between.
00:21:24.020 --> 00:21:28.050
And now let's look at the
transition probabilities.
00:21:28.050 --> 00:21:32.580
Suppose that right so now we
have i-1 lines that are busy.
00:21:36.280 --> 00:21:38.380
Or maybe, let me look here.
00:21:38.380 --> 00:21:41.460
Suppose that there's i
lines that are busy.
00:21:41.460 --> 00:21:44.770
What can happen the next time?
00:21:44.770 --> 00:21:48.180
What can happen is that the new
phone call gets placed, in
00:21:48.180 --> 00:21:53.670
which case my state moves up
by 1, or an existing call
00:21:53.670 --> 00:21:59.060
terminates, in which case my
state goes down by 1, or none
00:21:59.060 --> 00:22:03.530
of the two happens, in which
case I stay at the same state.
00:22:03.530 --> 00:22:06.930
Well, it's also possible that
the phone call gets terminated
00:22:06.930 --> 00:22:10.510
and a new phone call gets placed
sort of simultaneously.
00:22:10.510 --> 00:22:14.080
But when you take your time
slots to be very, very small,
00:22:14.080 --> 00:22:16.970
this is going to have a
negligible probability order
00:22:16.970 --> 00:22:19.850
of delta squared, so
we ignore this.
00:22:19.850 --> 00:22:22.190
So what's the probability of
an upwards transition?
00:22:22.190 --> 00:22:25.160
That's the probability that the
Poisson process records an
00:22:25.160 --> 00:22:29.250
arrival during a mini slot
of duration delta.
00:22:29.250 --> 00:22:31.270
By the definition of the
Poisson process, the
00:22:31.270 --> 00:22:35.550
probability of this happening
is just lambda delta.
00:22:35.550 --> 00:22:39.200
So each one of these upwards
transitions has the same
00:22:39.200 --> 00:22:41.000
probability of lambda delta.
00:22:41.000 --> 00:22:45.220
So you've got lambda deltas
everywhere in this diagram.
00:22:45.220 --> 00:22:49.180
How about, now, phone
call terminations?
00:22:49.180 --> 00:22:53.630
If you had the single call that
was active, so if you
00:22:53.630 --> 00:22:56.510
were here, what's the
probability that the phone
00:22:56.510 --> 00:22:57.820
call terminates?
00:22:57.820 --> 00:23:00.840
So the phone call has an
exponential duration with
00:23:00.840 --> 00:23:02.820
parameter mu.
00:23:02.820 --> 00:23:05.860
And we discussed before that an
exponential random variable
00:23:05.860 --> 00:23:08.940
can be thought of as the
first arrival time
00:23:08.940 --> 00:23:10.780
in a Poisson process.
00:23:10.780 --> 00:23:14.750
So the probability that you get
this event to happen over
00:23:14.750 --> 00:23:18.820
a delta time interval is
just mu times delta.
00:23:18.820 --> 00:23:22.280
So if you have a single phone
call that's happening right
00:23:22.280 --> 00:23:24.780
now, with probability
mu times delta, that
00:23:24.780 --> 00:23:27.070
call is going to terminate.
00:23:27.070 --> 00:23:30.020
But suppose that we have
i phone calls that
00:23:30.020 --> 00:23:31.750
are currently active.
00:23:31.750 --> 00:23:35.010
Each one of them has a
probability of mu delta, of
00:23:35.010 --> 00:23:39.200
terminating, but collectively
the probability that one of
00:23:39.200 --> 00:23:45.850
them terminates becomes
i times mu delta.
00:23:45.850 --> 00:23:48.570
So that's because you get the
mu delta contribution --
00:23:48.570 --> 00:23:51.540
the probability of termination
from each one of the different
00:23:51.540 --> 00:23:54.410
phone calls.
00:23:54.410 --> 00:23:58.290
OK, now this is an approximate
calculation, because it
00:23:58.290 --> 00:24:01.850
ignores the possibility that two
phone calls terminate at
00:24:01.850 --> 00:24:03.110
the same time.
00:24:03.110 --> 00:24:09.130
Again, the way to think of why
this is the correct rate, when
00:24:09.130 --> 00:24:13.610
you have i phone calls that are
simultaneously running and
00:24:13.610 --> 00:24:17.250
waiting for one of them to
terminate, this is like having
00:24:17.250 --> 00:24:21.170
i separate Poisson processes
that are running in parallel,
00:24:21.170 --> 00:24:23.720
and you ask for the probability
that one of those
00:24:23.720 --> 00:24:25.980
processes records an event.
00:24:25.980 --> 00:24:28.470
Now when you put all those
process together, it's like
00:24:28.470 --> 00:24:32.970
having a Poisson process with
total rate i times mu, and so
00:24:32.970 --> 00:24:36.210
i times mu delta is the overall
probability that
00:24:36.210 --> 00:24:39.580
something happens in terms of
phone call terminations at
00:24:39.580 --> 00:24:40.760
those times.
00:24:40.760 --> 00:24:43.470
So in any case, this is the
transition probability for
00:24:43.470 --> 00:24:46.280
downwards transitions.
00:24:46.280 --> 00:24:49.190
Now that we've got this, we
can analyze this chain.
00:24:49.190 --> 00:24:53.220
This chain has the birth death
form that we discussed towards
00:24:53.220 --> 00:24:54.770
the end of last lecture.
00:24:54.770 --> 00:24:58.360
And for birth death chains, it's
easy to write it out to
00:24:58.360 --> 00:25:00.710
find the steady state
probabilities.
00:25:00.710 --> 00:25:03.360
Instead of writing down the
balance equations in the
00:25:03.360 --> 00:25:07.950
general form, we think in terms
of a conservation of
00:25:07.950 --> 00:25:10.630
probabilities or of transitions
by looking at what
00:25:10.630 --> 00:25:14.640
happens across a particular
cut in this diagram.
00:25:14.640 --> 00:25:18.590
Number of transitions in the
chain that cross from here to
00:25:18.590 --> 00:25:21.310
here has to be approximately
equal to the number of
00:25:21.310 --> 00:25:24.850
transitions from here to there
because whatever comes up must
00:25:24.850 --> 00:25:27.520
come down and then come
up and so on.
00:25:27.520 --> 00:25:31.410
So the frequency with which
transitions of this kind are
00:25:31.410 --> 00:25:34.070
observed has to be the same as
the frequency of transitions
00:25:34.070 --> 00:25:35.690
of this kind.
00:25:35.690 --> 00:25:38.400
What's the frequency of how
often the transitions of this
00:25:38.400 --> 00:25:40.280
kind happen?
00:25:40.280 --> 00:25:45.350
And by frequency I mean quite
percentage of the mini slots
00:25:45.350 --> 00:25:48.120
involve a transition
of this kind?
00:25:48.120 --> 00:25:51.420
Well, for a transition of that
kind to happen we need to be
00:25:51.420 --> 00:25:55.730
at states i-1, which happens
this much of the time.
00:25:55.730 --> 00:25:59.360
And then the probability
lambda delta that the
00:25:59.360 --> 00:26:01.360
transition is of this kind.
00:26:01.360 --> 00:26:06.040
So the frequency of transitions
of with which this
00:26:06.040 --> 00:26:11.460
kind of transition is observed
is lambda delta times pi(i-1).
00:26:11.460 --> 00:26:17.680
This is the fraction of time
steps at which a transition
00:26:17.680 --> 00:26:20.180
from specifically this
state to specifically
00:26:20.180 --> 00:26:22.120
that state are observed.
00:26:22.120 --> 00:26:24.880
This has to be the same as
the frequency with which
00:26:24.880 --> 00:26:28.360
transitions of that kind are
observed, and that frequency
00:26:28.360 --> 00:26:32.310
is going to be i mu delta
times pi(i), and then we
00:26:32.310 --> 00:26:37.870
cancel the deltas, and we are
left with this equation here.
00:26:37.870 --> 00:26:43.260
So this equation expresses pi(i)
in terms of pi(i-1).
00:26:43.260 --> 00:26:47.020
So if we knew pi(0) we
can use that equation
00:26:47.020 --> 00:26:48.690
to determine pi(1).
00:26:48.690 --> 00:26:52.320
Once we know pi(1), we can use
that equation to determine
00:26:52.320 --> 00:26:55.490
pi(2), and so on,
you keep going.
00:26:55.490 --> 00:26:59.890
And the general formula that
comes out of this, I will not
00:26:59.890 --> 00:27:02.350
do the algebra, it's a
straightforward substitution,
00:27:02.350 --> 00:27:04.830
you find that pi(i), the steady
state probability of
00:27:04.830 --> 00:27:08.650
state i is given by this
expression, which involves the
00:27:08.650 --> 00:27:12.020
pi(0) from which we started.
00:27:12.020 --> 00:27:13.500
Now what is pi(0)?
00:27:13.500 --> 00:27:17.420
Well, we don't know yet, but
we can find it by using the
00:27:17.420 --> 00:27:19.520
normalization equation.
00:27:19.520 --> 00:27:22.680
The sum of pi(i) has
to be equal to 1.
00:27:22.680 --> 00:27:26.430
So the sum of all of those
numbers has to be equal to 1.
00:27:26.430 --> 00:27:30.370
And the only way that this can
happen is by setting pi(0) to
00:27:30.370 --> 00:27:33.320
be equal to that particular
number.
00:27:33.320 --> 00:27:38.970
So if I tell you the value of
capital B, you can set up this
00:27:38.970 --> 00:27:43.690
Markov chain, you can calculate
pi(0), and then you
00:27:43.690 --> 00:27:47.790
can calculate pi(i), and so you
know what fraction, you
00:27:47.790 --> 00:27:51.150
know the steady state
probabilities of this chain,
00:27:51.150 --> 00:27:53.920
so you can answer
the question.
00:27:53.920 --> 00:27:57.240
If I drop in at a random time,
how likely is it that I'm
00:27:57.240 --> 00:28:00.200
going to find the states
to be here, or the
00:28:00.200 --> 00:28:01.930
states to be there?
00:28:01.930 --> 00:28:03.790
So the steady state
probabilities are
00:28:03.790 --> 00:28:07.390
probabilities, but we also
interpret them as frequencies.
00:28:07.390 --> 00:28:12.000
So once I find pi(i), it also
tells me what fraction of the
00:28:12.000 --> 00:28:16.870
time is the state equal to i.
00:28:16.870 --> 00:28:20.030
And you can answer that question
for every possible i.
00:28:20.030 --> 00:28:22.180
Now, why did we do
this exercise?
00:28:22.180 --> 00:28:25.240
We're interested in
the probability of
00:28:25.240 --> 00:28:27.660
the system is busy.
00:28:27.660 --> 00:28:31.960
So if a person, a new phone
call gets placed, it just
00:28:31.960 --> 00:28:33.020
drops out of the sky.
00:28:33.020 --> 00:28:37.560
According to that Poisson
process, that new phone call
00:28:37.560 --> 00:28:41.770
is going to find the system
at a random state.
00:28:41.770 --> 00:28:45.290
That random state is described
in steady state by the
00:28:45.290 --> 00:28:47.440
probabilities pi(i)'s.
00:28:47.440 --> 00:28:51.940
And the probability that you
find the system to be busy is
00:28:51.940 --> 00:28:55.110
the probability that when you
drop in the state happens to
00:28:55.110 --> 00:29:01.380
be that particular number B. So
i sub b is the probability
00:29:01.380 --> 00:29:02.680
of being busy.
00:29:06.496 --> 00:29:09.470
And this is the probability
that you would like to be
00:29:09.470 --> 00:29:11.970
small in a well engineered
system.
00:29:11.970 --> 00:29:16.890
So you ask the question, how
should, given my lambda and
00:29:16.890 --> 00:29:22.570
mu, my design question is to
determine capital B the number
00:29:22.570 --> 00:29:25.720
of phone lines so that
this number is small.
00:29:31.290 --> 00:29:36.040
Could we have done, could we
figure out a good value for B
00:29:36.040 --> 00:29:39.670
by doing a back of the
envelope calculation?
00:29:39.670 --> 00:29:44.410
Let's suppose that lambda is
30 and that mu is 1/3.
00:29:49.190 --> 00:29:53.640
So I guess that's, let
us these rates to
00:29:53.640 --> 00:29:55.135
be calls per minute.
00:29:58.400 --> 00:30:03.100
And this mu, again, is
a rate per minute.
00:30:03.100 --> 00:30:07.860
Again, the units of mu are going
to be calls per minute.
00:30:07.860 --> 00:30:11.310
So since our time unit is
minutes, the mean duration of
00:30:11.310 --> 00:30:13.660
calls is 1/mu minutes.
00:30:13.660 --> 00:30:16.930
So a typical call, or
on the average a
00:30:16.930 --> 00:30:19.185
call lasts for 3 minutes.
00:30:23.900 --> 00:30:27.860
So you get 30 calls
per minute.
00:30:27.860 --> 00:30:31.540
Each call lasts for 3 minutes
on the average.
00:30:31.540 --> 00:30:36.290
So on the average, if
B was infinite,
00:30:36.290 --> 00:30:38.100
every call goes through.
00:30:38.100 --> 00:30:42.750
How many calls would be
active on the average?
00:30:42.750 --> 00:30:44.570
So you get 30 per minute.
00:30:44.570 --> 00:30:48.490
If a call lasted exactly 1
minute, then at any time you
00:30:48.490 --> 00:30:51.210
would have 30 calls
being active.
00:30:51.210 --> 00:30:54.850
Now a call lasts on the
average for 3 minutes.
00:30:54.850 --> 00:30:58.120
So during each minute
you generate 90
00:30:58.120 --> 00:31:00.460
minutes of talking time.
00:31:00.460 --> 00:31:05.400
So by thinking in terms of
averages you would expect that
00:31:05.400 --> 00:31:08.070
at any time there would
be about 90
00:31:08.070 --> 00:31:10.540
calls that are active.
00:31:10.540 --> 00:31:14.210
And if 90 calls are active on
the average, you could say OK,
00:31:14.210 --> 00:31:18.600
I'm going to set up my
capital B to be 90.
00:31:18.600 --> 00:31:22.250
But that's not very good,
because if the average number
00:31:22.250 --> 00:31:25.950
of phone calls that want to
happen is if the average
00:31:25.950 --> 00:31:29.900
number is 90, sometimes you're
going to have 85, sometimes
00:31:29.900 --> 00:31:31.460
you will have 95.
00:31:31.460 --> 00:31:34.300
And to be sure that the phone
calls will go through you
00:31:34.300 --> 00:31:37.790
probably want to choose your
capital B to be a number a
00:31:37.790 --> 00:31:40.460
little larger than 90.
00:31:40.460 --> 00:31:42.600
How much larger than 90?
00:31:42.600 --> 00:31:47.070
Well, this is a question that
you can answer numerically.
00:31:47.070 --> 00:31:50.280
So you go through the
following procedure.
00:31:50.280 --> 00:31:55.590
I tried different values of
capital B. For any given value
00:31:55.590 --> 00:31:59.230
of capital B, I do this
numerical calculation, I find
00:31:59.230 --> 00:32:03.530
the probability that the system
is busy, and then I ask
00:32:03.530 --> 00:32:07.060
what's the value of B that makes
my probability of being
00:32:07.060 --> 00:32:10.510
busy to be, let's say,
roughly 1 %.
00:32:10.510 --> 00:32:13.110
And if you do that calculation
with the parameters that they
00:32:13.110 --> 00:32:18.620
gave you, you find that B would
be something like 106.
00:32:18.620 --> 00:32:21.490
So with the parameters they gave
where you have, on the
00:32:21.490 --> 00:32:26.550
average, 90 phone calls being
active, you actually need some
00:32:26.550 --> 00:32:30.410
margin to protect against the
[?] fluctuation, if suddenly
00:32:30.410 --> 00:32:34.010
by chance more people want to
talk, and if you want to have
00:32:34.010 --> 00:32:37.890
a good guarantee that an
incoming person will have a
00:32:37.890 --> 00:32:40.860
very small probability of
finding a busy system, then
00:32:40.860 --> 00:32:44.860
you will need about
106 phone lines.
00:32:44.860 --> 00:32:49.740
So that's the calculation and
the argument that the Erlang
00:32:49.740 --> 00:32:52.250
went through a long time ago.
00:32:52.250 --> 00:32:55.610
It's actually interesting that
Erlang did this calculation
00:32:55.610 --> 00:32:58.590
before Markov chains
were invented.
00:32:58.590 --> 00:33:02.250
So Markov's work, and the
beginning of work on Markov
00:33:02.250 --> 00:33:06.770
chains, happens about 10-15
years after Erlang.
00:33:06.770 --> 00:33:10.180
So obviously he didn't call
that a Markov chain.
00:33:10.180 --> 00:33:13.350
But it was something that he
could study from first
00:33:13.350 --> 00:33:15.900
principles.
00:33:15.900 --> 00:33:19.160
So this is a pretty
useful thing.
00:33:19.160 --> 00:33:24.530
These probabilities that come
out of that model, at least in
00:33:24.530 --> 00:33:28.230
the old days, they would all
be very well tabulated in
00:33:28.230 --> 00:33:32.670
handbooks that every decent
phone company engineer would
00:33:32.670 --> 00:33:34.530
sort of have with them.
00:33:34.530 --> 00:33:38.890
So this is about as practical
as it gets.
00:33:38.890 --> 00:33:42.340
It's one of the sort of
standard real world
00:33:42.340 --> 00:33:43.810
applications of Markov chains.
00:33:47.040 --> 00:33:53.950
So now to close our subjects,
we're going to consider a
00:33:53.950 --> 00:33:57.310
couple of new skills and see how
we can calculate the few
00:33:57.310 --> 00:33:59.660
additional interesting
quantities that have to do
00:33:59.660 --> 00:34:01.340
with the Markov chain.
00:34:01.340 --> 00:34:04.800
So the problem we're going to
deal with here is the one I
00:34:04.800 --> 00:34:07.630
hinted that when I was talking
about this picture.
00:34:07.630 --> 00:34:09.870
You start at a transient
state, you're going to
00:34:09.870 --> 00:34:12.239
eventually end up
here or there.
00:34:12.239 --> 00:34:16.389
We want to find the
probabilities of one option of
00:34:16.389 --> 00:34:19.210
the two happening or the
other happening.
00:34:19.210 --> 00:34:23.880
So in this picture we
have a class of
00:34:23.880 --> 00:34:25.680
states that's are transient.
00:34:29.130 --> 00:34:34.219
These are transient because
you're going to move around
00:34:34.219 --> 00:34:37.170
those states, but there's a
transition that you can make,
00:34:37.170 --> 00:34:40.260
and you go to a state from
which you cannot escape
00:34:40.260 --> 00:34:41.560
afterwards.
00:34:41.560 --> 00:34:43.710
Are you going to end
up here or are you
00:34:43.710 --> 00:34:45.370
going to end up there?
00:34:45.370 --> 00:34:46.130
You don't know.
00:34:46.130 --> 00:34:47.280
It's random.
00:34:47.280 --> 00:34:50.940
Let's try to calculate the
probability that you
00:34:50.940 --> 00:34:54.800
end up at state 4.
00:34:54.800 --> 00:34:59.990
Now, the probability that you
end up at state 4 will depend
00:34:59.990 --> 00:35:02.200
on where you start.
00:35:02.200 --> 00:35:06.170
Because if you start here, you
probably have more chances of
00:35:06.170 --> 00:35:09.720
getting to 4 because you get
that chance immediately,
00:35:09.720 --> 00:35:12.390
whereas if you start here
there's more chances that
00:35:12.390 --> 00:35:16.260
you're going to escape that way
because it kind of takes
00:35:16.260 --> 00:35:17.770
you time to get there.
00:35:17.770 --> 00:35:20.810
It's more likely that
you exit right away.
00:35:20.810 --> 00:35:26.040
So the probability of exiting
and ending up at state 4 will
00:35:26.040 --> 00:35:28.190
depend on the initial state.
00:35:28.190 --> 00:35:33.900
That's why when we talk about
these absorption probability
00:35:33.900 --> 00:35:38.270
we include an index i that
tells us what the
00:35:38.270 --> 00:35:40.310
initial state is.
00:35:40.310 --> 00:35:44.350
And we want to find this
absorption probability, the
00:35:44.350 --> 00:35:46.820
probability that we end
up here for the
00:35:46.820 --> 00:35:48.660
different initial states.
00:35:48.660 --> 00:35:52.100
Now for some initial states this
is very easy to answer.
00:35:52.100 --> 00:35:55.280
If you start at state 4, what's
the probability that
00:35:55.280 --> 00:35:59.160
eventually you end up in
this part of the chain?
00:35:59.160 --> 00:35:59.820
It's 1.
00:35:59.820 --> 00:36:02.585
You're certain to be there,
that's where you started.
00:36:02.585 --> 00:36:06.140
If you start at state 5, what's
the probability that
00:36:06.140 --> 00:36:08.880
you end up eventually
at state 4?
00:36:08.880 --> 00:36:12.580
It's probability 0, there's
no way to get there.
00:36:12.580 --> 00:36:17.930
Now, how about if you start
at a state like state 2?
00:36:20.840 --> 00:36:26.350
If you start at state 2 then
there's a few different things
00:36:26.350 --> 00:36:27.840
that can happen.
00:36:27.840 --> 00:36:32.900
Either you end up at state 4
right away and this happens
00:36:32.900 --> 00:36:41.150
with probability 0.2, or you
end up at state 1, and this
00:36:41.150 --> 00:36:46.730
happens with probability 0.6.
00:36:46.730 --> 00:36:50.200
So if you end up at state
4, you are done.
00:36:50.200 --> 00:36:51.540
We are there.
00:36:51.540 --> 00:36:56.850
If you end up at state
1, then what?
00:36:56.850 --> 00:37:00.980
Starting from state 1 there's
two possibilities.
00:37:00.980 --> 00:37:05.360
Either eventually you're going
to end up at state 4, or
00:37:05.360 --> 00:37:10.010
eventually you're going
to end up at state 5.
00:37:10.010 --> 00:37:14.310
What's the probability
of this happening?
00:37:14.310 --> 00:37:19.680
We don't know what it is, but
it's what we defined to be a1.
00:37:19.680 --> 00:37:21.130
This is the probability --
00:37:21.130 --> 00:37:22.650
a1 is the probability --
00:37:22.650 --> 00:37:26.650
that eventually you settle in
state 4 given that the initial
00:37:26.650 --> 00:37:28.090
state was 1.
00:37:28.090 --> 00:37:30.580
So this probability is a1.
00:37:30.580 --> 00:37:34.510
So our event of interest
can happen in two ways.
00:37:34.510 --> 00:37:38.350
Either I go there directly,
or I go here
00:37:38.350 --> 00:37:39.710
with probability 0.6.
00:37:39.710 --> 00:37:43.730
And given that I go there,
eventually I end up at state
00:37:43.730 --> 00:37:46.540
4, which happens with
probability a1.
00:37:46.540 --> 00:37:52.140
So the total probability of
ending up at state 4 is going
00:37:52.140 --> 00:37:54.660
to be the sum of the
probabilities of the different
00:37:54.660 --> 00:37:57.230
ways that this event
can happen.
00:37:57.230 --> 00:38:04.250
So our equation, in this case,
is going to be, that's a2, is
00:38:04.250 --> 00:38:07.420
going to be 0.2 (that's the
probability of going there
00:38:07.420 --> 00:38:11.210
directly) plus with probability
0.8 I end up at
00:38:11.210 --> 00:38:17.160
state 1, and then from state 1
I will end up at state 4 with
00:38:17.160 --> 00:38:19.320
probability a1.
00:38:19.320 --> 00:38:25.330
So this is one particular
equation that we've got for
00:38:25.330 --> 00:38:28.560
what happens if we start
from this state.
00:38:28.560 --> 00:38:32.450
We can do a similar argument
starting from any other state.
00:38:32.450 --> 00:38:35.790
Starting from state i the
probability that eventually I
00:38:35.790 --> 00:38:39.960
end up at state 4 is, we
consider the different
00:38:39.960 --> 00:38:43.350
possible scenarios of where do
I go next, which is my state
00:38:43.350 --> 00:38:46.440
j, with probability Pij.
00:38:46.440 --> 00:38:50.980
Next time I go to j, and given
that I started at j, this is
00:38:50.980 --> 00:38:53.630
the probability that I
end up at state 4.
00:38:53.630 --> 00:38:56.920
So this equation that we have
here is just an abstract
00:38:56.920 --> 00:39:02.540
version in symbols of what we
wrote down for the particular
00:39:02.540 --> 00:39:04.540
case where the initial
state was 2.
00:39:04.540 --> 00:39:08.880
So you write down an equation
of this type for every state
00:39:08.880 --> 00:39:09.580
inside here.
00:39:09.580 --> 00:39:15.520
You'll have a separate equation
for a1, a2, and a3.
00:39:15.520 --> 00:39:19.200
And that's going to be a system
of 3 equations with 3
00:39:19.200 --> 00:39:25.250
unknowns, the a's inside
the transient states.
00:39:25.250 --> 00:39:29.190
So you can solve that 3 by
3 system of equations.
00:39:29.190 --> 00:39:33.590
Fortunately, it turns out to
have a unique solution, and so
00:39:33.590 --> 00:39:36.070
once you solve it you have found
the probabilities of
00:39:36.070 --> 00:39:40.880
absorption and the probability
that eventually you get
00:39:40.880 --> 00:39:42.610
absorbed at state 4.
00:39:51.540 --> 00:39:56.790
Now, in the picture that we had
here, this was a single
00:39:56.790 --> 00:39:59.570
state, and that one was
a single state.
00:39:59.570 --> 00:40:06.500
How do things change if our
recurrent, or trapping sets
00:40:06.500 --> 00:40:09.600
consist of multiple states?
00:40:09.600 --> 00:40:16.820
Well, it doesn't really matter
that we have multiple states.
00:40:16.820 --> 00:40:21.470
All that matters is that this
is one lump and once we get
00:40:21.470 --> 00:40:24.150
there we are stuck in there.
00:40:24.150 --> 00:40:33.170
So if the picture was, let's
say, like this, 0.1 and 0.2,
00:40:33.170 --> 00:40:36.960
that basically means that
whenever you are in that state
00:40:36.960 --> 00:40:42.620
there's a total probability of
0.3 of ending in that lump and
00:40:42.620 --> 00:40:45.100
getting stuck inside
that lump.
00:40:45.100 --> 00:40:50.660
So you would take that picture
and change it and make it
00:40:50.660 --> 00:40:57.330
instead a total probability of
0.3, of ending somewhere
00:40:57.330 --> 00:41:00.270
inside that lump.
00:41:00.270 --> 00:41:03.670
And similarly, you take this
lump and you view it as just
00:41:03.670 --> 00:41:07.570
one entity, and from any state
you record the total
00:41:07.570 --> 00:41:10.090
probability that given
that I'm here I
00:41:10.090 --> 00:41:12.140
end up in that entity.
00:41:12.140 --> 00:41:15.540
So basically, if the only
thing you care is the
00:41:15.540 --> 00:41:18.760
probability that you're going
to end up in this lump, you
00:41:18.760 --> 00:41:22.830
can replace that lump with a
single state, view it as a
00:41:22.830 --> 00:41:24.600
single state, and calculate
00:41:24.600 --> 00:41:26.180
probabilities using this formula.
00:41:33.830 --> 00:41:36.780
All right, so now we know
where the chain is
00:41:36.780 --> 00:41:37.860
going to get to.
00:41:37.860 --> 00:41:39.850
At least we know
probabilistically.
00:41:39.850 --> 00:41:42.690
We know with what probability
it is going to go here, and
00:41:42.690 --> 00:41:45.160
that also tells us the
probability that eventually
00:41:45.160 --> 00:41:47.160
it's going to get there.
00:41:47.160 --> 00:41:55.780
Other question, how long is it
going to take until we get to
00:41:55.780 --> 00:41:58.300
either this state
or that state?
00:41:58.300 --> 00:42:02.160
We can call that event
absorption, meaning that the
00:42:02.160 --> 00:42:05.130
state got somewhere into a
recurrent class from which it
00:42:05.130 --> 00:42:06.380
could not get out.
00:42:12.340 --> 00:42:12.510
Okay.
00:42:12.510 --> 00:42:15.570
Let's deal with that question
for the case where we have
00:42:15.570 --> 00:42:19.320
only 1 absorbing state.
00:42:19.320 --> 00:42:22.410
So here our Markov chain is a
little simpler than the one in
00:42:22.410 --> 00:42:23.580
the previous slide.
00:42:23.580 --> 00:42:25.870
We've got our transient
states, we've got our
00:42:25.870 --> 00:42:28.930
recurrent state, and once you
get into the recurrent state
00:42:28.930 --> 00:42:31.770
you just stay there.
00:42:31.770 --> 00:42:35.330
So here we're certain that no
matter where we start we're
00:42:35.330 --> 00:42:37.270
going to end up here.
00:42:37.270 --> 00:42:39.170
How long is it going to take?
00:42:39.170 --> 00:42:40.830
Well, we don't know.
00:42:40.830 --> 00:42:42.540
It's a random variable.
00:42:42.540 --> 00:42:44.660
The expected value of that
random variable,
00:42:44.660 --> 00:42:46.890
let's call it mu.
00:42:46.890 --> 00:42:50.510
But how long it takes to get
there certainly depends on
00:42:50.510 --> 00:42:52.470
where we start.
00:42:52.470 --> 00:42:56.390
So let's put in our notation
again this index i that
00:42:56.390 --> 00:42:59.220
indicates where we
started from.
00:42:59.220 --> 00:43:03.300
And now the argument is going to
be of the same type as the
00:43:03.300 --> 00:43:05.480
one we used before.
00:43:05.480 --> 00:43:11.620
We can think in terms of a tree
once more, that considers
00:43:11.620 --> 00:43:13.930
all the possible options.
00:43:13.930 --> 00:43:16.300
So suppose that you
start at state 1.
00:43:19.060 --> 00:43:24.110
Starting from state 1, the
expected time until you end up
00:43:24.110 --> 00:43:27.100
dropping states is mu1.
00:43:27.100 --> 00:43:30.700
Now, starting from state 1, what
are the possibilities?
00:43:30.700 --> 00:43:33.090
You make your first transition,
and that first
00:43:33.090 --> 00:43:36.550
transition is going to take
you either to state
00:43:36.550 --> 00:43:38.610
2 or to state 3.
00:43:38.610 --> 00:43:42.010
It takes you to state 2 with
probability 0.6, it takes you
00:43:42.010 --> 00:43:48.320
to state 3 with probability
0.4.
00:43:48.320 --> 00:43:52.490
Starting from state 2,
eventually you're going to get
00:43:52.490 --> 00:43:54.410
to state 4.
00:43:54.410 --> 00:43:55.990
How long does it take?
00:43:55.990 --> 00:43:58.480
We don't know, it's
a random variable.
00:43:58.480 --> 00:44:02.325
But the expected time until
this happens is mu2.
00:44:05.100 --> 00:44:08.250
Starting from state 2, how long
does it take you to get
00:44:08.250 --> 00:44:10.300
to state 4.
00:44:10.300 --> 00:44:13.870
And similarly starting from
state 3, it's going to take
00:44:13.870 --> 00:44:17.850
you on the average mu3
time steps until you
00:44:17.850 --> 00:44:20.070
get to state 4.
00:44:20.070 --> 00:44:24.450
So what's the expected value
of the time until I
00:44:24.450 --> 00:44:25.790
end at state 4?
00:44:31.070 --> 00:44:37.910
So with probability 0.6, I'm
going to end up at state 2 and
00:44:37.910 --> 00:44:42.650
from there on it's going to be
expected time mu2, and with
00:44:42.650 --> 00:44:45.850
probability 0.4 I'm going to
end up at state 3, and from
00:44:45.850 --> 00:44:48.730
there it's going to take
me so much time.
00:44:48.730 --> 00:44:52.790
So this is the expected time
it's going to take me after
00:44:52.790 --> 00:44:54.860
the first transition.
00:44:54.860 --> 00:44:59.420
But we also spent 1 time step
for the first transition.
00:44:59.420 --> 00:45:02.610
The total time to get there
is the time of the first
00:45:02.610 --> 00:45:06.330
transition, which is 1, plus
the expected time starting
00:45:06.330 --> 00:45:07.600
from the next state.
00:45:07.600 --> 00:45:10.210
This expression here is the
expected time starting from
00:45:10.210 --> 00:45:13.640
the next state, but we also need
to account for the first
00:45:13.640 --> 00:45:15.840
transition, so we add 1.
00:45:15.840 --> 00:45:18.010
And this is going
to be our mu1.
00:45:21.150 --> 00:45:25.370
So once more we have a linear
equation that ties together
00:45:25.370 --> 00:45:27.930
the different mu's.
00:45:27.930 --> 00:45:32.070
And the equation starting from
state 4 in this case, of
00:45:32.070 --> 00:45:35.620
course is going to be simple,
starting from that state the
00:45:35.620 --> 00:45:38.280
expected number of steps it
takes you to get there for the
00:45:38.280 --> 00:45:42.220
first time is of course, 0
because you're already there.
00:45:42.220 --> 00:45:45.640
So for that state this is fine,
and for all the other
00:45:45.640 --> 00:45:48.280
states you get an equation
of this form.
00:45:48.280 --> 00:45:51.410
Now we're going to have an
equation for every state.
00:45:51.410 --> 00:45:53.760
It's a system of linear
equations, once more we can
00:45:53.760 --> 00:45:57.290
solve them, and this gives us
the expected times until our
00:45:57.290 --> 00:46:03.890
chain gets absorbed in
this absorbing state.
00:46:03.890 --> 00:46:06.650
And it's nice to know that
this system of equations
00:46:06.650 --> 00:46:09.510
always has a unique solution.
00:46:09.510 --> 00:46:13.220
OK so this was the expected
time to absorption.
00:46:13.220 --> 00:46:16.720
For this case where we had this
scene absorbing state.
00:46:16.720 --> 00:46:23.740
Suppose that we have our
transient states and that we
00:46:23.740 --> 00:46:27.020
have multiple recurrent
classes, or
00:46:27.020 --> 00:46:28.590
multiple absorbing states.
00:46:33.850 --> 00:46:37.470
Suppose you've got the
picture like this.
00:46:37.470 --> 00:46:42.040
And we want to calculate the
expected time until we get
00:46:42.040 --> 00:46:44.110
here or there.
00:46:44.110 --> 00:46:47.830
Expected time until we get
to an absorbing state.
00:46:47.830 --> 00:46:50.320
What's the trick?
00:46:50.320 --> 00:46:55.010
Well, we can lump both of these
states together and
00:46:55.010 --> 00:47:00.660
think of them as just one bad
state, one place for which
00:47:00.660 --> 00:47:03.730
we're interested in how long
it takes us to get there.
00:47:03.730 --> 00:47:10.250
So lump them as one state, and
accordingly kind of merge all
00:47:10.250 --> 00:47:11.570
of those probabilities.
00:47:11.570 --> 00:47:15.470
So starting from here, my
probability that the next I
00:47:15.470 --> 00:47:18.690
end up in this lump and they
get absorbed is going to be
00:47:18.690 --> 00:47:21.630
this probability plus
that probability.
00:47:21.630 --> 00:47:24.010
So we would change
that picture.
00:47:24.010 --> 00:47:30.080
Think of this as being
just one big state.
00:47:30.080 --> 00:47:36.080
And sort of add those two
probabilities together to come
00:47:36.080 --> 00:47:39.360
up with a single probability,
which is the probability that
00:47:39.360 --> 00:47:42.590
starting from here next
time I find myself at
00:47:42.590 --> 00:47:44.440
some absorbing state.
00:47:44.440 --> 00:47:48.510
So once you know how to deal
with a situation like this,
00:47:48.510 --> 00:47:52.030
you can also find expected times
to absorption for the
00:47:52.030 --> 00:47:55.190
case where you've got multiple
absorbing states.
00:47:55.190 --> 00:47:57.990
You just lump all of those
multiple absorbing states into
00:47:57.990 --> 00:47:59.240
a single one.
00:48:01.550 --> 00:48:05.040
Finally, there's a
kind of related
00:48:05.040 --> 00:48:09.600
quantity that's of interest.
00:48:09.600 --> 00:48:13.480
The question is almost the same
as in the previous slide,
00:48:13.480 --> 00:48:16.950
except that here we do not have
any absorbing states.
00:48:16.950 --> 00:48:20.335
Rather, we have a single
recurrent class of states.
00:48:25.320 --> 00:48:27.840
You start at some state i.
00:48:27.840 --> 00:48:31.780
You have a special state,
that is state s.
00:48:31.780 --> 00:48:35.600
And you ask the question, how
long is it going to take me
00:48:35.600 --> 00:48:39.040
until I get to s for
the first time?
00:48:39.040 --> 00:48:41.140
It's a single recurrent
class of states.
00:48:41.140 --> 00:48:43.730
So you know that the state keeps
circulating here and it
00:48:43.730 --> 00:48:46.610
keeps visiting all of
the possible states.
00:48:46.610 --> 00:48:49.510
So eventually this state
will be visited.
00:48:49.510 --> 00:48:52.070
How long does it take
for this to happen?
00:48:55.290 --> 00:48:55.425
Ok.
00:48:55.425 --> 00:48:58.820
So we're interested in how
long it takes for this to
00:48:58.820 --> 00:49:02.080
happen, how long it takes until
we get to s for the
00:49:02.080 --> 00:49:03.000
first time.
00:49:03.000 --> 00:49:06.480
And we don't care about what
happens afterwards.
00:49:06.480 --> 00:49:10.110
So we might as well change this
picture and remove the
00:49:10.110 --> 00:49:15.470
transitions out of s and to make
them self transitions.
00:49:15.470 --> 00:49:18.140
Is the answer going to change?
00:49:18.140 --> 00:49:19.280
No.
00:49:19.280 --> 00:49:22.890
The only thing that we changed
was what happens
00:49:22.890 --> 00:49:25.760
after you get to s.
00:49:25.760 --> 00:49:28.810
But what happens after you
get to s doesn't matter.
00:49:28.810 --> 00:49:31.730
The question we're dealing with
is how long does it take
00:49:31.730 --> 00:49:33.900
us to get to s.
00:49:33.900 --> 00:49:37.010
So essentially, it's after we
do this transformation --
00:49:37.010 --> 00:49:40.740
it's the same question as
before, what's the time it
00:49:40.740 --> 00:49:43.990
takes until eventually
we hit this state.
00:49:43.990 --> 00:49:46.390
And it's now in this new
picture, this state is an
00:49:46.390 --> 00:49:48.410
absorbing state.
00:49:48.410 --> 00:49:50.710
Or you can just think from
first principles.
00:49:50.710 --> 00:49:55.720
Starting from the state itself,
s, it takes you 0 time
00:49:55.720 --> 00:49:57.430
steps until you get to s.
00:49:57.430 --> 00:50:01.610
Starting from anywhere else,
you need one transition and
00:50:01.610 --> 00:50:05.050
then after the first transition
you find yourself
00:50:05.050 --> 00:50:09.560
at state j with probability Pij
and from then on you are
00:50:09.560 --> 00:50:13.350
going to take expected time
Tj until you get to that
00:50:13.350 --> 00:50:14.710
terminal state s.
00:50:14.710 --> 00:50:17.340
So once more these equations
have a unique solution, you
00:50:17.340 --> 00:50:19.510
can solve them and
find the answer.
00:50:19.510 --> 00:50:22.530
And finally, there's a related
question, which is the mean
00:50:22.530 --> 00:50:24.210
recurrence time of s.
00:50:24.210 --> 00:50:30.490
In that question you start
at s, the chain will move
00:50:30.490 --> 00:50:34.090
randomly, and you ask how long
is it going to take until I
00:50:34.090 --> 00:50:37.160
come back to s for
the next time.
00:50:37.160 --> 00:50:38.390
So notice the difference.
00:50:38.390 --> 00:50:42.670
Here we're talking the first
time after time 0, whereas
00:50:42.670 --> 00:50:45.500
here it's just the first
time anywhere.
00:50:45.500 --> 00:50:51.250
So here if you start from
s, Ts* is not 0.
00:50:51.250 --> 00:50:54.450
You want to do at least one
transition and that's how long
00:50:54.450 --> 00:50:57.360
it's going to take me until
it gets back to s.
00:50:57.360 --> 00:51:00.900
Well, how long does it take
me until I get back to s?
00:51:00.900 --> 00:51:06.060
I do my first transition, and
then after my first transition
00:51:06.060 --> 00:51:11.920
I calculate the expected time
from the next state how long
00:51:11.920 --> 00:51:15.350
it's going to take me until
I come back to s.
00:51:15.350 --> 00:51:20.200
So all of these equations that I
wrote down, they all kind of
00:51:20.200 --> 00:51:22.500
look the same.
00:51:22.500 --> 00:51:23.620
But they are different.
00:51:23.620 --> 00:51:27.210
So you can either memorize all
of these equations, or instead
00:51:27.210 --> 00:51:30.580
what's better is to just
to get the basic idea.
00:51:30.580 --> 00:51:33.140
That is, to calculate
probabilities or expected
00:51:33.140 --> 00:51:35.830
values you use the total
probability or total
00:51:35.830 --> 00:51:38.260
expectation theorem and
conditional the first
00:51:38.260 --> 00:51:40.810
transition and take
it from there.
00:51:40.810 --> 00:51:43.230
So you're going to get a little
bit of practice with
00:51:43.230 --> 00:51:47.276
these skills in recitation
tomorrow, and of course it's
00:51:47.276 --> 00:51:48.730
in your problem set as well.