WEBVTT
00:00:00.040 --> 00:00:02.460
The following content is
provided under a Creative
00:00:02.460 --> 00:00:03.870
Commons license.
00:00:03.870 --> 00:00:06.910
Your support will help MIT
OpenCourseWare continue to
00:00:06.910 --> 00:00:08.700
offer high-quality, educational
00:00:08.700 --> 00:00:10.560
resources for free.
00:00:10.560 --> 00:00:13.460
To make a donation or view
additional materials from
00:00:13.460 --> 00:00:19.290
hundreds of MIT courses, visit
MIT OpenCourseWare at
00:00:19.290 --> 00:00:21.764
ocw.mit.edu.
00:00:21.764 --> 00:00:22.590
PROFESSOR: All right.
00:00:22.590 --> 00:00:26.070
So today, we're going to start
by taking stock of what we
00:00:26.070 --> 00:00:27.670
discussed last time, review the
00:00:27.670 --> 00:00:29.810
definition of Markov chains.
00:00:29.810 --> 00:00:32.400
And then most of the lecture,
we're going to concentrate on
00:00:32.400 --> 00:00:34.520
their steady-state behavior.
00:00:34.520 --> 00:00:38.860
Meaning, we're going to look at
what does a Markov chain do
00:00:38.860 --> 00:00:40.820
if it has run for a long time.
00:00:40.820 --> 00:00:42.860
What can we say about
the probabilities of
00:00:42.860 --> 00:00:44.610
the different states?
00:00:44.610 --> 00:00:47.390
So what I would like to repeat
is a statement I made last
00:00:47.390 --> 00:00:49.730
time that Markov chains
is a very, very
00:00:49.730 --> 00:00:51.910
useful class of models.
00:00:51.910 --> 00:00:56.820
Pretty much anything in
the real world can be
00:00:56.820 --> 00:01:01.380
approximately modeled by a
Markov chain provided that you
00:01:01.380 --> 00:01:05.019
set your states in
the proper way.
00:01:05.019 --> 00:01:06.890
So we're going to see
some examples.
00:01:06.890 --> 00:01:09.310
You're going to see more
examples in the problems
00:01:09.310 --> 00:01:11.140
you're going to do in homework
and recitation.
00:01:13.880 --> 00:01:15.810
On the other hand, we're
not going to go
00:01:15.810 --> 00:01:17.830
too deep into examples.
00:01:17.830 --> 00:01:20.310
Rather, we're going to develop
the general methodology.
00:02:11.440 --> 00:02:12.690
OK.
00:02:19.310 --> 00:02:19.730
All right.
00:02:19.730 --> 00:02:22.280
Markov models can be
pretty general.
00:02:22.280 --> 00:02:24.670
They can run in continuous
or discrete time.
00:02:24.670 --> 00:02:27.540
They can have continuous or
discrete state spaces.
00:02:27.540 --> 00:02:30.280
In this class, we're going to
stick just to the case where
00:02:30.280 --> 00:02:33.880
the state space is discrete and
time is discrete because
00:02:33.880 --> 00:02:35.570
this is the simplest case.
00:02:35.570 --> 00:02:38.760
And also, it's the one where
you build your intuition
00:02:38.760 --> 00:02:40.830
before going to more
general cases
00:02:40.830 --> 00:02:42.840
perhaps in other classes.
00:02:42.840 --> 00:02:47.810
So the state is discrete
and finite.
00:02:47.810 --> 00:02:50.130
There's a finite number
of states.
00:02:50.130 --> 00:02:53.540
At any point in time, the
process is sitting on one of
00:02:53.540 --> 00:02:54.580
those states.
00:02:54.580 --> 00:03:00.520
Time is discrete, so at each
unit of time, somebody
00:03:00.520 --> 00:03:03.000
whistles and then
the state jumps.
00:03:03.000 --> 00:03:06.110
And when it jumps, it can either
land in the same place,
00:03:06.110 --> 00:03:08.180
or it can land somewhere else.
00:03:08.180 --> 00:03:10.740
And the evolution of the
process is described by
00:03:10.740 --> 00:03:13.130
transition probabilities.
00:03:13.130 --> 00:03:16.740
Pij is the probability that the
next state is j given that
00:03:16.740 --> 00:03:18.630
the current state is i.
00:03:18.630 --> 00:03:21.640
And the most important property
that the Markov chain
00:03:21.640 --> 00:03:24.690
has, the definition of a
Markov chain or Markov
00:03:24.690 --> 00:03:28.880
process, is that this
probability, Pij, is the same
00:03:28.880 --> 00:03:31.490
every time that you
land at state i --
00:03:31.490 --> 00:03:36.320
no matter how you got there
and also no matter
00:03:36.320 --> 00:03:38.000
what time it is.
00:03:38.000 --> 00:03:41.650
So the model we have is time
homogeneous, which basically
00:03:41.650 --> 00:03:44.630
means that those transition
probabilities are the same at
00:03:44.630 --> 00:03:45.720
every time.
00:03:45.720 --> 00:03:49.320
So the model is time invariant
in that sense.
00:03:49.320 --> 00:03:52.670
So we're interested in what the
chain or the process is
00:03:52.670 --> 00:03:54.780
going to do in the longer run.
00:03:54.780 --> 00:03:57.060
So we're interested, let's say,
in the probability that
00:03:57.060 --> 00:04:00.760
starting at a certain state, n
times steps later, we find
00:04:00.760 --> 00:04:03.760
ourselves at some particular
state j.
00:04:03.760 --> 00:04:06.350
Fortunately, we can calculate
those probabilities
00:04:06.350 --> 00:04:07.660
recursively.
00:04:07.660 --> 00:04:12.990
Of course, at the first time 1,
the probability of being 1
00:04:12.990 --> 00:04:16.950
time later at state j given that
we are right now at state
00:04:16.950 --> 00:04:20.910
i, by definition, this is just
the transition probabilities.
00:04:20.910 --> 00:04:26.000
So by knowing these, we can
start a recursion that tells
00:04:26.000 --> 00:04:27.730
us the transition probabilities
00:04:27.730 --> 00:04:31.430
for more than n steps.
00:04:31.430 --> 00:04:32.865
This recursion, it's
a formula.
00:04:32.865 --> 00:04:34.000
It's always true.
00:04:34.000 --> 00:04:36.860
You can copy it or
memorize it.
00:04:36.860 --> 00:04:40.460
But there is a big idea behind
that formula that you should
00:04:40.460 --> 00:04:41.480
keep in mind.
00:04:41.480 --> 00:04:43.750
And basically, the divide
and conquer idea.
00:04:43.750 --> 00:04:47.530
It's an application of the
total probability law.
00:04:47.530 --> 00:04:49.080
So let's fix i.
00:04:49.080 --> 00:04:52.360
The probability that you find
yourself at state j, you break
00:04:52.360 --> 00:04:55.770
it up into the probabilities of
the different ways that you
00:04:55.770 --> 00:04:57.850
can get to state j.
00:04:57.850 --> 00:04:59.240
What are those different ways?
00:04:59.240 --> 00:05:02.460
The different ways are the
different states k at which
00:05:02.460 --> 00:05:05.220
you might find yourself
the previous time.
00:05:05.220 --> 00:05:08.400
So with some probability, with
this probability, you find
00:05:08.400 --> 00:05:11.070
yourself at state k
the previous time.
00:05:11.070 --> 00:05:13.350
And then with probability
Pkj, you make a
00:05:13.350 --> 00:05:15.030
transition to state j.
00:05:15.030 --> 00:05:19.000
So this is a possible scenario
that takes you to state j
00:05:19.000 --> 00:05:21.090
after n transitions.
00:05:21.090 --> 00:05:25.350
And by summing over all the k's,
then we have considered
00:05:25.350 --> 00:05:28.430
all the possible scenarios.
00:05:28.430 --> 00:05:31.590
Now, before we move to the more
serious stuff, let's do a
00:05:31.590 --> 00:05:38.420
little bit of warm up to get
a handle on how we use
00:05:38.420 --> 00:05:42.170
transition probabilities to
calculate more general
00:05:42.170 --> 00:05:45.250
probabilities, then talk about
some structural properties of
00:05:45.250 --> 00:05:47.980
Markov chains, and then
eventually get to the main
00:05:47.980 --> 00:05:51.310
business of today, which is
a steady-state behavior.
00:05:51.310 --> 00:05:56.380
So somebody gives you this
chain, and our convention is
00:05:56.380 --> 00:06:00.340
that those arcs that are not
shown here corresponds to 0
00:06:00.340 --> 00:06:01.510
probabilities.
00:06:01.510 --> 00:06:05.760
And each one of the arcs that's
shown has a non-zero
00:06:05.760 --> 00:06:09.060
probability, and somebody
gives it to us.
00:06:09.060 --> 00:06:11.650
Suppose that the chain
starts at state 1.
00:06:11.650 --> 00:06:15.720
We want to calculate the
probability that it follows
00:06:15.720 --> 00:06:16.830
this particular path.
00:06:16.830 --> 00:06:20.620
That is, it goes to 2,
then to 6, then to 7.
00:06:20.620 --> 00:06:22.920
How do we calculate the
probability of a particular
00:06:22.920 --> 00:06:24.280
trajectory?
00:06:24.280 --> 00:06:26.700
Well, this is the
probability--
00:06:26.700 --> 00:06:30.980
so it's the probability of the
trajectory from 1 that you go
00:06:30.980 --> 00:06:34.480
to 2, then to 6, then to 7.
00:06:34.480 --> 00:06:38.020
So the probability of this
trajectory is we use the
00:06:38.020 --> 00:06:39.270
multiplication rule.
00:06:39.270 --> 00:06:42.140
The probability of several
things happening is the
00:06:42.140 --> 00:06:44.700
probability that the first
thing happens, which is a
00:06:44.700 --> 00:06:47.140
transition from 1 to 2.
00:06:47.140 --> 00:06:53.190
And then given that we are at
state 2, we multiply with a
00:06:53.190 --> 00:06:57.120
conditional probability that
the next event happens.
00:06:57.120 --> 00:07:02.370
That is, that X2 is equal to 6
given that right now, we are
00:07:02.370 --> 00:07:03.570
at state 1.
00:07:03.570 --> 00:07:07.760
And that conditional probability
is just P26.
00:07:07.760 --> 00:07:09.840
And notice that this conditional
probability
00:07:09.840 --> 00:07:13.380
applies no matter how
we got to state 2.
00:07:13.380 --> 00:07:15.340
This is the Markov assumption.
00:07:15.340 --> 00:07:17.970
So we don't care about the
fact that we came in in a
00:07:17.970 --> 00:07:19.120
particular way.
00:07:19.120 --> 00:07:22.370
Given that we came in here, this
probability P26, that the
00:07:22.370 --> 00:07:24.640
next transition takes us to 6.
00:07:24.640 --> 00:07:28.620
And then given that all that
stuff happened, so given that
00:07:28.620 --> 00:07:32.060
right now, we are at state 6,
we need to multiply with a
00:07:32.060 --> 00:07:34.680
conditional probability that the
next transition takes us
00:07:34.680 --> 00:07:36.020
to state 7.
00:07:36.020 --> 00:07:39.920
And this is just the P67.
00:07:39.920 --> 00:07:44.140
So to find the probability
of following a specific
00:07:44.140 --> 00:07:49.280
trajectory, you just multiply
the transition probabilities
00:07:49.280 --> 00:07:50.885
along the particular
trajectory.
00:07:54.490 --> 00:07:58.150
Now, if you want to calculate
something else, such as for
00:07:58.150 --> 00:08:02.100
example, the probability that
4 time steps later, I find
00:08:02.100 --> 00:08:06.250
myself at state 7 given that
they started, let's say, at
00:08:06.250 --> 00:08:07.680
this state.
00:08:07.680 --> 00:08:09.920
How do you calculate
this probability?
00:08:09.920 --> 00:08:15.350
One way is to use the recursion
for the Rijs that we
00:08:15.350 --> 00:08:17.990
know that it is always valid.
00:08:17.990 --> 00:08:20.970
But for short and simple
examples, and with a small
00:08:20.970 --> 00:08:24.890
time horizon, perhaps you can do
this in a brute force way.
00:08:24.890 --> 00:08:27.100
What would be the
brute force way?
00:08:27.100 --> 00:08:30.410
This is the event that 4 time
steps later, I find
00:08:30.410 --> 00:08:32.210
myself at state 7.
00:08:32.210 --> 00:08:36.350
This event can happen
in various ways.
00:08:36.350 --> 00:08:41.070
So we can take stock of all the
different ways, and write
00:08:41.070 --> 00:08:42.760
down their probabilities.
00:08:42.760 --> 00:08:44.800
So starting from 2.
00:08:44.800 --> 00:08:49.870
One possibility is to follow
this trajectory, 1 transition,
00:08:49.870 --> 00:08:53.790
2 transitions, 3 transitions,
4 transitions.
00:08:53.790 --> 00:08:55.810
And that takes me to state 7.
00:08:55.810 --> 00:08:57.760
What's the probability
of this trajectory?
00:08:57.760 --> 00:09:05.160
It's P26 times P67 times
P76 and then times P67.
00:09:05.160 --> 00:09:08.000
So this is a probability of a
particular trajectory that
00:09:08.000 --> 00:09:11.040
takes you to state 7
after 4 time steps.
00:09:11.040 --> 00:09:14.790
But there's other trajectories
as well.
00:09:14.790 --> 00:09:16.300
What could be it?
00:09:16.300 --> 00:09:23.470
I might start from state 2, go
to state 6, stay at state 6,
00:09:23.470 --> 00:09:26.450
stay at state 6 once more.
00:09:26.450 --> 00:09:31.670
And then from state
6, go to state 7.
00:09:31.670 --> 00:09:36.800
And so there must be one more.
00:09:36.800 --> 00:09:38.470
What's the other one?
00:09:38.470 --> 00:09:45.620
I guess I could go 1, 2, 6, 7.
00:09:45.620 --> 00:09:45.690
OK.
00:09:45.690 --> 00:09:48.012
That's the other trajectory.
00:09:48.012 --> 00:10:01.850
Plus P21 times P12 times
P26 and times P67.
00:10:01.850 --> 00:10:05.580
So the transition probability,
the overall probability of
00:10:05.580 --> 00:10:09.490
finding ourselves at state 7,
is broken down as the sum of
00:10:09.490 --> 00:10:12.580
the probabilities of all the
different ways that I can get
00:10:12.580 --> 00:10:16.000
to state 7 in exactly 4 steps.
00:10:16.000 --> 00:10:19.460
So we could always do that
without knowing much about
00:10:19.460 --> 00:10:21.750
Markov chains or the general
formula for the
00:10:21.750 --> 00:10:24.490
Rij's that we had.
00:10:24.490 --> 00:10:26.370
What's the trouble with
this procedure?
00:10:26.370 --> 00:10:29.190
The trouble with this procedure
is that the number
00:10:29.190 --> 00:10:34.450
of possible trajectories becomes
quite large if this
00:10:34.450 --> 00:10:37.550
index is a little bigger.
00:10:37.550 --> 00:10:41.510
If this 4 was 100, and you
ask how many different
00:10:41.510 --> 00:10:45.240
trajectories of length 100 are
there to take me from here to
00:10:45.240 --> 00:10:48.170
there, that number of
trajectories would be huge.
00:10:48.170 --> 00:10:51.120
It grows exponentially with
the time horizon.
00:10:51.120 --> 00:10:55.110
And this kind of calculation
would be impossible.
00:10:55.110 --> 00:10:58.310
The basic equation, the
recursion that have for the
00:10:58.310 --> 00:11:01.650
Rij's is basically a clever
way of organizing this
00:11:01.650 --> 00:11:04.870
computation so that the amount
of computation that you do is
00:11:04.870 --> 00:11:06.960
not exponential in
the time horizon.
00:11:06.960 --> 00:11:11.100
Rather, it's sort of linear
with the time horizon.
00:11:11.100 --> 00:11:14.590
For each time step you need in
the time horizon, you just
00:11:14.590 --> 00:11:17.260
keep repeating the same
iteration over and over.
00:11:20.460 --> 00:11:20.565
OK.
00:11:20.565 --> 00:11:24.940
Now, the other thing that we
discussed last time, briefly,
00:11:24.940 --> 00:11:28.530
was a classification of the
different states of the Markov
00:11:28.530 --> 00:11:31.510
chain in two different types.
00:11:31.510 --> 00:11:37.020
A Markov chain, in general, has
states that are recurrent,
00:11:37.020 --> 00:11:39.670
which means that from a
recurrent state, I can go
00:11:39.670 --> 00:11:41.000
somewhere else.
00:11:41.000 --> 00:11:44.910
But from that somewhere else,
there's always some way of
00:11:44.910 --> 00:11:46.040
coming back.
00:11:46.040 --> 00:11:49.910
So if you have a chain of this
form, no matter where you go,
00:11:49.910 --> 00:11:52.500
no matter where you start,
you can always come
00:11:52.500 --> 00:11:54.540
back where you started.
00:11:54.540 --> 00:11:56.800
States of this kind are
called recurrent.
00:11:56.800 --> 00:12:00.320
On the other hand, if you have
a few states all this kind, a
00:12:00.320 --> 00:12:04.560
transition of this type, then
these states are transient in
00:12:04.560 --> 00:12:07.560
the sense that from those
states, it's possible to go
00:12:07.560 --> 00:12:11.520
somewhere else from which place
there's no way to come
00:12:11.520 --> 00:12:13.580
back where you started.
00:12:13.580 --> 00:12:18.370
The general structure of a
Markov chain is basically a
00:12:18.370 --> 00:12:20.660
collection of transient
states.
00:12:20.660 --> 00:12:23.960
You're certain that you are
going to leave the transient
00:12:23.960 --> 00:12:27.370
states eventually.
00:12:27.370 --> 00:12:30.300
And after you leave the
transient states, you enter
00:12:30.300 --> 00:12:33.570
into a class of states in
which you are trapped.
00:12:33.570 --> 00:12:35.890
You are trapped if you
get inside here.
00:12:35.890 --> 00:12:39.040
You are trapped if you
get inside there.
00:12:39.040 --> 00:12:41.270
This is a recurrent
class of states.
00:12:41.270 --> 00:12:43.210
From any state, you can
get to any other
00:12:43.210 --> 00:12:44.380
state within this class.
00:12:44.380 --> 00:12:46.100
That's another recurrent
class.
00:12:46.100 --> 00:12:49.380
From any state inside here,
you can get anywhere else
00:12:49.380 --> 00:12:50.680
inside that class.
00:12:50.680 --> 00:12:52.930
But these 2 classes, you
do not communicate.
00:12:52.930 --> 00:12:56.310
If you start here, there's
no way to get there.
00:12:56.310 --> 00:12:59.820
If you have 2 recurrent classes,
then it's clear that
00:12:59.820 --> 00:13:02.310
the initial conditions
of your Markov chain
00:13:02.310 --> 00:13:04.130
matter in the long run.
00:13:04.130 --> 00:13:07.450
If you start here, you will be
stuck inside here for the long
00:13:07.450 --> 00:13:09.140
run and similarly about here.
00:13:09.140 --> 00:13:11.680
So the initial conditions
do make a difference.
00:13:11.680 --> 00:13:14.720
On the other hand, if this class
was not here and you
00:13:14.720 --> 00:13:17.300
only had that class, what would
happen to the chain?
00:13:17.300 --> 00:13:18.480
Let's say you start here.
00:13:18.480 --> 00:13:19.490
You move around.
00:13:19.490 --> 00:13:21.730
At some point, you make
that transition.
00:13:21.730 --> 00:13:23.260
You get stuck in here.
00:13:23.260 --> 00:13:26.410
And inside here, you keep
circulating, because of the
00:13:26.410 --> 00:13:30.210
randomness, you keep visiting
all states over and over.
00:13:30.210 --> 00:13:35.800
And hopefully or possibly, in
the long run, it doesn't
00:13:35.800 --> 00:13:39.610
matter exactly what time it is
or where you started, but the
00:13:39.610 --> 00:13:43.860
probability of being at that
particular state is the same
00:13:43.860 --> 00:13:46.310
no matter what the initial
condition was.
00:13:46.310 --> 00:13:48.820
So with a single recurrent
class, we hope that the
00:13:48.820 --> 00:13:50.750
initial conditions
do not matter.
00:13:50.750 --> 00:13:55.780
With 2 or more recurrent
classes, initial conditions
00:13:55.780 --> 00:13:58.660
will definitely matter.
00:13:58.660 --> 00:14:03.620
So how many recurrent classes we
have is something that has
00:14:03.620 --> 00:14:06.790
to do with the long-term
behavior of the chain and the
00:14:06.790 --> 00:14:09.790
extent to which initial
conditions matter.
00:14:09.790 --> 00:14:16.000
Another way that initial
conditions may matter is if a
00:14:16.000 --> 00:14:19.360
chain has a periodic
structure.
00:14:19.360 --> 00:14:21.990
There are many ways of
defining periodicity.
00:14:21.990 --> 00:14:25.240
The one that I find sort of the
most intuitive and with
00:14:25.240 --> 00:14:27.410
the least amount
of mathematical
00:14:27.410 --> 00:14:29.670
symbols is the following.
00:14:29.670 --> 00:14:34.630
The state space of a chain is
said to be periodic if you can
00:14:34.630 --> 00:14:39.510
lump the states into a number
of clusters called
00:14:39.510 --> 00:14:42.550
d clusters or groups.
00:14:42.550 --> 00:14:45.980
And the transition diagram has
the property that from a
00:14:45.980 --> 00:14:48.860
cluster, you always
make a transition
00:14:48.860 --> 00:14:50.870
into the next cluster.
00:14:50.870 --> 00:14:52.860
So here d is equal to 2.
00:14:52.860 --> 00:14:55.570
We have two subsets of
the state space.
00:14:55.570 --> 00:14:58.130
Whenever we're here, next
time we'll be there.
00:14:58.130 --> 00:15:01.080
Whenever we're here, next
time we will be there.
00:15:01.080 --> 00:15:03.830
So this chain has a periodic
structure.
00:15:03.830 --> 00:15:05.880
There may be still
some randomness.
00:15:05.880 --> 00:15:10.270
When I jump from here to here,
the state to which I jump may
00:15:10.270 --> 00:15:14.490
be random, but I'm sure that I'm
going to be inside here.
00:15:14.490 --> 00:15:17.610
And then next time, I will be
sure that I'm inside here.
00:15:17.610 --> 00:15:20.310
This would be a structure of a
diagram in which we have a
00:15:20.310 --> 00:15:21.410
period of 3.
00:15:21.410 --> 00:15:25.540
If you start in this lump, you
know that the next time, you
00:15:25.540 --> 00:15:27.480
would be in a state
inside here.
00:15:27.480 --> 00:15:30.830
Next time, you'll be in a state
inside here, and so on.
00:15:30.830 --> 00:15:35.220
So these chains certainly have
a periodic structure.
00:15:35.220 --> 00:15:37.860
And that periodicity
gets maintained.
00:15:37.860 --> 00:15:42.500
If I start, let's say, at this
lump, at even times,
00:15:42.500 --> 00:15:44.500
I'm sure I'm here.
00:15:44.500 --> 00:15:47.660
At odd times, I'm
sure I am here.
00:15:47.660 --> 00:15:51.600
So the exact time does matter
in determining the
00:15:51.600 --> 00:15:54.660
probabilities of the
different states.
00:15:54.660 --> 00:15:57.140
And in particular, the
probability of being at the
00:15:57.140 --> 00:16:00.500
particular state cannot convert
to a state value.
00:16:00.500 --> 00:16:03.410
The probability of being at the
state inside here is going
00:16:03.410 --> 00:16:06.300
to be 0 for all times.
00:16:06.300 --> 00:16:08.290
In general, it's going
to be some positive
00:16:08.290 --> 00:16:10.160
number for even times.
00:16:10.160 --> 00:16:13.810
So it goes 0 positive, zero,
positive, 0 positive.
00:16:13.810 --> 00:16:15.370
Doesn't settle to anything.
00:16:15.370 --> 00:16:19.860
So when we have periodicity,
we do not expect the states
00:16:19.860 --> 00:16:22.600
probabilities to converge to
something, but rather, we
00:16:22.600 --> 00:16:24.580
expect them to oscillate.
00:16:24.580 --> 00:16:26.830
Now, how can we tell whether
a Markov chain
00:16:26.830 --> 00:16:29.920
is periodic or not?
00:16:29.920 --> 00:16:33.900
There are systematic ways of
doing it, but usually with the
00:16:33.900 --> 00:16:36.560
types of examples we see in this
class, we just eyeball
00:16:36.560 --> 00:16:39.890
the chain, and we tell whether
it's periodic or not.
00:16:39.890 --> 00:16:45.240
So is this chain down here, is
it the periodic one or not?
00:16:45.240 --> 00:16:48.870
How many people think
it's periodic?
00:16:48.870 --> 00:16:50.680
No one.
00:16:50.680 --> 00:16:51.200
One.
00:16:51.200 --> 00:16:54.070
How many people think
it's not periodic?
00:16:54.070 --> 00:16:54.560
OK.
00:16:54.560 --> 00:16:56.270
Not periodic?
00:16:56.270 --> 00:16:57.570
Let's see.
00:16:57.570 --> 00:16:59.490
Let me do some drawing here.
00:17:03.230 --> 00:17:04.119
OK.
00:17:04.119 --> 00:17:05.369
Is it periodic?
00:17:07.856 --> 00:17:09.140
It is.
00:17:09.140 --> 00:17:14.180
From a red state, you can only
get to a white state.
00:17:14.180 --> 00:17:17.849
And from a white state, you can
only get to a red state.
00:17:17.849 --> 00:17:20.660
So this chain, even though it's
not apparent from the
00:17:20.660 --> 00:17:24.589
picture, actually has
this structure.
00:17:24.589 --> 00:17:28.600
We can group the states into red
states and white states.
00:17:28.600 --> 00:17:32.810
And from reds, we always go to
a white, and from a white, we
00:17:32.810 --> 00:17:34.500
always go to a red.
00:17:34.500 --> 00:17:36.540
So this tells you
that sometimes
00:17:36.540 --> 00:17:38.680
eyeballing is not as easy.
00:17:38.680 --> 00:17:40.810
If you have lots and lots of
states, you might have some
00:17:40.810 --> 00:17:43.280
trouble doing this exercise.
00:17:43.280 --> 00:17:47.230
On the other hand, something
very useful to know.
00:17:47.230 --> 00:17:50.270
Sometimes it's extremely
easy to tell that the
00:17:50.270 --> 00:17:52.360
chain is not periodic.
00:17:52.360 --> 00:17:53.770
What's that case?
00:17:53.770 --> 00:17:58.600
Suppose that your chain has a
self-transition somewhere.
00:17:58.600 --> 00:18:01.670
Then automatically,
you know that your
00:18:01.670 --> 00:18:04.660
chain is not periodic.
00:18:04.660 --> 00:18:08.560
So remember, the definition of
periodicity requires that if
00:18:08.560 --> 00:18:12.230
you are in a certain group of
states, next time, you will be
00:18:12.230 --> 00:18:14.380
in a different group.
00:18:14.380 --> 00:18:16.450
But if you have
self-transitions, that
00:18:16.450 --> 00:18:17.600
property is not true.
00:18:17.600 --> 00:18:20.790
If you have a possible
self-transition, it's possible
00:18:20.790 --> 00:18:24.530
that you stay inside your own
group for the next time step.
00:18:24.530 --> 00:18:29.560
So whenever you have a
self-transition, this implies
00:18:29.560 --> 00:18:31.440
that the chain is
not periodic.
00:18:34.280 --> 00:18:39.240
And usually that's the simplest
and easy way that we
00:18:39.240 --> 00:18:44.210
can tell most of the time that
the chain is not periodic.
00:18:44.210 --> 00:18:49.110
So now, we come to the big topic
of today, the central
00:18:49.110 --> 00:18:53.080
topic, which is the question
about what does the chain do
00:18:53.080 --> 00:18:55.550
in the long run.
00:18:55.550 --> 00:19:00.060
The question we are asking and
which we motivated last time
00:19:00.060 --> 00:19:02.510
by looking at an example.
00:19:02.510 --> 00:19:05.790
It's something that did happen
in our example of last time.
00:19:05.790 --> 00:19:08.000
So we're asking whether
this happens for
00:19:08.000 --> 00:19:09.440
every Markov chain.
00:19:09.440 --> 00:19:12.400
We're asking the question
whether the probability of
00:19:12.400 --> 00:19:18.250
being at state j at some
time n settles to a
00:19:18.250 --> 00:19:20.350
steady-state value.
00:19:20.350 --> 00:19:22.900
Let's call it pi sub j.
00:19:22.900 --> 00:19:26.960
That these were asking whether
this quantity has a limit as n
00:19:26.960 --> 00:19:29.400
goes to infinity, so that
we can talk about the
00:19:29.400 --> 00:19:32.300
steady-state probability
of state j.
00:19:32.300 --> 00:19:36.060
And furthermore, we asked
whether the steady-state
00:19:36.060 --> 00:19:38.900
probability of that state
does not depend
00:19:38.900 --> 00:19:40.800
on the initial state.
00:19:40.800 --> 00:19:44.110
In other words, after the chain
runs for a long, long
00:19:44.110 --> 00:19:48.620
time, it doesn't matter exactly
what time it is, and
00:19:48.620 --> 00:19:51.990
it doesn't matter where the
chain started from.
00:19:51.990 --> 00:19:54.860
You can tell me the probability
that the state is
00:19:54.860 --> 00:19:58.700
a particular j is approximately
the steady-state
00:19:58.700 --> 00:20:00.450
probability pi sub j.
00:20:00.450 --> 00:20:03.470
It doesn't matter exactly what
time it is as long as you tell
00:20:03.470 --> 00:20:06.900
me that a lot of time
has elapsed so
00:20:06.900 --> 00:20:09.520
that n is a big number.
00:20:09.520 --> 00:20:11.150
So this is the question.
00:20:11.150 --> 00:20:14.210
We have seen examples, and we
understand that this is not
00:20:14.210 --> 00:20:16.600
going to be the case always.
00:20:16.600 --> 00:20:19.880
For example, as I just
discussed, if we have 2
00:20:19.880 --> 00:20:23.850
recurrent classes, where
we start does matter.
00:20:23.850 --> 00:20:28.060
The probability pi(j) of being
in that state j is going to be
00:20:28.060 --> 00:20:32.650
0 if we start here, but it would
be something positive if
00:20:32.650 --> 00:20:34.710
we were to start in that lump.
00:20:34.710 --> 00:20:37.650
So the initial state does matter
if we have multiple
00:20:37.650 --> 00:20:39.690
recurrent classes.
00:20:39.690 --> 00:20:45.590
But if we have only a single
class of recurrent states from
00:20:45.590 --> 00:20:48.780
each one of which you can get
to any other one, then we
00:20:48.780 --> 00:20:49.980
don't have that problem.
00:20:49.980 --> 00:20:53.010
Then we expect initial
conditions to be forgotten.
00:20:53.010 --> 00:20:55.100
So that's one condition
that we need.
00:20:58.960 --> 00:21:01.670
And then the other condition
that we need is that the chain
00:21:01.670 --> 00:21:02.930
is not periodic.
00:21:02.930 --> 00:21:07.580
If the chain is periodic, then
these Rij's do not converge.
00:21:07.580 --> 00:21:09.510
They keep oscillating.
00:21:09.510 --> 00:21:13.190
If we do not have periodicity,
then there is hope that we
00:21:13.190 --> 00:21:16.070
will get the convergence
that we need.
00:21:16.070 --> 00:21:19.210
It turns out this is the big
theory of Markov chains-- the
00:21:19.210 --> 00:21:20.900
steady-state convergence
theorem.
00:21:20.900 --> 00:21:26.470
It turns out that yes, the
rijs do converge to a
00:21:26.470 --> 00:21:29.510
steady-state limit, which
we call a steady-state
00:21:29.510 --> 00:21:35.180
probability as long as these two
conditions are satisfied.
00:21:35.180 --> 00:21:37.500
We're not going to prove
this theorem.
00:21:37.500 --> 00:21:41.790
If you're really interested, the
end of chapter exercises
00:21:41.790 --> 00:21:45.670
basically walk you through a
proof of this result, but it's
00:21:45.670 --> 00:21:49.470
probably a little too much for
doing it in this class.
00:21:49.470 --> 00:21:52.140
What is the intuitive idea
behind this theorem?
00:21:52.140 --> 00:21:52.860
Let's see.
00:21:52.860 --> 00:21:56.830
Let's think intuitively
as to why the initial
00:21:56.830 --> 00:21:59.010
state doesn't matter.
00:21:59.010 --> 00:22:02.870
Think of two copies of the chain
that starts at different
00:22:02.870 --> 00:22:06.430
initial states, and the
state moves randomly.
00:22:06.430 --> 00:22:09.390
As the state moves randomly
starting from the two initial
00:22:09.390 --> 00:22:11.900
states a random trajectory.
00:22:11.900 --> 00:22:15.610
as long as you have a single
recurrent class at some point,
00:22:15.610 --> 00:22:19.170
and you don't have periodicity
at some point, those states,
00:22:19.170 --> 00:22:22.610
those two trajectories,
are going to collide.
00:22:22.610 --> 00:22:25.650
Just because there's enough
randomness there.
00:22:25.650 --> 00:22:28.830
Even though we started from
different places, the state is
00:22:28.830 --> 00:22:30.490
going to be the same.
00:22:30.490 --> 00:22:33.540
After the state becomes the
same, then the future of these
00:22:33.540 --> 00:22:37.100
trajectories, probabilistically,
is the same
00:22:37.100 --> 00:22:39.800
because they both started
at the same state.
00:22:39.800 --> 00:22:42.540
So this means that the
initial conditions
00:22:42.540 --> 00:22:45.330
stopped having any influence.
00:22:45.330 --> 00:22:50.040
That's sort of the high-level
idea of why the initial state
00:22:50.040 --> 00:22:50.820
gets forgotten.
00:22:50.820 --> 00:22:53.780
Even if you started at different
initial states, at
00:22:53.780 --> 00:22:57.030
some time, you may find yourself
to be in the same
00:22:57.030 --> 00:22:59.120
state as the other trajectory.
00:22:59.120 --> 00:23:05.370
And once that happens, your
initial conditions cannot have
00:23:05.370 --> 00:23:08.210
any effect into the future.
00:23:08.210 --> 00:23:09.230
All right.
00:23:09.230 --> 00:23:15.650
So let's see how we might
calculate those steady-state
00:23:15.650 --> 00:23:16.870
probabilities.
00:23:16.870 --> 00:23:19.850
The way we calculate the
steady-state probabilities is
00:23:19.850 --> 00:23:24.230
by taking this recursion, which
is always true for the
00:23:24.230 --> 00:23:27.150
end-step transition
probabilities, and take the
00:23:27.150 --> 00:23:29.400
limit of both sides.
00:23:29.400 --> 00:23:32.970
The limit of this side is the
steady-state probability of
00:23:32.970 --> 00:23:36.010
state j, which is pi sub j.
00:23:36.010 --> 00:23:38.140
The limit of this
side, we put the
00:23:38.140 --> 00:23:40.120
limit inside the summation.
00:23:40.120 --> 00:23:44.330
Now, as n goes to infinity,
n - also goes to infinity.
00:23:44.330 --> 00:23:48.530
So this Rik is going to be the
steady-state probability of
00:23:48.530 --> 00:23:51.150
state k starting from state i.
00:23:51.150 --> 00:23:53.170
Now where we started
doesn't matter.
00:23:53.170 --> 00:23:54.620
So this is just the
steady-state
00:23:54.620 --> 00:23:56.290
probability of state k.
00:23:56.290 --> 00:24:00.000
So this term converges to that
one, and this gives us an
00:24:00.000 --> 00:24:03.010
equation that's satisfied
by the steady-state
00:24:03.010 --> 00:24:04.030
probabilities.
00:24:04.030 --> 00:24:06.150
Actually, it's not
one equation.
00:24:06.150 --> 00:24:10.580
We get one equation for
each one of the j's.
00:24:10.580 --> 00:24:13.220
So if we have 10 possible
states, we're going to get the
00:24:13.220 --> 00:24:15.580
system of 10 linear equations.
00:24:15.580 --> 00:24:18.840
In the unknowns, pi(1)
up to pi(10).
00:24:18.840 --> 00:24:19.210
OK.
00:24:19.210 --> 00:24:20.800
10 unknowns, 10 equations.
00:24:20.800 --> 00:24:23.150
You might think that
we are in business.
00:24:23.150 --> 00:24:27.646
But actually, this system of
equations is singular.
00:24:27.646 --> 00:24:30.180
0 is a possible solution
of this system.
00:24:30.180 --> 00:24:32.900
If you plug pi equal to
zero everywhere, the
00:24:32.900 --> 00:24:33.900
equations are satisfied.
00:24:33.900 --> 00:24:37.550
It does not have a unique
solution, so maybe we need one
00:24:37.550 --> 00:24:40.580
more condition to get the
uniquely solvable system of
00:24:40.580 --> 00:24:41.900
linear equations.
00:24:41.900 --> 00:24:44.350
It turns out that this
system of equations
00:24:44.350 --> 00:24:46.150
has a unique solution.
00:24:46.150 --> 00:24:48.980
If you impose an additional
condition, which is pretty
00:24:48.980 --> 00:24:52.160
natural, the pi(j)'s are the
probabilities of the different
00:24:52.160 --> 00:24:54.960
states, so they should
add to 1.
00:24:54.960 --> 00:24:58.410
So you want this one equation
to the mix.
00:24:58.410 --> 00:25:05.490
And once you do that, then this
system of equations is
00:25:05.490 --> 00:25:07.340
going to have a unique
solution.
00:25:07.340 --> 00:25:09.550
And so we can find the
steady-state probabilities of
00:25:09.550 --> 00:25:12.980
the Markov chain by just
solving these linear
00:25:12.980 --> 00:25:16.130
equations, which is numerically
straightforward.
00:25:16.130 --> 00:25:18.790
Now, these equations are
quite important.
00:25:18.790 --> 00:25:23.240
I mean, they're the central
point in the Markov chain.
00:25:23.240 --> 00:25:24.220
They have a name.
00:25:24.220 --> 00:25:27.030
They're called the balance
equations.
00:25:27.030 --> 00:25:31.260
And it's worth interpreting
them in a
00:25:31.260 --> 00:25:33.450
somewhat different way.
00:25:33.450 --> 00:25:37.030
So intuitively, one can
sometimes think of
00:25:37.030 --> 00:25:39.290
probabilities as frequencies.
00:25:39.290 --> 00:25:45.780
For example, if I toss an
unbiased coin, probability 1/2
00:25:45.780 --> 00:25:49.600
of heads, you could also say
that if I keep flipping that
00:25:49.600 --> 00:25:52.510
coin, in the long run,
1/2 of the time, I'm
00:25:52.510 --> 00:25:54.500
going to see heads.
00:25:54.500 --> 00:25:58.910
Similarly, let's try an
interpretation of this pi(j),
00:25:58.910 --> 00:26:02.500
the steady-state probability,
the long-term probability of
00:26:02.500 --> 00:26:04.980
finding myself at state j.
00:26:04.980 --> 00:26:08.440
Let's try to interpret it as
the frequency with which I
00:26:08.440 --> 00:26:12.620
find myself at state j if
I run a very, very long
00:26:12.620 --> 00:26:14.940
trajectory over that
Markov chain.
00:26:14.940 --> 00:26:18.400
So the trajectory moves
around, visits states.
00:26:18.400 --> 00:26:22.470
It visits the different states
with different frequencies.
00:26:22.470 --> 00:26:27.380
And let's think of the
probability that you are at a
00:26:27.380 --> 00:26:32.270
certain state as being sort of
the same as the frequency of
00:26:32.270 --> 00:26:34.420
visiting that state.
00:26:34.420 --> 00:26:37.290
This turns out to be a
correct statement.
00:26:37.290 --> 00:26:41.040
If you were more rigorous, you
would have to prove it.
00:26:41.040 --> 00:26:44.390
But it's an interpretation which
is valid and which gives
00:26:44.390 --> 00:26:48.560
us a lot of intuition about what
these equation is saying.
00:26:48.560 --> 00:26:50.170
So let's think as follows.
00:26:50.170 --> 00:26:54.240
Let's focus on a particular
state j, and think of
00:26:54.240 --> 00:27:00.660
transitions into the state j
versus transitions out of the
00:27:00.660 --> 00:27:05.410
state j, or transitions into
j versus transitions
00:27:05.410 --> 00:27:07.080
starting from j.
00:27:07.080 --> 00:27:10.010
So transition starting
from that includes a
00:27:10.010 --> 00:27:11.260
self-transition.
00:27:14.980 --> 00:27:15.110
Ok.
00:27:15.110 --> 00:27:18.300
So how often do we get a
transition, if we interpret
00:27:18.300 --> 00:27:21.230
the pi(j)'s as frequencies,
how often do we get a
00:27:21.230 --> 00:27:23.110
transition into j?
00:27:23.110 --> 00:27:25.870
Here's how we think about it.
00:27:25.870 --> 00:27:31.110
A fraction pi(1) of the time,
we're going to be at state 1.
00:27:31.110 --> 00:27:35.070
Whenever we are at state 1,
there's going to be a
00:27:35.070 --> 00:27:40.550
probability, P1j, that we make
a transition of this kind.
00:27:40.550 --> 00:27:44.730
So out of the times that we're
at state 1, there's a
00:27:44.730 --> 00:27:50.141
frequency, P1j with which the
next transition is into j.
00:27:53.230 --> 00:27:57.870
So out of the overall number of
transitions that happen at
00:27:57.870 --> 00:28:01.820
the trajectory, what fraction
of those transitions is
00:28:01.820 --> 00:28:03.700
exactly of that kind?
00:28:03.700 --> 00:28:06.710
That fraction of transitions is
the fraction of time that
00:28:06.710 --> 00:28:11.940
you find yourself at 1 times the
fraction with which out of
00:28:11.940 --> 00:28:15.700
one you happen to visit
next state j.
00:28:15.700 --> 00:28:19.300
So we interpreted this number
as the frequency of
00:28:19.300 --> 00:28:21.610
transitions of this kind.
00:28:21.610 --> 00:28:24.780
At any given time, our chain
can do transitions of
00:28:24.780 --> 00:28:28.470
different kinds, transitions of
the general form from some
00:28:28.470 --> 00:28:30.670
k, I go to some l.
00:28:30.670 --> 00:28:33.600
So we try to do some
accounting.
00:28:33.600 --> 00:28:37.740
How often does a transition of
each particular kind happen?
00:28:37.740 --> 00:28:40.950
And this is the frequency with
which transitions of that
00:28:40.950 --> 00:28:42.970
particular kind happens.
00:28:42.970 --> 00:28:44.610
Now, what's the total
frequency of
00:28:44.610 --> 00:28:46.870
transitions into state j?
00:28:46.870 --> 00:28:49.880
Transitions into state j can
happen by having a transition
00:28:49.880 --> 00:28:54.510
from 1 to j, from 2 to j,
or from state m to j.
00:28:54.510 --> 00:28:58.590
So to find the total frequency
with which we would observe
00:28:58.590 --> 00:29:03.960
transitions into j is going
to be this particular sum.
00:29:03.960 --> 00:29:09.090
Now, you are at state j if and
only if the last transition
00:29:09.090 --> 00:29:11.440
was into state j.
00:29:11.440 --> 00:29:16.230
So the frequency with which you
are at j is the frequency
00:29:16.230 --> 00:29:20.100
with which transitions
into j happen.
00:29:20.100 --> 00:29:24.020
So this equation expresses
exactly that statement.
00:29:24.020 --> 00:29:27.740
The probability of being at
state j is the sum of the
00:29:27.740 --> 00:29:32.440
probabilities that the last
transition was into state j.
00:29:32.440 --> 00:29:35.120
Or in terms of frequencies, the
frequency with which you
00:29:35.120 --> 00:29:39.170
find yourself at state j is the
sum of the frequencies of
00:29:39.170 --> 00:29:42.580
all the possible transition
types that take you
00:29:42.580 --> 00:29:45.360
inside state j.
00:29:45.360 --> 00:29:47.770
So that's a useful intuition
to have, and we're going to
00:29:47.770 --> 00:29:52.360
see an example a little later
that it gives us short cuts
00:29:52.360 --> 00:29:55.240
into analyzing Markov chains.
00:29:55.240 --> 00:29:58.270
But before we move,
let's revisit the
00:29:58.270 --> 00:30:01.450
example from last time.
00:30:01.450 --> 00:30:03.560
And let us write down
the balance
00:30:03.560 --> 00:30:06.090
equations for this example.
00:30:06.090 --> 00:30:09.380
So the steady-state probability
that I find myself
00:30:09.380 --> 00:30:16.370
at state 1 is the probability
that the previous time I was
00:30:16.370 --> 00:30:21.550
at state 1 and I made
a self-transition--
00:30:21.550 --> 00:30:25.540
So the probability that I was
here last time and I made a
00:30:25.540 --> 00:30:28.290
transition of this kind, plus
the probability that the last
00:30:28.290 --> 00:30:32.220
time I was here and I made a
transition of that kind.
00:30:32.220 --> 00:30:36.100
So plus pi(2) times 0.2.
00:30:36.100 --> 00:30:43.060
And similarly, for the other
states, the steady-state
00:30:43.060 --> 00:30:46.250
probably that I find myself at
state 2 is the probability
00:30:46.250 --> 00:30:50.650
that last time I was at state 1
and I made a transition into
00:30:50.650 --> 00:30:53.780
state 2, plus the probability
that the last time I was at
00:30:53.780 --> 00:30:57.750
state 2 and I made the
transition into state 1.
00:30:57.750 --> 00:30:59.910
Now, these are two
equations and two
00:30:59.910 --> 00:31:02.040
unknowns, pi(1) and pi(2).
00:31:02.040 --> 00:31:06.150
But you notice that both of
these equations tell you the
00:31:06.150 --> 00:31:07.170
same thing.
00:31:07.170 --> 00:31:12.226
They tell you that 0.5pi(1)
equals 0.2pi(2).
00:31:17.770 --> 00:31:21.820
Either of these equations tell
you exactly this if you move
00:31:21.820 --> 00:31:22.910
terms around.
00:31:22.910 --> 00:31:25.450
So these two equations are
not really two equations.
00:31:25.450 --> 00:31:27.010
It's just one equation.
00:31:27.010 --> 00:31:30.520
They are linearly dependent
equations, and in order to
00:31:30.520 --> 00:31:33.320
solve the problem, we need the
additional condition that
00:31:33.320 --> 00:31:36.300
pi(1) + pi(2) is equal to 1.
00:31:36.300 --> 00:31:38.420
Now, we have our system
of two equations,
00:31:38.420 --> 00:31:39.880
which you can solve.
00:31:39.880 --> 00:31:45.030
And once you solve it, you find
that pi(1) is 2/7 and
00:31:45.030 --> 00:31:48.610
pi(2) is 5/7.
00:31:48.610 --> 00:31:52.880
So these are the steady state
probabilities of the two
00:31:52.880 --> 00:31:54.130
different states.
00:31:56.600 --> 00:32:01.770
If we start this chain, at some
state, let's say state 1,
00:32:01.770 --> 00:32:07.050
and we let it run for a long,
long time, the chain settles
00:32:07.050 --> 00:32:08.580
into steady state.
00:32:08.580 --> 00:32:09.440
What does that mean?
00:32:09.440 --> 00:32:12.470
It does not mean that
the state itself
00:32:12.470 --> 00:32:13.960
enters steady state.
00:32:13.960 --> 00:32:17.390
The state will keep jumping
around forever and ever.
00:32:17.390 --> 00:32:21.040
It will keep visiting both
states once in a while.
00:32:21.040 --> 00:32:23.250
So the jumping never ceases.
00:32:23.250 --> 00:32:25.790
The thing that gets into
steady state is the
00:32:25.790 --> 00:32:30.180
probability of finding
yourself at state 1.
00:32:30.180 --> 00:32:34.020
So the probability that you find
yourself at state 1 at
00:32:34.020 --> 00:32:37.640
time one trillion is
approximately 2/7.
00:32:37.640 --> 00:32:40.990
The probability you find
yourself at state 1 at time
00:32:40.990 --> 00:32:45.590
two trillions is again,
approximately 2/7.
00:32:45.590 --> 00:32:48.640
So the probability of being in
that state settles into a
00:32:48.640 --> 00:32:52.270
steady value.
00:32:52.270 --> 00:32:55.630
That's what the steady-state
convergence means.
00:32:55.630 --> 00:32:58.370
It's convergence of
probabilities, not convergence
00:32:58.370 --> 00:33:00.680
of the process itself.
00:33:00.680 --> 00:33:04.750
And again, the two main things
that are happening in this
00:33:04.750 --> 00:33:07.940
example, and more generally,
when we have a single class
00:33:07.940 --> 00:33:10.650
and no periodicity, is
that the initial
00:33:10.650 --> 00:33:12.600
state does not matter.
00:33:12.600 --> 00:33:15.980
There's enough randomness here
so that no matter where you
00:33:15.980 --> 00:33:19.720
start, the randomness kind of
washes out any memory of where
00:33:19.720 --> 00:33:20.780
you started.
00:33:20.780 --> 00:33:23.270
And also in this example,
clearly, we do not have
00:33:23.270 --> 00:33:27.500
periodicity because
we have self arcs.
00:33:27.500 --> 00:33:30.960
And this, in particular, implies
that the exact time
00:33:30.960 --> 00:33:32.210
does not matter.
00:33:35.740 --> 00:33:41.510
So now, we're going to spend
the rest of our time by
00:33:41.510 --> 00:33:45.330
looking into a special class
of chains that's a little
00:33:45.330 --> 00:33:47.450
easier to deal with,
but still, it's
00:33:47.450 --> 00:33:49.436
an important class.
00:33:49.436 --> 00:33:52.010
So what's the moral from here?
00:33:52.010 --> 00:33:55.680
This was a simple example with
two states, and we could find
00:33:55.680 --> 00:33:59.440
the steady-state probabilities
by solving a simple system of
00:33:59.440 --> 00:34:01.320
two-by-two equations.
00:34:01.320 --> 00:34:05.690
If you have a chain with 100
states, it's no problem for a
00:34:05.690 --> 00:34:09.120
computer to solve a system
of 100-by-100 equations.
00:34:09.120 --> 00:34:12.800
But you can certainly not do it
by hand, and usually, you
00:34:12.800 --> 00:34:15.560
cannot get any closed-form
formulas, so you do not
00:34:15.560 --> 00:34:17.969
necessarily get a
lot of insight.
00:34:17.969 --> 00:34:21.330
So one looks for special
structures or models that
00:34:21.330 --> 00:34:25.610
maybe give you a little more
insight or maybe lead you to
00:34:25.610 --> 00:34:27.690
closed-form formulas.
00:34:27.690 --> 00:34:32.130
And an interesting subclass of
Markov chains in which all of
00:34:32.130 --> 00:34:35.080
these nice things do happen,
is the class
00:34:35.080 --> 00:34:39.080
of birth/death processes.
00:34:39.080 --> 00:34:41.500
So what's a birth/death
process?
00:34:41.500 --> 00:34:44.510
It's a Markov chain who's
diagram looks
00:34:44.510 --> 00:34:46.810
basically like this.
00:34:46.810 --> 00:34:52.020
So the states of the Markov
chain start from 0 and go up
00:34:52.020 --> 00:34:54.460
to some finite integer m.
00:34:54.460 --> 00:34:57.400
What's special about this chain
is that if you are at a
00:34:57.400 --> 00:35:02.710
certain state, next time you can
either go up by 1, you can
00:35:02.710 --> 00:35:06.640
go down by 1, or you
can stay in place.
00:35:06.640 --> 00:35:09.820
So it's like keeping track
of some population
00:35:09.820 --> 00:35:11.310
at any given time.
00:35:11.310 --> 00:35:13.820
One person gets born,
or one person
00:35:13.820 --> 00:35:15.680
dies, or nothing happens.
00:35:15.680 --> 00:35:19.150
Again, we're not accounting
for twins here.
00:35:19.150 --> 00:35:24.430
So we're given this structure,
and we are given the
00:35:24.430 --> 00:35:27.660
transition probabilities, the
probabilities associated with
00:35:27.660 --> 00:35:29.630
transitions of the
different types.
00:35:29.630 --> 00:35:32.880
So we use P's for the upward
transitions, Q's for the
00:35:32.880 --> 00:35:34.550
downward transitions.
00:35:34.550 --> 00:35:37.830
An example of a chain of this
kind was the supermarket
00:35:37.830 --> 00:35:40.410
counter model that we
discussed last time.
00:35:40.410 --> 00:35:45.080
That is, a customer arrives,
so this increments
00:35:45.080 --> 00:35:46.310
the state by 1.
00:35:46.310 --> 00:35:49.670
Or a customer finishes service,
in which case, the
00:35:49.670 --> 00:35:53.020
state gets decremented by 1,
or nothing happens in which
00:35:53.020 --> 00:35:55.500
you stay in place, and so on.
00:35:55.500 --> 00:35:59.880
In the supermarket model, these
P's inside here were all
00:35:59.880 --> 00:36:03.780
taken to be equal because we
assume that the arrival rate
00:36:03.780 --> 00:36:07.400
was sort of constant
at each time slot.
00:36:07.400 --> 00:36:10.960
But you can generalize a little
bit by assuming that
00:36:10.960 --> 00:36:15.330
these transition probabilities
P1 here, P2 there, and so on
00:36:15.330 --> 00:36:18.190
may be different from
state to state.
00:36:18.190 --> 00:36:21.750
So in general, from state
i, there's going to be a
00:36:21.750 --> 00:36:24.400
transition probability
Pi that the next
00:36:24.400 --> 00:36:26.210
transition is upwards.
00:36:26.210 --> 00:36:29.820
And there's going to be a
probability Qi that the next
00:36:29.820 --> 00:36:31.910
transition is downwards.
00:36:31.910 --> 00:36:35.310
And so from that state, the
probability that the next
00:36:35.310 --> 00:36:37.640
transition is downwards is
going to be Q_(i+1).
00:36:40.930 --> 00:36:43.650
So this is the structure
of our chain.
00:36:43.650 --> 00:36:47.580
As I said, it's a crude model
of what happens at the
00:36:47.580 --> 00:36:53.820
supermarket counter but it's
also a good model for lots of
00:36:53.820 --> 00:36:55.370
types of service systems.
00:36:55.370 --> 00:36:59.400
Again, you have a server
somewhere that has a buffer.
00:36:59.400 --> 00:37:00.970
Jobs come into the buffer.
00:37:00.970 --> 00:37:02.460
So the buffer builds up.
00:37:02.460 --> 00:37:06.880
The server processes jobs, so
the buffer keeps going down.
00:37:06.880 --> 00:37:10.470
And the state of the chain would
be the number of jobs
00:37:10.470 --> 00:37:12.570
that you have inside
your buffer.
00:37:12.570 --> 00:37:18.690
Or you could be thinking about
active phone calls out of a
00:37:18.690 --> 00:37:19.680
certain city.
00:37:19.680 --> 00:37:22.520
Each time that the phone call
is placed, the number of
00:37:22.520 --> 00:37:24.090
active phone calls
goes up by 1.
00:37:24.090 --> 00:37:28.360
Each time that the phone call
stops happening, is
00:37:28.360 --> 00:37:31.790
terminated, then the count
goes down by 1.
00:37:31.790 --> 00:37:34.710
So it's for processes of this
kind that a model with this
00:37:34.710 --> 00:37:36.890
structure is going to show up.
00:37:36.890 --> 00:37:39.240
And they do show up in
many, many models.
00:37:39.240 --> 00:37:43.730
Or you can think about the
number of people in a certain
00:37:43.730 --> 00:37:45.690
population that have
a disease.
00:37:45.690 --> 00:37:51.010
So 1 more person gets the
flu, the count goes up.
00:37:51.010 --> 00:37:55.320
1 more person gets healed,
the count goes down.
00:37:55.320 --> 00:37:58.350
And these probabilities in such
an epidemic model would
00:37:58.350 --> 00:38:02.270
certainly depend on
the current state.
00:38:02.270 --> 00:38:06.280
If lots of people already have
the flu, the probability that
00:38:06.280 --> 00:38:10.010
another person catches it
would be pretty high.
00:38:10.010 --> 00:38:13.800
Whereas, if no one has the flu,
then the probability that
00:38:13.800 --> 00:38:16.820
you get a transition where
someone catches the flu, that
00:38:16.820 --> 00:38:18.960
probability would
be pretty small.
00:38:18.960 --> 00:38:26.990
So the transition rates, the
incidence of new people who
00:38:26.990 --> 00:38:30.010
have the disease definitely
depends on how many people
00:38:30.010 --> 00:38:31.560
already have the disease.
00:38:31.560 --> 00:38:34.970
And that motivates cases where
those P's, the upward
00:38:34.970 --> 00:38:39.220
transition probabilities,
depend on the
00:38:39.220 --> 00:38:42.400
state of the chain.
00:38:42.400 --> 00:38:44.040
So how do we study this chain?
00:38:44.040 --> 00:38:49.220
You can sit down and write the
system of n linear equations
00:38:49.220 --> 00:38:50.550
in the pi's.
00:38:50.550 --> 00:38:52.920
And this way, find the
steady-state probabilities of
00:38:52.920 --> 00:38:53.790
this chain.
00:38:53.790 --> 00:38:55.850
But this is a little harder.
00:38:55.850 --> 00:38:59.310
It's more work than one
actually needs to do.
00:38:59.310 --> 00:39:03.200
There's a very clever shortcut
that applies
00:39:03.200 --> 00:39:05.160
to birth/death processes.
00:39:05.160 --> 00:39:08.410
And it's based on the frequency
interpretation that
00:39:08.410 --> 00:39:10.000
we discussed a little
while ago.
00:39:14.070 --> 00:39:17.390
Let's put a line somewhere in
the middle of this chain, and
00:39:17.390 --> 00:39:21.710
focus on the relation between
this part and that part in
00:39:21.710 --> 00:39:22.750
more detail.
00:39:22.750 --> 00:39:25.460
So think of the chain continuing
in this direction,
00:39:25.460 --> 00:39:26.390
that direction.
00:39:26.390 --> 00:39:30.020
But let's just focus on 2
adjacent states, and look at
00:39:30.020 --> 00:39:32.270
this particular cut.
00:39:32.270 --> 00:39:34.270
What is the chain going to do?
00:39:34.270 --> 00:39:35.550
Let's say it starts here.
00:39:35.550 --> 00:39:37.250
It's going to move around.
00:39:37.250 --> 00:39:40.280
At some point, it makes a
transition to the other side.
00:39:40.280 --> 00:39:42.860
And that's a transition
from i to i+1.
00:39:42.860 --> 00:39:45.470
It stays on the other
side for some time.
00:39:45.470 --> 00:39:48.490
It gets here, and eventually,
it's going to make a
00:39:48.490 --> 00:39:50.340
transition to this side.
00:39:50.340 --> 00:39:53.110
Then it keeps moving
and so on.
00:39:53.110 --> 00:39:57.680
Now, there's a certain balance
that must be obeyed here.
00:39:57.680 --> 00:40:01.300
The number of upward transitions
through this line
00:40:01.300 --> 00:40:04.220
cannot be very different from
the number of downward
00:40:04.220 --> 00:40:06.080
transitions.
00:40:06.080 --> 00:40:09.530
Because we cross this
way, then next time,
00:40:09.530 --> 00:40:10.630
we'll cross that way.
00:40:10.630 --> 00:40:12.580
Then next time, we'll
cross this way.
00:40:12.580 --> 00:40:13.970
We'll cross that way.
00:40:13.970 --> 00:40:18.940
So the frequency with which
transitions of this kind occur
00:40:18.940 --> 00:40:21.500
has to be the same as the
long-term frequency that
00:40:21.500 --> 00:40:24.430
transitions of that
kind occur.
00:40:24.430 --> 00:40:28.510
You cannot go up 100 times and
go down only 50 times.
00:40:28.510 --> 00:40:31.740
If you have gone up 100 times,
it means that you have gone
00:40:31.740 --> 00:40:36.960
down 99, or 100, or 101,
but nothing much more
00:40:36.960 --> 00:40:38.570
different than that.
00:40:38.570 --> 00:40:41.440
So the frequency with
which transitions of
00:40:41.440 --> 00:40:43.670
this kind get observed.
00:40:43.670 --> 00:40:47.610
That is, out of a large number
of transitions, what fraction
00:40:47.610 --> 00:40:49.890
of transitions are
of these kind?
00:40:49.890 --> 00:40:52.160
That fraction has to be the
same as the fraction of
00:40:52.160 --> 00:40:54.870
transitions that happened
to be of that kind.
00:40:54.870 --> 00:40:56.620
What are these fractions?
00:40:56.620 --> 00:40:58.480
We discussed that before.
00:40:58.480 --> 00:41:04.840
The fraction of times at which
transitions of this kind are
00:41:04.840 --> 00:41:08.340
observed is the fraction of time
that we happen to be at
00:41:08.340 --> 00:41:09.350
that state.
00:41:09.350 --> 00:41:11.790
And out of the times that we
are in that state, the
00:41:11.790 --> 00:41:15.010
fraction of transitions that
happen to be upward
00:41:15.010 --> 00:41:16.040
transitions.
00:41:16.040 --> 00:41:20.820
So this is the frequency with
which transitions of this kind
00:41:20.820 --> 00:41:22.430
are observed.
00:41:22.430 --> 00:41:25.350
And with the same argument,
this is the frequency with
00:41:25.350 --> 00:41:28.670
which transitions of that
kind are observed.
00:41:28.670 --> 00:41:31.120
Since these two frequencies
are the same, these two
00:41:31.120 --> 00:41:34.390
numbers must be the same, and
we get an equation that
00:41:34.390 --> 00:41:38.350
relates the Pi to P_(i+1).
00:41:38.350 --> 00:41:43.040
This has a nice form because
it gives us a recursion.
00:41:43.040 --> 00:41:45.760
If we knew pi(i), we could then
00:41:45.760 --> 00:41:48.860
immediately calculate pi(i+1).
00:41:48.860 --> 00:41:51.860
So it's a system of equations
that's very
00:41:51.860 --> 00:41:54.860
easy to solve almost.
00:41:54.860 --> 00:41:57.320
But how do we get started?
00:41:57.320 --> 00:42:01.850
If I knew pi(0), I could find
by pi(1) and then use this
00:42:01.850 --> 00:42:05.330
recursion to find pi(2),
pi(3), and so on.
00:42:05.330 --> 00:42:06.970
But we don't know pi(0).
00:42:06.970 --> 00:42:09.920
It's one more unknown.
00:42:09.920 --> 00:42:14.290
It's an unknown, and we need
to actually use the extra
00:42:14.290 --> 00:42:20.000
normalization condition that
the sum of the pi's is 1.
00:42:20.000 --> 00:42:23.810
And after we use that
normalization condition, then
00:42:23.810 --> 00:42:26.580
we can find all of the pi's.
00:42:32.550 --> 00:42:38.450
So you basically fix pi(0) as a
symbol, solve this equation
00:42:38.450 --> 00:42:41.580
symbolically, and
everything gets
00:42:41.580 --> 00:42:43.830
expressed in terms of pi(0).
00:42:43.830 --> 00:42:46.510
And then use that normalization
condition to
00:42:46.510 --> 00:42:48.700
find pi(0), and you're done.
00:42:48.700 --> 00:42:51.570
Let's illustrate the details
of this procedure on a
00:42:51.570 --> 00:42:53.690
particular special case.
00:42:53.690 --> 00:42:57.360
So in our special case, we're
going to simplify things now
00:42:57.360 --> 00:43:01.760
by assuming that all those
upward P's are the same, and
00:43:01.760 --> 00:43:05.780
all of those downward
Q's are the same.
00:43:05.780 --> 00:43:08.860
So at each point in time, if
you're sitting somewhere in
00:43:08.860 --> 00:43:13.060
the middle, you have probability
P of moving up and
00:43:13.060 --> 00:43:16.710
probability Q of moving down.
00:43:16.710 --> 00:43:23.460
This rho, the ratio of P/Q is
frequency of going up versus
00:43:23.460 --> 00:43:25.670
frequency of going down.
00:43:25.670 --> 00:43:29.070
If it's a service system, you
can think of it as a measure
00:43:29.070 --> 00:43:32.140
of how loaded the system is.
00:43:32.140 --> 00:43:39.100
If P is equal to Q, it's means
that if you're at this state,
00:43:39.100 --> 00:43:42.700
you're equally likely to move
left or right, so the system
00:43:42.700 --> 00:43:44.820
is kind of balanced.
00:43:44.820 --> 00:43:46.980
The state doesn't have a
tendency to move in this
00:43:46.980 --> 00:43:49.450
direction or in that
direction.
00:43:49.450 --> 00:43:53.860
If rho is bigger than 1 so that
P is bigger than Q, it
00:43:53.860 --> 00:43:56.730
means that whenever I'm at some
state in the middle, I'm
00:43:56.730 --> 00:44:00.830
more likely to move right rather
than move left, which
00:44:00.830 --> 00:44:04.230
means that my state, of course
it's random, but it has a
00:44:04.230 --> 00:44:07.170
tendency to move in
that direction.
00:44:07.170 --> 00:44:10.280
And if you think of this as a
number of customers in queue,
00:44:10.280 --> 00:44:13.750
it means your system has the
tendency to become loaded and
00:44:13.750 --> 00:44:15.420
to build up a queue.
00:44:15.420 --> 00:44:19.790
So rho being bigger than 1
corresponds to a heavy load,
00:44:19.790 --> 00:44:21.570
where queues build up.
00:44:21.570 --> 00:44:25.470
Rho less than 1 corresponds to
the system where queues have
00:44:25.470 --> 00:44:27.310
the tendency to drain down.
00:44:30.880 --> 00:44:32.680
Now, let's write down
the equations.
00:44:32.680 --> 00:44:40.540
We have this recursion P_(i+1)
is Pi times Pi over Qi.
00:44:40.540 --> 00:44:44.040
In our case here, the P's and
the Q's do not depend on the
00:44:44.040 --> 00:44:47.190
particular index, so we
get this relation.
00:44:47.190 --> 00:44:51.830
And this P over Q is just
the load factor rho.
00:44:51.830 --> 00:44:54.790
Once you look at this equation,
clearly you realize
00:44:54.790 --> 00:44:58.440
that by pi(1) is rho
times pi(0).
00:44:58.440 --> 00:45:02.286
pi(2) is going to be --
00:45:02.286 --> 00:45:04.480
So we'll do it in detail.
00:45:04.480 --> 00:45:08.130
So pi(1) is pi(0) times rho.
00:45:08.130 --> 00:45:15.580
pi(2) is pi(1) times rho,
which is pi(0) times
00:45:15.580 --> 00:45:18.350
rho-squared.
00:45:18.350 --> 00:45:21.340
And then you continue doing
this calculation.
00:45:21.340 --> 00:45:25.530
And you find that you can
express every pi(i) in terms
00:45:25.530 --> 00:45:31.490
of pi(0) and you get this
factor of rho^i.
00:45:31.490 --> 00:45:34.840
And then you use the last
equation that we have -- that
00:45:34.840 --> 00:45:38.110
the sum of the probabilities
has to be equal to 1.
00:45:38.110 --> 00:45:41.670
And that equation is going to
tell us that the sum over all
00:45:41.670 --> 00:45:50.890
i's from 0 to m of pi(0) rho
to the i is equal to 1.
00:45:50.890 --> 00:45:58.730
And therefore, pi(0) is 1 over
(the sum over the rho to the i
00:45:58.730 --> 00:46:03.100
for i going from 0 to m).
00:46:03.100 --> 00:46:09.870
So now we found pi(0), and by
plugging in this expression,
00:46:09.870 --> 00:46:12.680
we have the steady-state
probabilities of all of the
00:46:12.680 --> 00:46:14.950
different states.
00:46:14.950 --> 00:46:18.990
Let's look at some special
cases of this.
00:46:18.990 --> 00:46:24.390
Suppose that rho
is equal to 1.
00:46:24.390 --> 00:46:30.410
If rho is equal to 1, then
pi(i) is equal to pi(0).
00:46:30.410 --> 00:46:33.250
It means that all
the steady-state
00:46:33.250 --> 00:46:35.840
probabilities are equal.
00:46:35.840 --> 00:46:39.380
It's means that every
state is equally
00:46:39.380 --> 00:46:42.750
likely in the long run.
00:46:42.750 --> 00:46:44.790
So this is an example.
00:46:44.790 --> 00:46:48.730
It's called a symmetric
random walk.
00:46:48.730 --> 00:46:53.260
It's a very popular model for
modeling people who are drunk.
00:46:53.260 --> 00:46:56.730
So you start at a state
at any point in time.
00:46:56.730 --> 00:47:00.140
Either you stay in place, or you
have an equal probability
00:47:00.140 --> 00:47:02.910
of going left or going right.
00:47:02.910 --> 00:47:06.220
There's no bias in
either direction.
00:47:06.220 --> 00:47:10.710
You might think that in such a
process, you will tend to kind
00:47:10.710 --> 00:47:14.750
of get stuck near one end
or the other end.
00:47:14.750 --> 00:47:17.320
Well, it's not really clear
what to expect.
00:47:17.320 --> 00:47:21.260
It turns out that in such a
model, in the long run, the
00:47:21.260 --> 00:47:24.610
drunk person is equally
likely to be at any
00:47:24.610 --> 00:47:26.430
one of those states.
00:47:26.430 --> 00:47:31.300
The steady-state probability is
the same for all i's if rho
00:47:31.300 --> 00:47:33.880
is equal to 1.
00:47:33.880 --> 00:47:39.570
And so if you show up at a
random time, and you ask where
00:47:39.570 --> 00:47:43.670
is my state, you will be told
it's equally likely to be at
00:47:43.670 --> 00:47:46.600
any one of those places.
00:47:46.600 --> 00:47:48.370
So let's make that note.
00:47:48.370 --> 00:47:51.980
If rho equal to 1, implies
that all the
00:47:51.980 --> 00:47:53.660
pi(i)'s are 1/(M+1) --
00:47:57.040 --> 00:48:01.320
M+1 because that's how many
states we have in our model.
00:48:01.320 --> 00:48:04.210
Now, let's look at
a different case.
00:48:04.210 --> 00:48:08.630
Suppose that M is
a huge number.
00:48:08.630 --> 00:48:13.600
So essentially, our supermarket
has a very large
00:48:13.600 --> 00:48:19.600
space, a lot of space to
store their customers.
00:48:19.600 --> 00:48:24.130
But suppose that the system
is on the stable side.
00:48:24.130 --> 00:48:27.900
P is less than Q, which means
that there's a tendency for
00:48:27.900 --> 00:48:31.420
customers to be served faster
than they arrive.
00:48:31.420 --> 00:48:35.500
The drift in this chain, it
tends to be in that direction.
00:48:35.500 --> 00:48:41.920
So when rho is less than 1,
which is this case, and when M
00:48:41.920 --> 00:48:45.810
is going to infinity, this
infinite sum is the sum of a
00:48:45.810 --> 00:48:47.690
geometric series.
00:48:47.690 --> 00:48:50.480
And you recognize it
(hopefully) --
00:48:53.480 --> 00:48:56.520
this series is going
to 1/(1-rho).
00:48:56.520 --> 00:49:00.280
And because it's in the
denominator, pi(0) ends up
00:49:00.280 --> 00:49:02.970
being 1-rho.
00:49:02.970 --> 00:49:06.230
So by taking the limit as M
goes to infinity, in this
00:49:06.230 --> 00:49:09.460
case, and when rho is less than
1 so that this series is
00:49:09.460 --> 00:49:12.220
convergent, we get
this formula.
00:49:12.220 --> 00:49:15.780
So we get the closed-form
formula for the pi(i)'s.
00:49:15.780 --> 00:49:19.840
In particular, pi(i) is (1-
rho)(rho to the i).
00:49:19.840 --> 00:49:21.150
to
00:49:21.150 --> 00:49:24.520
So these pi(i)'s are essentially
a probability
00:49:24.520 --> 00:49:26.200
distribution.
00:49:26.200 --> 00:49:32.100
They tell us if we show up at
time 1 billion and we ask,
00:49:32.100 --> 00:49:33.900
where is my state?
00:49:33.900 --> 00:49:37.500
You will be told that
the state is 0.
00:49:37.500 --> 00:49:41.040
Your system is empty with
probability 1-rho, minus or
00:49:41.040 --> 00:49:44.410
there's one customer in the
system, and that happens with
00:49:44.410 --> 00:49:46.460
probability (rho
- 1) times rho.
00:49:46.460 --> 00:49:49.950
And it keeps going
down this way.
00:49:49.950 --> 00:49:53.920
And it's pretty much a geometric
distribution except
00:49:53.920 --> 00:49:58.130
that it has shifted so that it
starts at 0 whereas the usual
00:49:58.130 --> 00:50:00.890
geometric distribution
starts at 1.
00:50:00.890 --> 00:50:04.670
So this is a mini introduction
into queuing theory.
00:50:04.670 --> 00:50:08.730
This is the first and simplest
model that one encounters when
00:50:08.730 --> 00:50:10.850
you start studying
queuing theory.
00:50:10.850 --> 00:50:13.360
This is clearly a model of a
queueing phenomenon such as
00:50:13.360 --> 00:50:16.450
the supermarket counter with
the P's corresponding to
00:50:16.450 --> 00:50:19.280
arrivals, the Q's corresponding
to departures.
00:50:19.280 --> 00:50:22.690
And this particular queuing
system when M is very, very
00:50:22.690 --> 00:50:26.670
large and rho is less than 1,
has a very simple and nice
00:50:26.670 --> 00:50:28.880
solution in closed form.
00:50:28.880 --> 00:50:31.650
And that's why it's
very much liked.
00:50:31.650 --> 00:50:33.610
And let me just take
two seconds to
00:50:33.610 --> 00:50:38.030
draw one last picture.
00:50:38.030 --> 00:50:40.300
So this is the probability
of the different i's.
00:50:40.300 --> 00:50:41.420
It gives you a PMF.
00:50:41.420 --> 00:50:43.890
This PMF has an expected
value.
00:50:43.890 --> 00:50:47.090
And the expectation, the
expected number of customers
00:50:47.090 --> 00:50:50.560
in the system, is given
by this formula.
00:50:50.560 --> 00:50:54.230
And this formula, which is
interesting to anyone who
00:50:54.230 --> 00:50:57.070
tries to analyze a system
of this kind,
00:50:57.070 --> 00:50:58.430
tells you the following.
00:50:58.430 --> 00:51:03.970
That as long as a rho is less
than 1, then the expected
00:51:03.970 --> 00:51:07.270
number of customers in
the system is finite.
00:51:07.270 --> 00:51:09.970
But if rho becomes very
close to 1 --
00:51:09.970 --> 00:51:13.790
So if your load factor is
something like .99, you expect
00:51:13.790 --> 00:51:17.630
to have a large number of
customers in the system at any
00:51:17.630 --> 00:51:19.040
given time.
00:51:19.040 --> 00:51:20.440
OK.
00:51:20.440 --> 00:51:22.420
All right.
00:51:22.420 --> 00:51:23.270
Have a good weekend.
00:51:23.270 --> 00:51:25.390
We'll continue next time.