WEBVTT

00:00:01.780 --> 00:00:05.600
OK, so we have just seen that
if we have a single recurrent

00:00:05.600 --> 00:00:08.740
class, which is not periodic,
then the Markov chain reaches

00:00:08.740 --> 00:00:13.560
steady state, and the steady
state probabilities satisfy

00:00:13.560 --> 00:00:16.730
the following
system of equations.

00:00:16.730 --> 00:00:19.760
These equations are essential
in the study of Markov chains,

00:00:19.760 --> 00:00:21.150
and they have a name.

00:00:21.150 --> 00:00:23.780
They are called the
balance equations.

00:00:23.780 --> 00:00:26.140
In fact, it's worth
looking at them

00:00:26.140 --> 00:00:29.990
in a somewhat different way than
how we introduced them so far.

00:00:29.990 --> 00:00:32.475
Using a frequency
interpretation.

00:00:32.475 --> 00:00:34.880
Along the way, it
will shed some light

00:00:34.880 --> 00:00:39.640
on the how or why they are
called balance equations.

00:00:39.640 --> 00:00:41.660
Intuitively, one
can sometimes think

00:00:41.660 --> 00:00:43.700
of probabilities as frequencies.

00:00:43.700 --> 00:00:46.420
For example, if we keep
tossing a fair coin, which

00:00:46.420 --> 00:00:49.700
has a probability half of Heads,
and then in the long run, half

00:00:49.700 --> 00:00:51.960
of the time we are
going to see Heads.

00:00:51.960 --> 00:00:54.900
So let us try an interpretation
of this kind for pi

00:00:54.900 --> 00:00:58.940
of j, the steady state
probability of state j.

00:00:58.940 --> 00:01:01.270
Imagine the evolution
of a Markov chain

00:01:01.270 --> 00:01:04.440
as a particle jumping
from state to state.

00:01:04.440 --> 00:01:07.015
And imagine an observer
at a given state.

00:01:07.015 --> 00:01:10.160
So imagine that you
have an observer here,

00:01:10.160 --> 00:01:11.039
in a given state j.

00:01:11.039 --> 00:01:13.240
And imagine that
this observer will

00:01:13.240 --> 00:01:17.890
keep counting every time the
particle visits the state j.

00:01:17.890 --> 00:01:19.840
So you have this
observer keeping

00:01:19.840 --> 00:01:23.780
track every time the particle is
in state j and keep recording.

00:01:23.780 --> 00:01:26.050
So, for example, one
record at time two,

00:01:26.050 --> 00:01:31.080
and saw it at time
four, eight, maybe n.

00:01:31.080 --> 00:01:34.490
And you can look at the
total number of time

00:01:34.490 --> 00:01:37.530
this observer saw the
particle being in j,

00:01:37.530 --> 00:01:38.789
and you can define it.

00:01:38.789 --> 00:01:42.700
Let's call it y of j of n.

00:01:42.700 --> 00:01:47.400
So yj of n represents
the total number of ones

00:01:47.400 --> 00:01:49.690
that you have up to time n.

00:01:49.690 --> 00:01:52.289
So it's the total
number, so we divide by n

00:01:52.289 --> 00:01:53.780
to have the frequency.

00:01:53.780 --> 00:01:56.940
So here that would be
the frequency of time

00:01:56.940 --> 00:02:02.040
this observer saw the particle
in state j up to time n.

00:02:02.040 --> 00:02:07.150
Well, when n is very,
very large, so n large,

00:02:07.150 --> 00:02:12.690
that frequency
approaches pi of j.

00:02:12.690 --> 00:02:14.820
In fact, we can make
it more rigorous

00:02:14.820 --> 00:02:18.800
by saying that that
converges to pi of j

00:02:18.800 --> 00:02:21.700
when n goes to infinity
in a rigorous fashion

00:02:21.700 --> 00:02:24.190
that we will not discuss here.

00:02:24.190 --> 00:02:29.210
So we have now a frequency
interpretation of pi of j.

00:02:29.210 --> 00:02:32.470
Now, let us think
about a frequency

00:02:32.470 --> 00:02:36.510
interpretation of
transitions from 1 to j.

00:02:36.510 --> 00:02:39.870
So again, you have
a new observer,

00:02:39.870 --> 00:02:42.200
and this observer
look at it here,

00:02:42.200 --> 00:02:45.720
and every time the particle
pass here, he put a one.

00:02:45.720 --> 00:02:48.810
So, for example, maybe
one here and here.

00:02:48.810 --> 00:02:53.730
So if you think about it,
you're looking at from 1

00:02:53.730 --> 00:02:57.060
to j, and of n, that would
be the total number of ones

00:02:57.060 --> 00:03:00.340
that you observe
here up to time n.

00:03:00.340 --> 00:03:03.630
And if you divide by 1,
that's the frequency,

00:03:03.630 --> 00:03:05.750
and so what is this frequency?

00:03:05.750 --> 00:03:08.360
Well, let's look at it this way.

00:03:08.360 --> 00:03:10.890
So how often do we
have such a transition?

00:03:10.890 --> 00:03:15.020
Well, a fraction pi1 of
the time, the particle

00:03:15.020 --> 00:03:19.050
is in state 1, and
whenever at state 1,

00:03:19.050 --> 00:03:24.140
there is going to be a
probability p1j of going there.

00:03:24.140 --> 00:03:27.620
There might be other
ways to go, but out

00:03:27.620 --> 00:03:30.990
of all the time the
particle is in state one,

00:03:30.990 --> 00:03:36.750
the frequency of time it will
transition to j will be pi 1j.

00:03:36.750 --> 00:03:40.600
So out of all possible
transitions that can happen,

00:03:40.600 --> 00:03:42.900
the fraction of
these transitions

00:03:42.900 --> 00:03:49.000
that will happen from 1 to
j will be pi 1 times p1j.

00:03:52.079 --> 00:03:54.820
Again, this is when
n is large, and this

00:03:54.820 --> 00:03:58.040
can be made more rigorous.

00:03:58.040 --> 00:04:03.400
Now, what's the total frequency
of transitions into state j?

00:04:03.400 --> 00:04:06.190
So these are
transitions leaving.

00:04:09.020 --> 00:04:12.290
These are the transitions
of interest here.

00:04:12.290 --> 00:04:19.750
So think about a third observer
looking here and recording

00:04:19.750 --> 00:04:23.270
every time the particle
goes through here, here,

00:04:23.270 --> 00:04:26.610
here, or here.

00:04:26.610 --> 00:04:29.150
So what is the frequency
of transition here?

00:04:29.150 --> 00:04:33.630
Well, it will be the sum of
all the possible transitions

00:04:33.630 --> 00:04:35.170
that we have observed there.

00:04:35.170 --> 00:04:40.034
And so this is going to be this
and that corresponds to this.

00:04:42.880 --> 00:04:44.980
Now, the last step
of the argument

00:04:44.980 --> 00:04:50.880
is to see that the
particle is in state j, if

00:04:50.880 --> 00:04:55.680
and only if the last
transition was into state j.

00:04:55.680 --> 00:04:58.940
And this explains that this
part, which we have calculated

00:04:58.940 --> 00:05:03.980
here, will be the same as this
one and that explain that.

00:05:03.980 --> 00:05:07.220
So this equation expresses
exactly the statement

00:05:07.220 --> 00:05:08.030
that we made.

00:05:08.030 --> 00:05:11.100
That's useful intuition
to have, and we

00:05:11.100 --> 00:05:15.020
are going to see an example
later on how it gives us

00:05:15.020 --> 00:05:18.425
shortcuts into
analyzing Markov chains.