WEBVTT
00:00:00.040 --> 00:00:02.460
The following content is
provided under a Creative
00:00:02.460 --> 00:00:03.870
Commons license.
00:00:03.870 --> 00:00:06.910
Your support will help MIT
OpenCourseWare continue to
00:00:06.910 --> 00:00:10.560
offer high quality educational
resources for free.
00:00:10.560 --> 00:00:13.460
To make a donation or view
additional materials from
00:00:13.460 --> 00:00:19.290
hundreds of MIT courses, visit
MIT OpenCourseWare at
00:00:19.290 --> 00:00:20.540
ocw.mit.edu.
00:00:22.170 --> 00:00:23.620
PROFESSOR: So let's
get started.
00:00:23.620 --> 00:00:27.900
With the quiz out of the way we
can now move ahead again.
00:00:30.560 --> 00:00:33.500
I want to talk a little
bit about the
00:00:33.500 --> 00:00:36.350
major renewal theorem.
00:00:36.350 --> 00:00:39.740
Partly review, but partly it's
something that we need because
00:00:39.740 --> 00:00:44.530
we're going on to talk about
countable-state Markov chains,
00:00:44.530 --> 00:00:51.170
and almost all the analysis
there is based on what happens
00:00:51.170 --> 00:00:55.640
when you're dealing with
renewal processes.
00:00:55.640 --> 00:00:58.540
Especially discrete renewal
processes that are the
00:00:58.540 --> 00:01:03.470
renewals when you have
recurrences from one state to
00:01:03.470 --> 00:01:07.450
the same state at some
later point.
00:01:07.450 --> 00:01:12.810
So, in order to do that, we also
have to talk a little bit
00:01:12.810 --> 00:01:18.000
about age and duration at a
given time instead of in terms
00:01:18.000 --> 00:01:25.110
of the sample average, because
that's sort of essential also.
00:01:25.110 --> 00:01:27.630
And it gives you a nice
interpretation of why these
00:01:27.630 --> 00:01:33.040
peculiar things happen about
duration being so much longer
00:01:33.040 --> 00:01:37.880
than it seems like it should be,
and things of that sort.
00:01:37.880 --> 00:01:40.800
We'll probably spend close to
half the lecture dealing with
00:01:40.800 --> 00:01:44.080
that, and the other half dealing
with countable-state
00:01:44.080 --> 00:01:45.330
Markov chains.
00:01:48.070 --> 00:01:51.010
We really have three major
theorems dealing
00:01:51.010 --> 00:01:52.260
with renewal processes.
00:01:55.440 --> 00:01:59.350
One of them is a sample path
time average, which we've
00:01:59.350 --> 00:02:01.960
called the strong law
for renewals.
00:02:01.960 --> 00:02:07.450
It says that if you look at
individual sample paths, it
00:02:07.450 --> 00:02:15.040
says that the limit as an
individual sample path, m as
00:02:15.040 --> 00:02:19.640
the number of arrivals over that
sample path if you look
00:02:19.640 --> 00:02:23.540
at the limit from 0 to infinity
of the number of
00:02:23.540 --> 00:02:28.320
arrivals divided by time,
that's 1 over x-bar.
00:02:28.320 --> 00:02:32.340
And the set of sample paths
for which that's true has
00:02:32.340 --> 00:02:33.450
probability 1.
00:02:33.450 --> 00:02:37.660
That's what that statement
is supposed to say.
00:02:37.660 --> 00:02:42.050
The next one is the elementary
renewal theorem.
00:02:42.050 --> 00:02:47.200
When you look at it carefully,
that doesn't say anything, or
00:02:47.200 --> 00:02:49.390
hardly anything.
00:02:49.390 --> 00:02:53.610
All it says is that if you look
at the limit as t goes to
00:02:53.610 --> 00:02:57.400
infinity, and you also take
the expected value, the
00:02:57.400 --> 00:03:03.350
expected value is the number
of renewals over a period t
00:03:03.350 --> 00:03:04.280
divided by t.
00:03:04.280 --> 00:03:09.230
In other words, the rate of
renewals expected value of
00:03:09.230 --> 00:03:11.310
that goes to 1 over x-bar.
00:03:11.310 --> 00:03:14.950
Also, this is also leading to
the point of view that the
00:03:14.950 --> 00:03:19.860
rate of renewals is 1 over the
expected value of the expected
00:03:19.860 --> 00:03:22.480
in a renewal time.
00:03:22.480 --> 00:03:30.210
Now, why doesn't that mean
anything just by itself?
00:03:30.210 --> 00:03:34.850
Well, if you look at a set of
random variables, which is,
00:03:34.850 --> 00:03:39.390
for example, [? zero ?]
00:03:39.390 --> 00:03:44.860
most of the time and a very
large value with a very small
00:03:44.860 --> 00:03:45.630
probability.
00:03:45.630 --> 00:03:51.530
Suppose we look at a set of non
IID random variables where
00:03:51.530 --> 00:04:12.380
x sub i is equal to 0, the
probability 1 minus p, and is
00:04:12.380 --> 00:04:25.920
equal to some very large value,
say 1 over p, with
00:04:25.920 --> 00:04:29.770
probability p.
00:04:32.670 --> 00:04:34.800
Let's make it 1 over
p squared.
00:04:34.800 --> 00:04:38.140
Then, in that case, this doesn't
tell you anything the
00:04:38.140 --> 00:04:43.290
expected value of x sub i if we
look at x sub i as being n
00:04:43.290 --> 00:04:44.160
of t over t.
00:04:44.160 --> 00:04:49.020
The expected value of x
sub i gets very large.
00:04:49.020 --> 00:04:55.570
There's a worse thing that can
happen here though, which is
00:04:55.570 --> 00:04:59.330
suppose you have this situation
where n of t
00:04:59.330 --> 00:05:02.200
fluctuates within.
00:05:02.200 --> 00:05:18.130
And n of t typically looks like
t over x-bar plus the
00:05:18.130 --> 00:05:19.380
square root of t.
00:05:22.670 --> 00:05:28.970
Now, if you look at this,
I mean this is expected
00:05:28.970 --> 00:05:30.330
value of n of t.
00:05:33.520 --> 00:05:38.300
And as t wanders around,
expected value of n of t goes
00:05:38.300 --> 00:05:42.060
up linearly the way it's
supposed to, but it fluctuates
00:05:42.060 --> 00:05:45.500
around that point with
something like
00:05:45.500 --> 00:05:46.520
square root of t.
00:05:46.520 --> 00:05:51.660
If you divide by t, everything
is fine, but if you look at n
00:05:51.660 --> 00:05:56.660
of t over some large period of
time, it's going to move all
00:05:56.660 --> 00:05:58.630
over the place.
00:05:58.630 --> 00:06:02.460
So knowing that the expected
value of n of t over t is
00:06:02.460 --> 00:06:06.290
equal to 1 over x-bar really
doesn't tell you everything
00:06:06.290 --> 00:06:10.670
you'd like to know about
this kind of process.
00:06:10.670 --> 00:06:13.770
And finally, we have Blackwell's
theorem, which is
00:06:13.770 --> 00:06:16.420
getting closer to what
we'd like to know.
00:06:16.420 --> 00:06:21.520
What I'm trying to argue here is
that, in a sense, the major
00:06:21.520 --> 00:06:25.940
theorems, the things that you
really want to know, there's a
00:06:25.940 --> 00:06:30.860
strong law and the strong law
says with probability 1 all
00:06:30.860 --> 00:06:33.530
these paths behave
in the same way.
00:06:33.530 --> 00:06:39.340
And Blackwell's theorem, if
you take m of t as the
00:06:39.340 --> 00:06:42.140
expected value of n of t, that's
the thing which might
00:06:42.140 --> 00:06:45.530
wander all over the
place here.
00:06:45.530 --> 00:06:51.480
Blackwell says that if the
inter-renewal interval is an
00:06:51.480 --> 00:06:58.210
arithmetic random variable,
namely if it only takes on
00:06:58.210 --> 00:07:08.590
values that integers time some
span called lambda then the
00:07:08.590 --> 00:07:15.830
limit of the expected value of
n of t plus lambda, namely 1
00:07:15.830 --> 00:07:20.870
unit beyond where we start,
minus the expected value of t
00:07:20.870 --> 00:07:22.360
is lambda over x-bar.
00:07:22.360 --> 00:07:29.100
It says you only move up each
unit of time by a constant
00:07:29.100 --> 00:07:31.260
here divided by x-bar.
00:07:31.260 --> 00:07:35.620
If you look at this long term
behavior, you're still moving
00:07:35.620 --> 00:07:38.520
up at a rate of 1 over x-bar.
00:07:38.520 --> 00:07:43.750
But since you can only have
jumps at intervals of lambda,
00:07:43.750 --> 00:07:46.360
that's what causes that
lambda there.
00:07:46.360 --> 00:07:48.900
Most of the time when we try
to look at what's going on
00:07:48.900 --> 00:07:51.970
here, and for most of the
examples that we want to look
00:07:51.970 --> 00:07:54.700
at, we'll just set lambda
equal to 1.
00:07:54.700 --> 00:07:56.800
Especially when we look
at Markov chains.
00:07:56.800 --> 00:08:01.050
For Markov chains you only get
changes every step of the
00:08:01.050 --> 00:08:04.970
Markov chain, and you might as
well visualize steps of a
00:08:04.970 --> 00:08:08.032
Markov chain as being
at unit times
00:08:08.032 --> 00:08:09.259
AUDIENCE: I didn't quite
catch the point
00:08:09.259 --> 00:08:10.509
of the first example.
00:08:12.942 --> 00:08:14.415
What was the point of that?
00:08:14.415 --> 00:08:18.850
You said the elementary renewal
theorem [INAUDIBLE].
00:08:18.850 --> 00:08:21.620
PROFESSOR: Oh, the point in this
example is, you might not
00:08:21.620 --> 00:08:30.620
even have an expectation, but at
the same time this kind of
00:08:30.620 --> 00:08:37.460
situation is a situation where
n of t effectively moves up.
00:08:37.460 --> 00:08:40.990
Well, n of t over t effectively
moves up at a nice
00:08:40.990 --> 00:08:46.380
regular way, and you have a
strong law there, you have a
00:08:46.380 --> 00:08:50.410
weak law there, but you don't
have the situation you want.
00:08:50.410 --> 00:08:54.290
Namely, looking at an expected
value does not always tell you
00:08:54.290 --> 00:08:57.170
everything you'd like to know.
00:08:57.170 --> 00:09:00.850
I'm saying there's more to life
than expected values.
00:09:00.850 --> 00:09:04.390
And this one says the other
thing that if you look over
00:09:04.390 --> 00:09:06.980
time you're going to have things
wobble around quite a
00:09:06.980 --> 00:09:12.960
bit, and Blackwell's theorem
says, yes that wobbling around
00:09:12.960 --> 00:09:16.645
can occur over time, but it
doesn't happen very fast.
00:09:20.010 --> 00:09:21.720
The second one here
is kind of funny.
00:09:25.830 --> 00:09:29.410
This is probably why this is
called somebody's theorem
00:09:29.410 --> 00:09:32.650
instead of some lemma that
everybody's known since the
00:09:32.650 --> 00:09:33.710
17th century.
00:09:33.710 --> 00:09:37.780
Blackwell was still
doing research.
00:09:37.780 --> 00:09:42.220
About 10 years ago, I heard him
give a lecture in the math
00:09:42.220 --> 00:09:44.030
department here.
00:09:44.030 --> 00:09:50.620
He was not ancient at that time,
so this was probably
00:09:50.620 --> 00:09:53.960
then sometime around the '50s
or '60s, back when Blackwell
00:09:53.960 --> 00:09:55.620
did a lot of work on stochastic
00:09:55.620 --> 00:09:57.260
processes was being done.
00:09:57.260 --> 00:10:00.680
The reason why this result
is not trivial is because
00:10:00.680 --> 00:10:03.870
of this part here.
00:10:03.870 --> 00:10:09.520
If you have a renewal process
where some renewals occur, say
00:10:09.520 --> 00:10:13.440
with interval 1, and some
renewals occur with interval
00:10:13.440 --> 00:10:16.730
pi is a good example of it.
00:10:16.730 --> 00:10:20.200
Then as t gets larger and
larger, the set of times at
00:10:20.200 --> 00:10:24.250
which renewals can occur becomes
more and more dense.
00:10:24.250 --> 00:10:27.770
But along with it becoming more
and more dense, the jumps
00:10:27.770 --> 00:10:30.870
you get at each one of those
times gets smaller and
00:10:30.870 --> 00:10:36.050
smaller, and pretty soon n of t,
this expected value of n of
00:10:36.050 --> 00:10:41.070
t, is looking like something
which, if you don't have your
00:10:41.070 --> 00:10:44.590
glasses on, it looks like it's
going up exactly the way it
00:10:44.590 --> 00:10:48.320
should be going up, but if you
put your glasses on you see an
00:10:48.320 --> 00:10:50.950
enormous amount of fine
structure there.
00:10:50.950 --> 00:10:53.200
And the fine structure
never goes away.
00:10:53.200 --> 00:10:55.845
We'll talk more about that
in the next slide.
00:11:01.840 --> 00:11:06.140
You can really look at this in
a much simpler way, it's just
00:11:06.140 --> 00:11:08.810
that people like to look at
the expected number of
00:11:08.810 --> 00:11:12.380
renewals at different times.
00:11:12.380 --> 00:11:17.540
Since renewals can only occur
at time separated by lambda,
00:11:17.540 --> 00:11:20.580
and since you can't have two
renewals at a time, the only
00:11:20.580 --> 00:11:24.430
question is, do you get a
renewal at m lambda or don't
00:11:24.430 --> 00:11:27.450
you got a renewal at m lambda?
00:11:27.450 --> 00:11:31.600
And therefore this limit here,
a limit of m of t plus lambda
00:11:31.600 --> 00:11:35.590
minus m of t, is really the
question of whether you've
00:11:35.590 --> 00:11:38.450
gotten a renewal at
t plus lambda.
00:11:38.450 --> 00:11:42.520
So, you can rewrite this
condition as the limit of the
00:11:42.520 --> 00:11:46.180
probability of a renewal
at m lambda as equal
00:11:46.180 --> 00:11:48.420
to lambda over x-bar.
00:11:48.420 --> 00:11:52.410
What happens with the
scaling here?
00:11:52.410 --> 00:11:55.153
I mean, is this scaling right?
00:12:03.400 --> 00:12:05.140
Well, the expected time between
00:12:05.140 --> 00:12:08.260
renewals is 1 over x-bar.
00:12:08.260 --> 00:12:15.000
If I take this renewal process
and I scale it, measuring it
00:12:15.000 --> 00:12:17.430
in milliseconds instead
of seconds,
00:12:17.430 --> 00:12:20.140
what's going to happen?
00:12:20.140 --> 00:12:24.800
1 over x-bar is going to change
by a factor of 1,000.
00:12:24.800 --> 00:12:27.760
You really want lambda to change
by a factor of 1,000,
00:12:27.760 --> 00:12:32.140
because the probability of a
jump at one of these possible
00:12:32.140 --> 00:12:35.910
places for a jump is
still the same.
00:12:35.910 --> 00:12:39.310
So you need a lambda over
the x-bar here.
00:12:39.310 --> 00:12:42.510
If you model an arithmetic
renewal process as a Markov
00:12:42.510 --> 00:12:47.450
chain, starting in renewal state
0, this essentially says
00:12:47.450 --> 00:12:52.610
that p sub 0, 0 the probability
of going from 0 to
00:12:52.610 --> 00:12:56.840
0 and in steps is
some constant.
00:12:56.840 --> 00:12:59.810
I mean, calling pi 0, which is
just what we've always called
00:12:59.810 --> 00:13:04.820
it, and pi 0 has to be, in this
case, lambda over x-bar
00:13:04.820 --> 00:13:07.670
or 1 over x-bar.
00:13:07.670 --> 00:13:11.370
But what it's saying is that
this reaches a constant.
00:13:11.370 --> 00:13:13.570
You know, that's the hard
thing to prove.
00:13:13.570 --> 00:13:16.850
It's not hard to find
this number.
00:13:16.850 --> 00:13:21.300
It's hard to prove that it does
reach a limit, and that's
00:13:21.300 --> 00:13:24.730
the same thing that you're
trying to prove here, so this
00:13:24.730 --> 00:13:27.760
and this are really saying
the same thing.
00:13:27.760 --> 00:13:30.740
That the probability of renewal
at m lambda is lambda
00:13:30.740 --> 00:13:38.900
over x-bar, and expected value
of renewals at a given time is
00:13:38.900 --> 00:13:43.560
also 1 over x-bar.
00:13:43.560 --> 00:13:47.260
So, that's really the best
you could hope for.
00:13:50.090 --> 00:13:55.120
If you look at the
non-arithmetic case, I think
00:13:55.120 --> 00:13:58.120
one way of understanding it is
to take the results that we
00:13:58.120 --> 00:14:03.180
had before, which we stated as
part of Blackwell's theorem,
00:14:03.180 --> 00:14:05.380
which is this one.
00:14:05.380 --> 00:14:10.320
Divide both sides by delta
and see what happens.
00:14:10.320 --> 00:14:15.190
If you divide m of t plus delta
minus m of t by delta,
00:14:15.190 --> 00:14:17.520
it looks like you're trying
to go to a limit and get a
00:14:17.520 --> 00:14:18.800
derivative.
00:14:18.800 --> 00:14:20.050
We know we can't get
a derivative.
00:14:22.780 --> 00:14:25.950
So, what's going on then?
00:14:25.950 --> 00:14:34.070
It says for any delta that you
want the limit as t goes to
00:14:34.070 --> 00:14:40.010
infinity of this ratio has
to be 1 over x-bar.
00:14:40.010 --> 00:14:44.500
So, it says that if you take the
limit as delta goes to 0
00:14:44.500 --> 00:14:48.960
of this quantity here, you
still get 1 over x-bar.
00:14:48.960 --> 00:14:52.530
But this is a good example of
the case where you really
00:14:52.530 --> 00:14:56.350
can't interchange these
two limits.
00:14:56.350 --> 00:14:59.190
This is not a mathematical
fine point.
00:14:59.190 --> 00:15:03.200
I mean, this is something that
really cuts to the grain of
00:15:03.200 --> 00:15:06.630
what renewal processes
are all about.
00:15:06.630 --> 00:15:12.240
So, you really ought to think
through on your own why this
00:15:12.240 --> 00:15:17.320
makes sense when you take the
limit as delta goes to 0 after
00:15:17.320 --> 00:15:21.350
you take this limit, whereas if
you try to interchange the
00:15:21.350 --> 00:15:25.130
limits then you'd be trying to
say you're taking the limit as
00:15:25.130 --> 00:15:28.070
t approaches infinity of a
derivative here, and there
00:15:28.070 --> 00:15:29.870
isn't any derivative,
so there isn't any
00:15:29.870 --> 00:15:31.500
limit, so nothing works.
00:15:34.150 --> 00:15:39.790
Let's look a little bit at age
and duration at a particular
00:15:39.790 --> 00:15:41.670
value of t.
00:15:41.670 --> 00:15:45.200
Because we looked at age and
duration only in terms of
00:15:45.200 --> 00:15:49.590
sample paths, and we went to
the limit and we got very
00:15:49.590 --> 00:15:50.860
surprising results.
00:15:50.860 --> 00:15:58.370
We found out that the expected
age was the expected value of
00:15:58.370 --> 00:16:01.830
the inter-renewal interval
squared divided by the
00:16:01.830 --> 00:16:05.560
expected value of the
inter-renewal interval all
00:16:05.560 --> 00:16:10.900
divided by 2, which didn't
seem to make any sense.
00:16:10.900 --> 00:16:14.650
Because the inter-renewal time
was just this random variable
00:16:14.650 --> 00:16:18.790
x, the expected inter-renewal
time is x-bar.
00:16:18.790 --> 00:16:21.430
And yet you have this
enormous duration.
00:16:21.430 --> 00:16:24.370
And we sort of motivated this
in a number of ways.
00:16:24.370 --> 00:16:31.740
We drew some pictures and all of
that, but it didn't really
00:16:31.740 --> 00:16:33.680
come together.
00:16:33.680 --> 00:16:37.030
When we look at this way
I think it will come
00:16:37.030 --> 00:16:38.660
together for you.
00:16:38.660 --> 00:16:43.140
So let's assume an arithmetic
renewal process will span 1.
00:16:45.760 --> 00:16:50.290
I partly want to look at an
arithmetic process because
00:16:50.290 --> 00:16:53.130
it's much easier mathematically,
and partly
00:16:53.130 --> 00:16:57.160
because that's the kind of thing
we'll be interested in
00:16:57.160 --> 00:17:01.890
going to Markov chains with a
countable number of states.
00:17:01.890 --> 00:17:09.810
You're looking at an integer
value of t, z of t which is
00:17:09.810 --> 00:17:18.050
the age of time t is how long
it's been since the last
00:17:18.050 --> 00:17:19.319
renewal occurred.
00:17:19.319 --> 00:17:26.000
So the age at time t is t minus
s of 2, in this case,
00:17:26.000 --> 00:17:30.060
and in general it's
t minus s sub n.
00:17:30.060 --> 00:17:33.170
So that's what the age is.
00:17:33.170 --> 00:17:41.570
The duration x tilde of t, is
going to be the interval from
00:17:41.570 --> 00:17:47.570
this last renewal up to the
next renewal after t.
00:17:47.570 --> 00:17:49.850
So, I've written
this out here.
00:17:49.850 --> 00:17:57.690
x tilde of t is the
renewal time for
00:17:57.690 --> 00:18:00.850
the n of t-th renewal.
00:18:00.850 --> 00:18:07.300
n of t is this value here.
00:18:07.300 --> 00:18:11.350
So what we're doing is we're
looking at how long it takes
00:18:11.350 --> 00:18:13.080
to get from here to here.
00:18:13.080 --> 00:18:16.520
If I tell you that this interval
starts here, this
00:18:16.520 --> 00:18:20.590
distribution here, we'll have
the distribution of x of t.
00:18:20.590 --> 00:18:23.970
If all I tell you is we're
looking around t for the
00:18:23.970 --> 00:18:27.820
previous interval in the
next interval, then
00:18:27.820 --> 00:18:29.010
it's something different.
00:18:29.010 --> 00:18:30.980
How do we make sense
out of this?
00:18:30.980 --> 00:18:33.720
Well, this is the picture that
will make sense out it for
00:18:33.720 --> 00:18:38.290
you, so I hope this
makes sense.
00:18:38.290 --> 00:18:42.530
If you look at an integer value
of t, the z of t this
00:18:42.530 --> 00:18:46.090
age is going to be some
integer greater
00:18:46.090 --> 00:18:47.340
than or equal to 0.
00:18:47.340 --> 00:18:50.470
Is it possible for
the age to be 0?
00:18:50.470 --> 00:18:54.480
Yes, of course it is, because
t is some integer value.
00:18:54.480 --> 00:18:57.940
An arrival could have just come
in at time t, and then
00:18:57.940 --> 00:18:59.190
the age is 0.
00:19:03.940 --> 00:19:08.780
The time from 1 renewal until
the next has to be at least 1,
00:19:08.780 --> 00:19:12.310
because you only get renewals
in at integer times, and you
00:19:12.310 --> 00:19:14.890
can have two renewals
at the same time.
00:19:14.890 --> 00:19:19.200
So x tilde of t has to
be bigger than 1.
00:19:19.200 --> 00:19:22.690
How do we express this
in a different way?
00:19:22.690 --> 00:19:26.590
Well, let's let q sub j be the
probability this is an
00:19:26.590 --> 00:19:29.130
arrival at time j.
00:19:29.130 --> 00:19:33.430
If you want to write that down
with an equation, it's the sum
00:19:33.430 --> 00:19:37.910
over all n greater than or equal
to 1 of the probability
00:19:37.910 --> 00:19:41.710
that the n-th arrival
occurs at time j.
00:19:41.710 --> 00:19:52.160
In other words, q sub j is the
probability that the first
00:19:52.160 --> 00:19:55.600
arrival occurs at time j, plus
the probability that the
00:19:55.600 --> 00:19:58.570
second arrival occurs at time
j, plus the probability a
00:19:58.570 --> 00:20:00.420
third arrival occurs
at time j.
00:20:00.420 --> 00:20:03.060
Those are all disjoint events,
you can't have two
00:20:03.060 --> 00:20:05.660
of them be the same.
00:20:05.660 --> 00:20:10.940
So this q sub j is just the
probability that there is an
00:20:10.940 --> 00:20:12.920
arrival at time j.
00:20:12.920 --> 00:20:14.170
What's that a function of?
00:20:20.360 --> 00:20:23.180
It's a function of
the arrivals that
00:20:23.180 --> 00:20:25.675
occur before time j.
00:20:28.500 --> 00:20:30.110
And it's independent.
00:20:30.110 --> 00:20:32.670
That's how long the next
arrival takes.
00:20:32.670 --> 00:20:38.120
If I tell you, yes, there was an
arrival here conditional on
00:20:38.120 --> 00:20:40.600
the fact that there was
an arrival here,
00:20:40.600 --> 00:20:43.090
which arrival is it?
00:20:43.090 --> 00:20:45.430
I'm not talking about t or
anything else, I'm just
00:20:45.430 --> 00:20:48.170
saying, suppose we know there's
an arrival here.
00:20:48.170 --> 00:20:50.830
What you would look at would be
the previous arrivals, the
00:20:50.830 --> 00:20:54.040
previous inter-renewal
intervals, and in terms of
00:20:54.040 --> 00:20:58.220
that you would sort out what
that probability is.
00:20:58.220 --> 00:21:04.370
Then if there's an arrival here,
what's the probability
00:21:04.370 --> 00:21:09.810
that x tilde of t is equal to
this particular value here?
00:21:09.810 --> 00:21:13.150
Well, now this is where the
argument gets tricky.
00:21:13.150 --> 00:21:17.690
q sub j is a probability
of arrival of time j.
00:21:17.690 --> 00:21:23.960
What I maintain is the joint
probability that z of t is
00:21:23.960 --> 00:21:28.020
equal to i and this duration
here is equal to k.
00:21:28.020 --> 00:21:33.910
In other words, this is equal to
i here, this is equal to k.
00:21:33.910 --> 00:21:38.950
The probability of that this q
of t minus i, in other words,
00:21:38.950 --> 00:21:44.180
the probability that this is an
arrival at time t minus i,
00:21:44.180 --> 00:21:50.290
and the probability that this
inter-renewal interval here
00:21:50.290 --> 00:21:55.220
has duration k where the
restriction is that k has to
00:21:55.220 --> 00:21:57.860
be bigger than i.
00:21:57.860 --> 00:22:00.960
And that's what's fishy
about this.
00:22:00.960 --> 00:22:09.480
But it's perfectly correct,
just so long as I stick to
00:22:09.480 --> 00:22:15.980
values of i and k, where
k is bigger than i.
00:22:15.980 --> 00:22:20.050
i is the age here and
x hat is t, x
00:22:20.050 --> 00:22:23.570
tilde of t is the duration.
00:22:23.570 --> 00:22:28.030
And I can rewrite that joint
probability as a probability
00:22:28.030 --> 00:22:31.710
that we get an arrival here, and
that the probability the
00:22:31.710 --> 00:22:39.060
next arrival takes this
time x tilde of t.
00:22:39.060 --> 00:22:44.080
Now, what we've done with this
is to replace the idea of
00:22:44.080 --> 00:22:47.520
looking at duration with the
idea of looking at an
00:22:47.520 --> 00:22:50.190
inter-renewal interval.
00:22:50.190 --> 00:22:55.400
In other words, this probability
here, this is not
00:22:55.400 --> 00:23:01.050
the pmf of x tilde, this
is the pmf of x itself.
00:23:01.050 --> 00:23:04.700
This is the thing we hope we
understand at this point.
00:23:04.700 --> 00:23:06.610
This is what you learned about
on the first day of
00:23:06.610 --> 00:23:10.650
probability theory, when you
started taking 6041 or
00:23:10.650 --> 00:23:11.890
whatever you took.
00:23:11.890 --> 00:23:14.730
You started learning about
random variables, and if they
00:23:14.730 --> 00:23:19.500
were discrete random variables
they had pmf's, and if you had
00:23:19.500 --> 00:23:23.150
a sequence of IID random
variables, this
00:23:23.150 --> 00:23:25.710
was the pmf you had.
00:23:25.710 --> 00:23:30.050
So this joint probability here
is really these very simple
00:23:30.050 --> 00:23:33.460
things that you already
understand.
00:23:33.460 --> 00:23:38.190
But it's only equal to this for
i less than or equal to t,
00:23:38.190 --> 00:23:43.290
0 less than i, less or equal
to t, and k greater than i.
00:23:43.290 --> 00:23:48.030
This restriction here that how
far back you go to the last
00:23:48.030 --> 00:23:52.470
arrival can't be any more than
t, it's really sort of a
00:23:52.470 --> 00:23:53.590
technical restriction.
00:23:53.590 --> 00:23:55.760
I mean, you need it to
be accurate, but
00:23:55.760 --> 00:23:57.330
it's not very important.
00:23:57.330 --> 00:24:01.110
The important thing is that
k has to be bigger than i
00:24:01.110 --> 00:24:07.060
because otherwise you get this
arrival here and it's not
00:24:07.060 --> 00:24:10.810
beyond t and therefore it's
not the interval that
00:24:10.810 --> 00:24:13.990
surrounds t, it's some
other interval.
00:24:13.990 --> 00:24:19.510
So, that tells you what you
need to know about this.
00:24:22.350 --> 00:24:28.660
So, the joint probability is
z of t, and x tilde of t is
00:24:28.660 --> 00:24:35.110
equal to i and k, is this
conventional probability here.
00:24:35.110 --> 00:24:37.830
So what we know, there's a q sub
i as the probability of an
00:24:37.830 --> 00:24:42.350
arrival of j that's equal to
the expected value of an
00:24:42.350 --> 00:24:46.240
arrival at j and is equal to
the expected number of
00:24:46.240 --> 00:24:49.380
arrivals that have occurred.
00:24:49.380 --> 00:24:52.500
Excuse me, that i there
should be a j and this
00:24:52.500 --> 00:24:55.400
i should be a j.
00:24:55.400 --> 00:24:58.680
Oh wait, this is q sub i.
00:24:58.680 --> 00:25:02.830
This should be an i and
this should be an i.
00:25:02.830 --> 00:25:05.500
If I'm looking at it on my
computer and I don't have my
00:25:05.500 --> 00:25:09.160
glasses cleaned I can't tell the
difference between them.
00:25:09.160 --> 00:25:12.790
q sub i is the probability
of an arrival at i.
00:25:12.790 --> 00:25:17.620
The expectation of the number
of arrivals at i is equal to
00:25:17.620 --> 00:25:20.390
the expected number of arrivals
at i minus expected
00:25:20.390 --> 00:25:22.860
number of arrivals
at i minus 1.
00:25:22.860 --> 00:25:31.260
So Blackwell says, what i is
00:25:31.260 --> 00:25:33.110
asymptotically when i gets large.
00:25:33.110 --> 00:25:36.095
He says that the limit as
t goes to infinity.
00:25:41.880 --> 00:25:45.120
I think this thing's
a little weak.
00:25:45.120 --> 00:25:46.630
In fact, it's terribly weak.
00:25:51.750 --> 00:25:56.440
So the thing that Blackwell says
is that q sub i goes to a
00:25:56.440 --> 00:25:58.720
constant when i gets
very large.
00:26:01.330 --> 00:26:03.580
Namely he says that the
probability that you get an
00:26:03.580 --> 00:26:07.520
arrival at time i when
i is very large
00:26:07.520 --> 00:26:09.040
is just some constant.
00:26:09.040 --> 00:26:11.180
It doesn't depend
on i anymore.
00:26:11.180 --> 00:26:13.945
It wobbles around for a while,
and then it becomes constant.
00:26:16.850 --> 00:26:24.810
This limit here is then 1 over
x-bar, which is what q sub i
00:26:24.810 --> 00:26:29.590
is, times the probability
of k.
00:26:29.590 --> 00:26:34.630
Now, that's very weird because
what it says is that this
00:26:34.630 --> 00:26:39.730
probability does not depend
on the age at all.
00:26:39.730 --> 00:26:42.840
It's just a function
of the duration.
00:26:42.840 --> 00:26:46.250
As a function of the duration,
it's just p sub x of
00:26:46.250 --> 00:26:47.500
k divided by x-bar.
00:26:50.090 --> 00:26:55.250
If I go back to what we were
looking at before, what this
00:26:55.250 --> 00:27:00.810
says is if I take this interval
here and shift it
00:27:00.810 --> 00:27:06.030
around, those pairs of points
will have exactly the same
00:27:06.030 --> 00:27:07.650
probability asymptotically.
00:27:12.830 --> 00:27:17.610
Now, if you remember what we
talked about this very vague
00:27:17.610 --> 00:27:22.270
idea called random incidents,
if you look at a sample path
00:27:22.270 --> 00:27:26.820
and then you throw a dart at the
sample path and you say,
00:27:26.820 --> 00:27:31.990
what's the duration beyond t,
what's the duration before t,
00:27:31.990 --> 00:27:34.250
this is doing the same thing.
00:27:34.250 --> 00:27:37.940
But it's doing it in a very
exact and precise way.
00:27:37.940 --> 00:27:41.330
So, this is really the
mathematical way to look at
00:27:41.330 --> 00:27:44.250
this random instance idea.
00:27:44.250 --> 00:27:58.210
And what it's saying, if we go
back to where we were, that's
00:27:58.210 --> 00:28:03.860
telling us the joint probability
of age and
00:28:03.860 --> 00:28:10.750
duration is just a function of
the inter-renewal interval.
00:28:10.750 --> 00:28:14.590
It doesn't depend on
what i is at all.
00:28:14.590 --> 00:28:17.390
It doesn't depend on how long
it's been since the last
00:28:17.390 --> 00:28:20.185
renewal, it only depends
on the size of the
00:28:20.185 --> 00:28:22.310
inter-renewal interval.
00:28:22.310 --> 00:28:29.010
So then we say, OK why don't we
try to find the pmf of age?
00:28:29.010 --> 00:28:29.930
How do we do that?
00:28:29.930 --> 00:28:33.640
Well, we have this joint
distribution of z and x with
00:28:33.640 --> 00:28:37.210
these constraints on it. k has
to be bigger than i to make
00:28:37.210 --> 00:28:41.630
sure the inter-renewal
interval covers t.
00:28:41.630 --> 00:28:45.520
And you look at that formula
there, and you say, OK i
00:28:45.520 --> 00:28:54.430
travels from 0 up to k minus
1, and k is going to travel
00:28:54.430 --> 00:28:58.500
from i plus 1 all the
way up to infinity.
00:28:58.500 --> 00:29:03.750
So, if I try to look at what the
probability of the age is,
00:29:03.750 --> 00:29:09.062
it's going to be the sum of k
equals i plus 1 up to infinity
00:29:09.062 --> 00:29:11.280
of the joint probability.
00:29:11.280 --> 00:29:18.170
Because if I fix what the age
is I'm just looking at all
00:29:18.170 --> 00:29:22.980
possible durations from i plus
1 all the way up to infinity.
00:29:22.980 --> 00:29:27.770
So, the marginal for z of
t is this complimentary
00:29:27.770 --> 00:29:30.160
distribution function
evaluated at
00:29:30.160 --> 00:29:33.510
i divided by x-bar.
00:29:33.510 --> 00:29:36.340
That's a little hard
to visualize.
00:29:36.340 --> 00:29:39.690
But if we look at the duration,
it's a whole lot
00:29:39.690 --> 00:29:42.540
easier to visualize and
see what's going on.
00:29:42.540 --> 00:29:46.510
If you want to take the marginal
of the duration the
00:29:46.510 --> 00:29:52.650
pmf the duration is equal
to k, what is it?
00:29:52.650 --> 00:29:56.410
We have to average out over
z of t, which is the age.
00:29:56.410 --> 00:30:02.690
The age can be anything from
0 up to k minus 1 if we're
00:30:02.690 --> 00:30:05.140
looking at a particular
value of k.
00:30:05.140 --> 00:30:11.940
What that is again is this
diagram here, and x-hat of t
00:30:11.940 --> 00:30:17.760
will have a particular value of
k here, here, here, here,
00:30:17.760 --> 00:30:21.130
all the way up to here.
00:30:21.130 --> 00:30:24.480
So what we're doing is adding
up all those values, knowing
00:30:24.480 --> 00:30:27.870
exactly what the idea of random
instance was doing, but
00:30:27.870 --> 00:30:30.260
doing it in a nice clean way.
00:30:35.850 --> 00:30:39.630
Back to the ranch, it says that
the limit is t goes to
00:30:39.630 --> 00:30:47.690
infinity of x tilde is going
to be k times pmf of
00:30:47.690 --> 00:30:50.840
x divided by x-bar.
00:30:50.840 --> 00:30:54.020
You need to divide by x-bar so
that this is a probability
00:30:54.020 --> 00:30:57.750
mass function, and we actually
had the x-bar there all along.
00:30:57.750 --> 00:31:02.230
But if we sum this up over
k, what do we get?
00:31:02.230 --> 00:31:06.310
We get k times the pmf
of x, which is the
00:31:06.310 --> 00:31:08.600
expected value of x.
00:31:08.600 --> 00:31:12.010
So, that's all very nice.
00:31:12.010 --> 00:31:16.690
But what happens if we try to
find the expected value of
00:31:16.690 --> 00:31:18.540
this duration function here?
00:31:22.100 --> 00:31:23.750
That's really no harder,
the expected
00:31:23.750 --> 00:31:26.450
value of the duration.
00:31:26.450 --> 00:31:29.370
Here's one of the few places
where we use pmf's to do
00:31:29.370 --> 00:31:31.050
everything.
00:31:31.050 --> 00:31:36.890
It's a sum from k equals 1 to
infinity of k, which is the k
00:31:36.890 --> 00:31:45.920
we had before, k times
px of k over x-bar.
00:31:45.920 --> 00:31:50.240
So it's k times kpx of
k divided by x-bar.
00:31:50.240 --> 00:31:56.570
That's k squared times the
probability mass function of x
00:31:56.570 --> 00:32:00.460
divided by x-bar, which is the
expected value of x squared
00:32:00.460 --> 00:32:02.970
divided by the expected
value of x.
00:32:08.610 --> 00:32:12.770
Now, that sounds a little weird
also, but I think it's a
00:32:12.770 --> 00:32:20.300
whole lot less weird than the
argument using sample paths.
00:32:20.300 --> 00:32:24.110
I mean, you can look at this and
you can track down every
00:32:24.110 --> 00:32:27.560
little bit of it, and
you can be very sure
00:32:27.560 --> 00:32:29.150
of what it's saying.
00:32:29.150 --> 00:32:33.200
You can find the expected age
in sort of the same way, and
00:32:33.200 --> 00:32:37.300
it's done in the text, and it
comes out to be expected value
00:32:37.300 --> 00:32:42.730
of x squared divided by 2 times
the expected value of x,
00:32:42.730 --> 00:32:49.170
which is exactly what the
argument was for if you're
00:32:49.170 --> 00:32:52.510
looking at the sample
path point.
00:32:52.510 --> 00:32:55.787
But you lose a 1/2.
00:32:55.787 --> 00:32:57.510
AUDIENCE: When you look
at this [INAUDIBLE]
00:32:57.510 --> 00:32:59.104
it was for non-arithmetic
distributions.
00:32:59.104 --> 00:33:01.040
And here it's for arithmetic.
00:33:01.040 --> 00:33:04.912
So it's always true that the
[INAUDIBLE], even for
00:33:04.912 --> 00:33:06.162
arithmetic?
00:33:08.230 --> 00:33:08.510
PROFESSOR: Yes.
00:33:08.510 --> 00:33:13.890
The previous argument we went
through using sample paths was
00:33:13.890 --> 00:33:17.330
for arithmetic distribution, as
non-arithmetic was for any
00:33:17.330 --> 00:33:19.810
distribution at all.
00:33:19.810 --> 00:33:22.920
This argument here, you have
to distinguish between
00:33:22.920 --> 00:33:26.880
arithmetic and non-arithmetic,
and you have to distinguish
00:33:26.880 --> 00:33:30.930
because they act in very
different ways.
00:33:30.930 --> 00:33:34.700
I mean, if you read the text
and you read the section on
00:33:34.700 --> 00:33:38.810
non-arithmetic random variables,
you wind up with a
00:33:38.810 --> 00:33:43.340
lot of very tedious stuff that's
going on, a lot of
00:33:43.340 --> 00:33:46.230
really having to understand what
[INAUDIBLE] integration
00:33:46.230 --> 00:33:49.000
is all about.
00:33:49.000 --> 00:33:51.750
I mean, if you put in densities
it's all fine, but
00:33:51.750 --> 00:33:54.740
if you try to do it for these
weird distributions which are
00:33:54.740 --> 00:33:58.810
discrete but non-arithmetic,
you really have a lot of
00:33:58.810 --> 00:34:02.205
sweating to do to make sure that
any of this makes sense.
00:34:05.720 --> 00:34:07.300
Which is why I'm doing
it this way.
00:34:07.300 --> 00:34:09.790
But, when we do it this
way we get this extra
00:34:09.790 --> 00:34:11.880
factor of 1/2 here.
00:34:11.880 --> 00:34:14.000
And where does the factor
of 1/2 come from?
00:34:16.880 --> 00:34:20.719
Well, the factor of 1/2 comes
from the fact that all of this
00:34:20.719 --> 00:34:27.280
argument was looking at a t,
which has an integer value.
00:34:27.280 --> 00:34:31.350
Because we've made t the
integer, the age could be 0.
00:34:35.409 --> 00:34:39.380
If we make t non-integer, then
the age is going to be
00:34:39.380 --> 00:34:42.840
something between 0 and 1.
00:34:42.840 --> 00:34:50.409
And in fact, if we look at
what's happening, what we're
00:34:50.409 --> 00:34:58.760
going to find is that the age
of your integer values into
00:34:58.760 --> 00:35:04.230
the 6th, into the
6th plus one.
00:35:04.230 --> 00:35:06.590
And you know from the homework
that this might have to be 10
00:35:06.590 --> 00:35:09.620
to the 20th, but it doesn't
make any difference.
00:35:13.360 --> 00:35:20.380
And now the age here can be 0,
so the average age is going to
00:35:20.380 --> 00:35:24.100
be some value here.
00:35:24.100 --> 00:35:27.130
And as you look at larger and
larger t's, as you go from
00:35:27.130 --> 00:35:31.325
this integer to this integer,
the age is going to increase.
00:35:37.230 --> 00:35:40.710
And then at this point
there might be an
00:35:40.710 --> 00:35:42.040
arrival at this point.
00:35:42.040 --> 00:35:45.760
This is the probability of
an arrival at this point.
00:35:45.760 --> 00:35:47.015
Then it goes up again.
00:35:49.720 --> 00:35:53.730
It goes up exactly
the same way.
00:35:53.730 --> 00:35:56.180
It goes down.
00:35:56.180 --> 00:35:59.870
And the value it has at the
integer is its lowest value,
00:35:59.870 --> 00:36:02.730
because we're assuming that
when you look at the age
00:36:02.730 --> 00:36:06.080
you're looking at an age which
could be 0 if you look at this
00:36:06.080 --> 00:36:09.390
particular integer value.
00:36:09.390 --> 00:36:12.330
So, where's the problem?
00:36:12.330 --> 00:36:16.220
We take the sample average
over time, and what we're
00:36:16.220 --> 00:36:19.170
doing is we're averaging
over time here.
00:36:19.170 --> 00:36:23.890
So, you wind up with this point,
then you crawl up, then
00:36:23.890 --> 00:36:27.790
you go down again, you crawl
up, you go down again.
00:36:27.790 --> 00:36:31.340
And this average, which is the
thing we found before for the
00:36:31.340 --> 00:36:36.720
sample path average, is exactly
1/2 larger than what
00:36:36.720 --> 00:36:39.200
we found here.
00:36:39.200 --> 00:36:43.200
So, miraculously these two
numbers jive, which I find
00:36:43.200 --> 00:36:45.210
amazing after all this work.
00:36:45.210 --> 00:36:46.460
But they do.
00:36:48.480 --> 00:36:53.480
And the fact that you have the
1/2 there is sort of a check
00:36:53.480 --> 00:36:55.830
on the fact that people have
done the work right, because
00:36:55.830 --> 00:36:59.340
nobody would ever imagine that
was there until they actually
00:36:59.340 --> 00:37:01.450
went through it and found it.
00:37:05.750 --> 00:37:09.870
That, I think, explains why you
get these peculiar results
00:37:09.870 --> 00:37:11.815
for duration and for age.
00:37:11.815 --> 00:37:13.065
At least, I hope it does.
00:37:15.950 --> 00:37:18.755
Let's go on to countable-state
Markov chains.
00:37:22.480 --> 00:37:27.050
And the big change that occurs
when you go to countable-state
00:37:27.050 --> 00:37:32.130
chains is what you mean
by a recurring class.
00:37:32.130 --> 00:37:36.580
Before with finite state Markov
chains, we just blandly
00:37:36.580 --> 00:37:41.210
defined a recurrent state of
the state which had the
00:37:41.210 --> 00:37:44.760
property that wherever you
could go from that state,
00:37:44.760 --> 00:37:47.370
there was always some
way to get back.
00:37:47.370 --> 00:37:50.530
And since you had a finite
number of states, if there was
00:37:50.530 --> 00:37:53.220
some place you could go, you're
always going to get
00:37:53.220 --> 00:37:56.260
back to eventually, because
there was always some path
00:37:56.260 --> 00:37:59.700
that had some probability, and
you keep repeating the
00:37:59.700 --> 00:38:02.500
possibility of doing that.
00:38:02.500 --> 00:38:05.190
So, you didn't have to
worry about decision.
00:38:05.190 --> 00:38:09.860
Here you do have to
worry about it.
00:38:09.860 --> 00:38:16.850
It's particularly funny if you
look at a Markov chain model
00:38:16.850 --> 00:38:20.140
of a Bernoulli process.
00:38:20.140 --> 00:38:23.070
So, a Markov chain model of a
Bernoulli process, this is a
00:38:23.070 --> 00:38:28.730
countable-state process, you
start out at state 0.
00:38:28.730 --> 00:38:32.490
You flip your coin, your loaded
coin, which comes up 1,
00:38:32.490 --> 00:38:37.750
so the probability p and tails
with probability q, and if it
00:38:37.750 --> 00:38:41.430
comes up heads you
go to state 1.
00:38:41.430 --> 00:38:45.180
If it comes up tails, you
go to state minus 1.
00:38:45.180 --> 00:38:56.270
So this state here is really
the sum of the x of i's.
00:38:56.270 --> 00:39:03.910
In other words, it's the number
of successes minus the
00:39:03.910 --> 00:39:06.530
number of failures.
00:39:06.530 --> 00:39:09.360
So, we're looking at that sum
and we're looking at what
00:39:09.360 --> 00:39:12.540
happens as time gets
larger and larger.
00:39:12.540 --> 00:39:15.990
So, you go wandering around
here, moving up with
00:39:15.990 --> 00:39:20.110
probability p, moving down the
probability q, and you sort of
00:39:20.110 --> 00:39:23.060
get the idea that this
is going to diffuse
00:39:23.060 --> 00:39:26.090
after a long time.
00:39:26.090 --> 00:39:29.610
How do you do that
mathematically?
00:39:29.610 --> 00:39:32.600
And the text writes this
out carefully.
00:39:32.600 --> 00:39:37.010
I don't want to do that here,
because I think it's important
00:39:37.010 --> 00:39:41.690
for you in doing exercises and
things like that to start to
00:39:41.690 --> 00:39:45.580
see these arguments
automatically.
00:39:45.580 --> 00:39:52.940
So, the thing that's going to
happen as n gets large is that
00:39:52.940 --> 00:40:00.810
the variance of this state is
going to be n times the
00:40:00.810 --> 00:40:06.050
variance of a single up or
down random variable.
00:40:06.050 --> 00:40:11.090
You calculate that as 1 minus
p minus q squared, and if p
00:40:11.090 --> 00:40:16.060
and q are both strictly between
0 and 1, that variance
00:40:16.060 --> 00:40:19.170
is always positive.
00:40:19.170 --> 00:40:21.460
It can't be negative, you can't
have negative variance.
00:40:21.460 --> 00:40:25.970
But the important thing is it
keeps increasing within.
00:40:25.970 --> 00:40:30.690
So, it says that if you try to
draw a diagram, if you try to
00:40:30.690 --> 00:40:38.280
draw pmf of the probability that
s sub n is equal to 0, 1,
00:40:38.280 --> 00:40:42.400
2, 3, 4, and so forth, what
it's going to do is n gets
00:40:42.400 --> 00:40:43.460
very large.
00:40:43.460 --> 00:40:48.490
Because this pmf is going to
keep scaling outward, and it's
00:40:48.490 --> 00:40:51.770
going to scale outwards with
the square root of n.
00:40:54.520 --> 00:40:56.660
And we already know that it
starts to look Gaussian.
00:41:00.190 --> 00:41:06.550
And in particular what I mean
by looking Gaussian is that
00:41:06.550 --> 00:41:09.360
it's going to be a quantized
version of the Gaussian,
00:41:09.360 --> 00:41:15.720
because each time you increase
by 1 there's a probability
00:41:15.720 --> 00:41:18.770
that s sub n equals that
increased value.
00:41:18.770 --> 00:41:23.310
So you're spreading out the
individual values at each
00:41:23.310 --> 00:41:28.590
integer, have to spread,
have to come down.
00:41:28.590 --> 00:41:30.820
They can't do anything else.
00:41:30.820 --> 00:41:33.360
If you're spreading a
distribution out and it's an
00:41:33.360 --> 00:41:38.460
integer distribution, you can't
keep these values the
00:41:38.460 --> 00:41:42.080
same as they were before or
you would have a total
00:41:42.080 --> 00:41:45.440
probability, which would
be growing with the
00:41:45.440 --> 00:41:47.330
square root of n.
00:41:47.330 --> 00:41:50.890
And the probability that s sub
n is something is equal to 1.
00:41:50.890 --> 00:41:55.320
So as you spread out you have
this Gaussian distribution
00:41:55.320 --> 00:41:58.140
where the variance is growing
within and where the
00:41:58.140 --> 00:42:02.460
probability of each individual
value is going down as 1 over
00:42:02.460 --> 00:42:05.080
the square root of the n.
00:42:05.080 --> 00:42:08.520
Now, this is the sort of
argument which I hope can
00:42:08.520 --> 00:42:11.860
become automatic for you,
because this is the kind of
00:42:11.860 --> 00:42:17.100
thing you see in problems and
it's a big mess to analyze it.
00:42:17.100 --> 00:42:19.270
You have to go through a lot of
work, you have to be very
00:42:19.270 --> 00:42:22.660
careful about how you're
scaling things.
00:42:22.660 --> 00:42:25.860
And if you just look at this
saying, what's going to happen
00:42:25.860 --> 00:42:30.360
here as n gets large is that
this going to be like a
00:42:30.360 --> 00:42:34.260
Gaussian distribution, like
a quantized Gaussian
00:42:34.260 --> 00:42:37.260
distribution, and if it
goes out this way,
00:42:37.260 --> 00:42:40.090
it's got to go down.
00:42:40.090 --> 00:42:43.170
You can't go out without
going down.
00:42:43.170 --> 00:42:47.510
So, it says that no matter what
p is and no matter what q
00:42:47.510 --> 00:42:53.200
is, so long as neither of them
are 0, this thing is going to
00:42:53.200 --> 00:42:56.460
spread out, and the probability
that you're in any
00:42:56.460 --> 00:43:02.030
particular state after a long
time is going to 0.
00:43:02.030 --> 00:43:05.410
Now, that's not like the
behavior of finite state
00:43:05.410 --> 00:43:06.115
Markov chains.
00:43:06.115 --> 00:43:08.740
Because in the finite state
Markov chains, you have
00:43:08.740 --> 00:43:12.330
recurring classes, you have
steady state probabilities for
00:43:12.330 --> 00:43:14.070
those recurring classes.
00:43:14.070 --> 00:43:18.720
Here you don't have any steady
state probability distribution
00:43:18.720 --> 00:43:20.950
for this situation.
00:43:20.950 --> 00:43:26.630
Because in a steady state every
state has probability 0,
00:43:26.630 --> 00:43:30.960
and those probabilities don't
add to 1, so you just can't
00:43:30.960 --> 00:43:34.420
deal with this in any
sensible way.
00:43:34.420 --> 00:43:37.700
So, here we have a
countable-state Markov chain,
00:43:37.700 --> 00:43:42.630
which doesn't behave at all like
the finite state Markov
00:43:42.630 --> 00:43:45.220
chains we've looked at before.
00:43:45.220 --> 00:43:47.790
What's the period of this
chain, by the way?
00:43:47.790 --> 00:43:52.020
I said, here it's equal to
2, why is it equal to 2?
00:43:52.020 --> 00:43:57.020
Well, you start out at an even
number, s sub n is equal to 0,
00:43:57.020 --> 00:44:01.940
after one transition s sub n is
odd, after two transitions
00:44:01.940 --> 00:44:05.603
it's even again, so it keeps
oscillating between even and
00:44:05.603 --> 00:44:08.750
odd, which means there's
a period of two.
00:44:08.750 --> 00:44:13.170
So, we have a situation where
all states communicate.
00:44:13.170 --> 00:44:16.410
The definition of communicate is
the same as it was before.
00:44:16.410 --> 00:44:18.990
There's a path they get from
here to there, and there's a
00:44:18.990 --> 00:44:22.520
path to get back again.
00:44:22.520 --> 00:44:26.150
And classes are the same as
they were before, a set of
00:44:26.150 --> 00:44:30.960
states which all communicate
with each other, are all in
00:44:30.960 --> 00:44:36.250
the same class, and if i and j
communicate, then if j and k
00:44:36.250 --> 00:44:39.290
communicate, then there's a path
to get from i to j and a
00:44:39.290 --> 00:44:42.600
path to get from j to k, to the
path to get from i to k,
00:44:42.600 --> 00:44:44.780
to the path to get back again.
00:44:44.780 --> 00:44:55.713
So if i and j communicate and
j and k communicate, then i
00:44:55.713 --> 00:44:58.310
and k communicate also,
which is why you get
00:44:58.310 --> 00:45:01.400
classes out of this.
00:45:01.400 --> 00:45:03.140
So, we have classes.
00:45:03.140 --> 00:45:07.410
What we don't have is anything
relating to steady state
00:45:07.410 --> 00:45:10.330
probabilities, necessarily.
00:45:10.330 --> 00:45:13.480
So what we do?
00:45:13.480 --> 00:45:19.260
Well, another example, that's
called a birth-death chain, we
00:45:19.260 --> 00:45:23.270
will see this much more often
than the ugly thing I had in
00:45:23.270 --> 00:45:25.020
the last slide.
00:45:25.020 --> 00:45:27.660
The ugly thing I had in the
last slide is really much
00:45:27.660 --> 00:45:31.870
simpler, but it's harder
to analyze.
00:45:31.870 --> 00:45:33.120
It's gruesome.
00:45:35.380 --> 00:45:37.860
Well, as a matter of fact, we
probably ought to talk about
00:45:37.860 --> 00:45:40.950
this a little further.
00:45:40.950 --> 00:45:43.030
Because this does something.
00:45:43.030 --> 00:45:50.420
If you pick p equal to 1/2, this
thing is going to expand
00:45:50.420 --> 00:45:55.140
the center of it for any n is
going to stay at 0, so the
00:45:55.140 --> 00:45:57.020
probability of each
state is going to
00:45:57.020 --> 00:46:00.480
get smaller and smaller.
00:46:00.480 --> 00:46:05.610
What's the probability when you
start at 0 that you ever
00:46:05.610 --> 00:46:07.030
get back to 0 again?
00:46:11.680 --> 00:46:13.780
Now, that's not an easy
question, it's a question that
00:46:13.780 --> 00:46:16.830
takes a good deal of analysis
and a good deal of head
00:46:16.830 --> 00:46:18.200
scratching.
00:46:18.200 --> 00:46:21.720
But the answer is 1.
00:46:21.720 --> 00:46:24.740
There's a 0 steady state
probability of being in state
00:46:24.740 --> 00:46:29.050
0, but you wander away and
you eventually get back.
00:46:29.050 --> 00:46:32.450
There's always a path to come
back, and no matter how far
00:46:32.450 --> 00:46:36.020
away you get it's just as easy
to get back as it was to get
00:46:36.020 --> 00:46:39.350
out there, so it's certainly
plausible that you ought to
00:46:39.350 --> 00:46:42.870
get back with probability 1.
00:46:42.870 --> 00:46:46.660
And we will probably prove
that at some point.
00:46:46.660 --> 00:46:52.070
I'm not going to prove it today
because it would just be
00:46:52.070 --> 00:46:53.570
too much to prove for today.
00:46:56.490 --> 00:47:00.060
And, also, you will
need to have done
00:47:00.060 --> 00:47:02.750
a few of the exercises.
00:47:02.750 --> 00:47:07.660
The homework set for this week
is kind of easy, because
00:47:07.660 --> 00:47:11.010
you're probably all exhausted
after studying for the quiz.
00:47:11.010 --> 00:47:15.200
I hope the quiz was easy, but
even if it was easy you still
00:47:15.200 --> 00:47:16.720
had to study for it, and
that's the thing
00:47:16.720 --> 00:47:19.380
that takes the time.
00:47:19.380 --> 00:47:26.310
So this week we won't do too
much, but you will get some
00:47:26.310 --> 00:47:28.640
experience working with
a set of ideas.
00:47:32.140 --> 00:47:35.470
We want to go to this next
thing, which is called a
00:47:35.470 --> 00:47:38.210
birth-death chain.
00:47:38.210 --> 00:47:41.460
There are only non-negative
states here, 0, 1, 2.
00:47:41.460 --> 00:47:45.210
It looks exactly the same as the
previous chain, except you
00:47:45.210 --> 00:47:46.130
can't go negative.
00:47:46.130 --> 00:47:50.560
Whenever you go down to 0 you
bounce around and you then can
00:47:50.560 --> 00:47:52.890
go up again and come
back again as
00:47:52.890 --> 00:47:54.140
sort of like an accordion.
00:47:57.570 --> 00:48:02.270
If p is less than 1/2, you can
sort of imagine what's going
00:48:02.270 --> 00:48:02.880
to happen here.
00:48:02.880 --> 00:48:07.350
Every time you go up there's a
force pulling you back that's
00:48:07.350 --> 00:48:10.520
bigger than the force going up,
so you can imagine that
00:48:10.520 --> 00:48:13.960
you're going to stay clustered
pretty close to 0.
00:48:13.960 --> 00:48:17.460
If p is greater than 1/2, you're
going to go off into
00:48:17.460 --> 00:48:21.410
the wild blue yonder, and you're
never going to come
00:48:21.410 --> 00:48:28.890
back eventually, but if p is
equal to 1/2 then it's kind of
00:48:28.890 --> 00:48:31.410
hard to see what's going
to happen again.
00:48:31.410 --> 00:48:34.360
It's the same as the
situation before.
00:48:34.360 --> 00:48:40.600
You will come back with
probability 1 if p is 1/2, but
00:48:40.600 --> 00:48:43.980
you won't have any steady state
probability, which is a
00:48:43.980 --> 00:48:48.220
strange case, which we'll
talk about as we go on.
00:48:48.220 --> 00:48:50.980
You looked at the truncated case
of this in the homework,
00:48:50.980 --> 00:48:55.050
namely you looked at a case of
what happened if you just
00:48:55.050 --> 00:49:00.090
truncated this chain of some
value n and you found the
00:49:00.090 --> 00:49:02.310
steady state probabilities.
00:49:02.310 --> 00:49:07.100
And what you found was that if
p is equal to 1/2, the steady
00:49:07.100 --> 00:49:10.230
state probabilities
are uniform.
00:49:10.230 --> 00:49:14.760
If p is greater than 1/2, the
states over in this end have
00:49:14.760 --> 00:49:18.300
high probabilities, they travel
down geometrically to
00:49:18.300 --> 00:49:20.760
the states at this end.
00:49:20.760 --> 00:49:26.220
If p is less than 1/2, the
states are highly probable and
00:49:26.220 --> 00:49:28.760
these state go down
geometrically
00:49:28.760 --> 00:49:29.880
as you go out here.
00:49:29.880 --> 00:49:33.700
What do you think happens if you
take this truncated chain
00:49:33.700 --> 00:49:38.310
and then start moving n further
and further out?
00:49:38.310 --> 00:49:43.470
Well, if you have the nice case
where the probabilities
00:49:43.470 --> 00:49:48.930
are mostly clustered around 0,
then as you keep moving it out
00:49:48.930 --> 00:49:50.430
it doesn't make any
difference.
00:49:50.430 --> 00:49:53.070
In fact, you look at the answer
you've got and see what
00:49:53.070 --> 00:49:56.990
happens in the limit as you
take into infinity.
00:49:56.990 --> 00:50:01.970
But, if you look at the case
where p is greater than 1/2,
00:50:01.970 --> 00:50:05.070
then everything is clustered
up at the right n and every
00:50:05.070 --> 00:50:09.690
time you increase the number
of states by 1, blah.
00:50:09.690 --> 00:50:12.240
Everything goes to hell.
00:50:12.240 --> 00:50:17.080
Everything goes to hell unless
you realize that you just move
00:50:17.080 --> 00:50:19.820
everything up by 1, and then
you have the same case that
00:50:19.820 --> 00:50:21.320
you had before.
00:50:21.320 --> 00:50:23.910
So, you just keep moving
things up.
00:50:23.910 --> 00:50:28.750
But there isn't any steady
state, and as advertised
00:50:28.750 --> 00:50:30.140
things go to infinity.
00:50:30.140 --> 00:50:34.150
If you look at the case where
p is equal to 1/2, as you
00:50:34.150 --> 00:50:37.530
increase the number of states
what happens is all the states
00:50:37.530 --> 00:50:39.190
get less and less likely.
00:50:39.190 --> 00:50:42.660
You keep wandering around in
an aimless fashion, and
00:50:42.660 --> 00:50:44.170
nothing very interesting
happens.
00:50:49.680 --> 00:50:55.010
What we want to do, just to be
able to define recurrence, to
00:50:55.010 --> 00:50:59.680
mean that given that you start
off in some state i, there's a
00:50:59.680 --> 00:51:03.490
future return to state
i with probability 1.
00:51:03.490 --> 00:51:06.490
That's what recurrence should
mean, that's what recurrence
00:51:06.490 --> 00:51:08.320
means in English.
00:51:08.320 --> 00:51:10.630
Recurrence means
you come back.
00:51:10.630 --> 00:51:15.080
Since it's probabilistic, you
have to say, well we don't
00:51:15.080 --> 00:51:17.840
know when we're going to get
back, but we are going to get
00:51:17.840 --> 00:51:20.300
back eventually.
00:51:20.300 --> 00:51:23.030
It's what you say to a friend
you don't want to see.
00:51:23.030 --> 00:51:25.820
I'll see you sometime.
00:51:25.820 --> 00:51:28.900
And then it might be an infinite
expected time.
00:51:28.900 --> 00:51:32.130
But at least you said you're
going to make it back.
00:51:32.130 --> 00:51:35.710
We will see the birth-death
chain above is recurrent in
00:51:35.710 --> 00:51:40.970
this sense if p is less than
1/2, and it's not recurrent of
00:51:40.970 --> 00:51:45.500
p is greater than 1/2, and we're
clearly going to have to
00:51:45.500 --> 00:51:48.020
struggle a little bit to find
out what it is that
00:51:48.020 --> 00:51:50.320
p is equal to 1/2.
00:51:50.320 --> 00:51:53.500
And that's a strange case and
we'll call non-recurrent when
00:51:53.500 --> 00:51:56.260
we get to that point.
00:51:56.260 --> 00:51:59.220
We're going to use renewal
theory to study these
00:51:59.220 --> 00:52:05.200
recurrent chains, which is why
we did renewal theory first.
00:52:05.200 --> 00:52:08.460
But first we have to understand
first passage times
00:52:08.460 --> 00:52:09.560
a little better.
00:52:09.560 --> 00:52:12.080
We looked at first passage times
a little bit when we
00:52:12.080 --> 00:52:14.130
were dealing with
Markov chains.
00:52:14.130 --> 00:52:16.490
We looked at first passage
time by looking at the
00:52:16.490 --> 00:52:20.040
expected first passage time to
get from one state to another
00:52:20.040 --> 00:52:24.140
state, and we found a nice
clean way of doing that.
00:52:24.140 --> 00:52:27.150
We'll see how that relates to
this, but here instead of
00:52:27.150 --> 00:52:30.850
looking at the expected value
we want to find the
00:52:30.850 --> 00:52:35.230
probability that you have a
return after some particular
00:52:35.230 --> 00:52:37.624
period of time.
00:52:37.624 --> 00:52:38.995
AUDIENCE: So why did
[INAUDIBLE]?
00:52:41.740 --> 00:52:46.640
PROFESSOR: If p is greater
than 1/2, what's going to
00:52:46.640 --> 00:52:50.630
happen is you keep wandering
further and
00:52:50.630 --> 00:52:51.680
further off to the right.
00:52:51.680 --> 00:52:53.400
You can come back.
00:52:53.400 --> 00:52:56.790
There's a certain probability
that no matter how far you get
00:52:56.790 --> 00:52:59.280
out there, there's
a probability
00:52:59.280 --> 00:53:01.640
that you can get back.
00:53:01.640 --> 00:53:05.090
But there's also a probability
that you won't get back, that
00:53:05.090 --> 00:53:07.960
you keep getting bigger
and bigger.
00:53:07.960 --> 00:53:10.450
And this is not obvious.
00:53:10.450 --> 00:53:14.195
But it's something that we're
going to sort out as we go on.
00:53:14.195 --> 00:53:18.140
But it certainly is plausible
that there's a probability, a
00:53:18.140 --> 00:53:22.460
positive probability that
you'll never return.
00:53:22.460 --> 00:53:26.230
Because the further you go, the
harder it is to get back.
00:53:26.230 --> 00:53:28.670
And the drift is always
to the right.
00:53:28.670 --> 00:53:32.630
And since the drift is always to
the right, the further you
00:53:32.630 --> 00:53:38.160
get away, the less probable it
gets that you ever get back.
00:53:38.160 --> 00:53:40.880
And we will analyze this.
00:53:40.880 --> 00:53:43.085
I mean, for this case, it's
not very hard to analyze.
00:53:47.370 --> 00:53:48.620
Where were we?
00:53:50.770 --> 00:53:51.310
OK.
00:53:51.310 --> 00:53:55.640
The first pass each time
probability, we're going to
00:53:55.640 --> 00:53:58.960
call it f sub ij of n.
00:53:58.960 --> 00:54:02.440
This is the probability
given that you're in
00:54:02.440 --> 00:54:04.745
state i at time zero.
00:54:04.745 --> 00:54:08.960
It's the probability that you
reach state j for the first
00:54:08.960 --> 00:54:10.480
time a time n.
00:54:10.480 --> 00:54:15.340
It's not the probability that
you're in state j at time n,
00:54:15.340 --> 00:54:21.020
which is what we call
p sub i j to the n.
00:54:21.020 --> 00:54:23.440
It's just the probability
that you get there
00:54:23.440 --> 00:54:24.870
for the first time.
00:54:24.870 --> 00:54:26.120
At time n.
00:54:28.780 --> 00:54:34.120
If you remember, quick comment
about notation.
00:54:34.120 --> 00:54:49.170
We called p sub ij is the
probability that xj, xn equals
00:54:49.170 --> 00:54:53.990
j, given that x0 equals i.
00:54:53.990 --> 00:54:55.240
And f sub ij.
00:54:57.760 --> 00:55:02.820
And now the n is in parentheses
as the probability
00:55:02.820 --> 00:55:15.910
that xn is equal to j, given
that x0 equals i, and x1 x2,
00:55:15.910 --> 00:55:21.320
so forth are unequal to j.
00:55:21.320 --> 00:55:24.100
Now you see why I get confused
with i's and j's.
00:55:28.180 --> 00:55:31.200
The reason I'm using parentheses
here and using a
00:55:31.200 --> 00:55:34.580
subscript here is when we're
dealing with finite state
00:55:34.580 --> 00:55:39.200
Markov chains, it was just so
convenient to view this as the
00:55:39.200 --> 00:55:45.130
ij component of the matrix p
taken to the n-th power.
00:55:45.130 --> 00:55:49.430
And this is to remind you that
this is a matrix taken to the
00:55:49.430 --> 00:55:50.110
n-th power.
00:55:50.110 --> 00:55:52.170
And the you take
the ij element.
00:55:52.170 --> 00:55:54.310
There isn't any matrix
multiplication here.
00:55:54.310 --> 00:55:57.550
This is partly because we're
dealing with countable state
00:55:57.550 --> 00:55:59.210
and Markov chains here.
00:55:59.210 --> 00:56:03.330
But partly also because this is
an uglier thing that we're
00:56:03.330 --> 00:56:04.740
dealing with.
00:56:04.740 --> 00:56:07.960
But you can still
work this out.
00:56:07.960 --> 00:56:13.190
What's the probability that you
will be in state j for the
00:56:13.190 --> 00:56:17.630
first time at n, given
you're back in i?
00:56:17.630 --> 00:56:22.380
It's really the probability
that the sum of the
00:56:22.380 --> 00:56:25.780
probability that on the
first transition, you
00:56:25.780 --> 00:56:27.780
move from i to k.
00:56:27.780 --> 00:56:30.870
Some k unequal to j, because
if we're equal to j, we'd
00:56:30.870 --> 00:56:32.650
already be there, and we
wouldn't be looking at
00:56:32.650 --> 00:56:34.320
anything after that.
00:56:34.320 --> 00:56:37.260
So it's the probability
we move from i to k.
00:56:37.260 --> 00:56:42.430
And then we have n minus 1
transitions to get from k back
00:56:42.430 --> 00:56:44.040
to j for the first time.
00:56:47.060 --> 00:56:51.650
If you think you understand
this, most things like this,
00:56:51.650 --> 00:56:53.180
you can write them either way.
00:56:53.180 --> 00:56:56.450
You can look at the first
transition followed by n minus
00:56:56.450 --> 00:57:00.860
1 transitions after it, or you
can look at the n minus 1
00:57:00.860 --> 00:57:03.880
transitions first, followed
by the last transition.
00:57:03.880 --> 00:57:05.650
Here you can't do that.
00:57:05.650 --> 00:57:08.147
And you should look at it,
and figure out why.
00:57:08.147 --> 00:57:08.941
Yes?
00:57:08.941 --> 00:57:09.340
AUDIENCE: I'm sorry.
00:57:09.340 --> 00:57:12.852
I just was wondering if it's
the same to say that all of
00:57:12.852 --> 00:57:16.545
the [INAUDIBLE] are given, like
you would in [INAUDIBLE]?
00:57:16.545 --> 00:57:18.180
And [INAUDIBLE]
00:57:18.180 --> 00:57:20.546
all of them given just
[INAUDIBLE]?
00:57:20.546 --> 00:57:24.900
PROFESSOR: It's the probability
that all of the
00:57:24.900 --> 00:57:29.140
x1, x2, x3 and so forth
are not equal to j.
00:57:29.140 --> 00:57:31.370
AUDIENCE: Because here and
there, you did [INAUDIBLE]
00:57:31.370 --> 00:57:33.384
exactly the same question.
00:57:33.384 --> 00:57:35.610
In here, [INAUDIBLE]
00:57:35.610 --> 00:57:36.610
given [INAUDIBLE].
00:57:36.610 --> 00:57:39.770
And they're [INAUDIBLE]
the same?
00:57:39.770 --> 00:57:42.810
PROFESSOR: This is the
same as that, yes.
00:57:42.810 --> 00:57:45.840
Except it's not totally obvious
why it is, and I'm
00:57:45.840 --> 00:57:49.460
trying to explain why it is.
00:57:49.460 --> 00:57:52.820
I mean, you look at the first
transition you take.
00:57:52.820 --> 00:57:55.320
You start out in state i.
00:57:55.320 --> 00:57:59.690
The next place you go, if it's
state j, you're all through.
00:57:59.690 --> 00:58:02.340
And then you've gotten to
state j in one step, and
00:58:02.340 --> 00:58:03.940
that's the end of it.
00:58:03.940 --> 00:58:06.870
But if you don't get to state j
in one step, you get to some
00:58:06.870 --> 00:58:08.510
other state k.
00:58:08.510 --> 00:58:12.260
And then the question you ask
is what's the probability
00:58:12.260 --> 00:58:16.430
starting from state k that you
will reach j for the first
00:58:16.430 --> 00:58:19.070
time in n minus 1 steps?
00:58:23.170 --> 00:58:25.130
If you try to match up
these two equations,
00:58:25.130 --> 00:58:27.890
it's not at all obvious.
00:58:27.890 --> 00:58:32.100
If you look at what this first
transition probability means,
00:58:32.100 --> 00:58:34.820
then it's easy to get
this from that.
00:58:34.820 --> 00:58:36.660
And it's easy to get
this from that.
00:58:39.640 --> 00:58:42.300
But anyway, that's what it is.
00:58:42.300 --> 00:58:46.990
f sub ij of 1 is equal to pij,
and therefore this is equal to
00:58:46.990 --> 00:58:50.150
n greater than one.
00:58:50.150 --> 00:58:53.530
And you can use this recursion,
and you can go
00:58:53.530 --> 00:58:57.290
through and calculate all
of these n-th order
00:58:57.290 --> 00:59:00.600
probabilities, as some
[INAUDIBLE] is going to point
00:59:00.600 --> 00:59:02.160
out in just a minute.
00:59:02.160 --> 00:59:03.830
It takes a while to
do that for an
00:59:03.830 --> 00:59:06.580
infinite number of states.
00:59:06.580 --> 00:59:09.492
But we assume there's some nice
formula for doing it.
00:59:14.230 --> 00:59:16.680
I mean, this is a formula
you could in
00:59:16.680 --> 00:59:20.140
principal, compute it.
00:59:20.140 --> 00:59:23.320
OK, so here's the same
formula again.
00:59:23.320 --> 00:59:29.420
I want to relate that to the
probabilities of being in
00:59:29.420 --> 00:59:35.240
state j at time n, given the you
were in state i at time 0.
00:59:35.240 --> 00:59:38.540
Which by Chapman and
Kolmogorov is
00:59:38.540 --> 00:59:40.630
this equation here.
00:59:40.630 --> 00:59:41.750
That's the same.
00:59:41.750 --> 00:59:43.830
You go first to state
k, and then from
00:59:43.830 --> 00:59:47.130
kj, n minus one steps.
00:59:47.130 --> 00:59:50.390
Here you're summing over all k,
and you're summing over all
00:59:50.390 --> 00:59:55.240
k because in fact, if you get to
state j in the first step,
00:59:55.240 --> 00:59:57.910
you can still get to state
j again after n
00:59:57.910 --> 00:59:59.760
minus one more steps.
00:59:59.760 --> 01:00:02.800
Here you're only interested
in the first time you
01:00:02.800 --> 01:00:04.210
get to state j.
01:00:04.210 --> 01:00:08.590
Here you're interested in any
time you get to state j.
01:00:08.590 --> 01:00:12.910
I bring this up because remember
what you did when you
01:00:12.910 --> 01:00:19.450
solve that problem related to
rewards with a Markov chain?
01:00:19.450 --> 01:00:22.610
If you can think back to that.
01:00:22.610 --> 01:00:27.750
The way to solve the reward
problem, trying to find the
01:00:27.750 --> 01:00:32.790
first pass each time from sum
i to sum j was to just take
01:00:32.790 --> 01:00:37.830
all the outputs from state
j and remove them.
01:00:37.830 --> 01:00:40.540
If you look at this formula,
you'll see that that's exactly
01:00:40.540 --> 01:00:44.330
what we've done mathematically
by summing only over
01:00:44.330 --> 01:00:46.400
k unequal to j.
01:00:46.400 --> 01:00:50.930
Every time we get to j, we
terminate the whole thing, and
01:00:50.930 --> 01:00:52.360
we don't proceed any further.
01:00:52.360 --> 01:00:56.770
So this is just a mathematical
way of saying just a Markov
01:00:56.770 --> 01:01:02.010
chain by ripping all of the
outputs out of state j, and
01:01:02.010 --> 01:01:05.190
putting a self loop
in state j.
01:01:05.190 --> 01:01:07.620
OK, so this is really saying
the same sort of thing that
01:01:07.620 --> 01:01:10.570
that was saying, except that
was only giving us expected
01:01:10.570 --> 01:01:14.510
value, and this is giving
us the whole thing.
01:01:14.510 --> 01:01:20.550
Now, the next thing is we would
like to find what looks
01:01:20.550 --> 01:01:23.230
like a distribution function
for the same thing.
01:01:25.960 --> 01:01:28.020
This is the probability
of reaching j
01:01:28.020 --> 01:01:30.810
by time n or before.
01:01:30.810 --> 01:01:34.560
And the probability that you
reach state j by time n or
01:01:34.560 --> 01:01:39.910
before it's just the sum of
the probabilities that you
01:01:39.910 --> 01:01:44.150
reach state j for the first time
at some time m less than
01:01:44.150 --> 01:01:46.980
or equal to n.
01:01:46.980 --> 01:01:51.560
It's a probability of reaching
j by time n or before.
01:01:51.560 --> 01:01:55.800
If this limit now is equal
to 1, it means that with
01:01:55.800 --> 01:01:58.890
probability 1, you eventually
get there.
01:01:58.890 --> 01:01:59.884
Yes?
01:01:59.884 --> 01:02:00.872
AUDIENCE: I'm sorry,
Professor Gallager.
01:02:00.872 --> 01:02:04.824
I'm [INAUDIBLE] definition
of fij of n.
01:02:04.824 --> 01:02:06.800
So is it the thing you wrote
on the board, or is it the
01:02:06.800 --> 01:02:07.294
thing in the notes?
01:02:07.294 --> 01:02:08.776
Like, I don't see why
they're the same.
01:02:08.776 --> 01:02:11.450
Because the thing in the notes,
you're only given that
01:02:11.450 --> 01:02:13.600
x0 is equal to i.
01:02:13.600 --> 01:02:15.962
But the thing on the board,
you're given that x0 is equal
01:02:15.962 --> 01:02:18.704
to i, and you're given that
x1, x2, none of them
01:02:18.704 --> 01:02:20.320
are equal to j.
01:02:20.320 --> 01:02:22.142
PROFESSOR: Oh, I'm sorry.
01:02:22.142 --> 01:02:23.392
This is--
01:02:25.934 --> 01:02:27.943
you are absolutely right.
01:02:34.750 --> 01:02:39.675
And then given x0 equals i.
01:02:42.860 --> 01:02:44.110
Does that make sense now?
01:02:48.670 --> 01:02:52.240
I can't see in my mind if
that's equal to this.
01:02:52.240 --> 01:02:56.350
But if I sit there quietly and
look at it for five minutes, I
01:02:56.350 --> 01:02:58.445
realize why this is
equal to that.
01:02:58.445 --> 01:03:00.780
And that's why I write it down
wrong half the time.
01:03:06.500 --> 01:03:17.050
So, if this limit is equal to
1, it means I can define a
01:03:17.050 --> 01:03:21.110
random variable, which is the
amount of time that it takes
01:03:21.110 --> 01:03:26.690
to get to state j for
the first time.
01:03:26.690 --> 01:03:30.170
And that random variable is a
non defective random variable,
01:03:30.170 --> 01:03:37.160
because I always get to
state j eventually,
01:03:37.160 --> 01:03:38.410
starting from state i.
01:03:42.200 --> 01:03:46.930
Now, what was awkward about this
was the fact that I had
01:03:46.930 --> 01:03:49.340
to go through all these
probabilities before I could
01:03:49.340 --> 01:03:56.470
say let t sub ij be a random
variable, and let that random
01:03:56.470 --> 01:03:59.860
variable be the number of steps
that it takes to get to
01:03:59.860 --> 01:04:02.760
state j, starting in state i.
01:04:02.760 --> 01:04:05.370
And I couldn't do that because
it wasn't clear there was a
01:04:05.370 --> 01:04:06.880
random variable.
01:04:06.880 --> 01:04:10.740
If I said it was as effective
random variable, then it would
01:04:10.740 --> 01:04:13.050
be clear that there had
to be some sort of
01:04:13.050 --> 01:04:14.570
defective random variable.
01:04:14.570 --> 01:04:16.950
But then I'd have to deal with
the question of what do I
01:04:16.950 --> 01:04:19.150
really mean by defective
random variable?
01:04:19.150 --> 01:04:22.180
Incidentally, the notes does
not define what a defective
01:04:22.180 --> 01:04:24.170
random variable is.
01:04:24.170 --> 01:04:30.430
If you have a non-negative thing
that might be a random
01:04:30.430 --> 01:04:36.700
variable, the definition of a
defective random variable is
01:04:36.700 --> 01:04:43.370
that all these probabilities
exist, but the limit is not
01:04:43.370 --> 01:04:44.200
equal to 1.
01:04:44.200 --> 01:04:48.940
In other words, sometimes
the thing never happens.
01:04:48.940 --> 01:04:54.620
So that you can either look at
this as you have a thing like
01:04:54.620 --> 01:04:58.150
a random variable, but it
matched a lot of sample points
01:04:58.150 --> 01:04:59.670
into infinity.
01:04:59.670 --> 01:05:01.830
Or you can view it as
it maps a lot of
01:05:01.830 --> 01:05:03.660
sample points into nothing.
01:05:03.660 --> 01:05:07.840
But you still have a
distribution function for it.
01:05:07.840 --> 01:05:09.090
OK.
01:05:10.890 --> 01:05:14.800
But anyway, now we can talk
about a random variable, which
01:05:14.800 --> 01:05:20.950
is the time to get from state i
to state j, if in fact, it's
01:05:20.950 --> 01:05:23.700
certain that we're going
to get there.
01:05:23.700 --> 01:05:31.240
Now, if you start out with
a definition of this
01:05:31.240 --> 01:05:38.380
distribution function here, and
you play with this formula
01:05:38.380 --> 01:05:42.170
here, you play with that
formula, and that formula a
01:05:42.170 --> 01:05:47.010
little bit, you can rewrite
the formula for this
01:05:47.010 --> 01:05:51.520
distribution function like thing
in the following way.
01:05:51.520 --> 01:05:55.660
This is only different from the
thing we wrote before by
01:05:55.660 --> 01:05:57.520
the presence of pij here.
01:05:57.520 --> 01:06:02.430
Otherwise, little fij of n is
equal to just a sum over here
01:06:02.430 --> 01:06:03.740
without that.
01:06:03.740 --> 01:06:09.000
With this, you keep adding up,
and it keeps getting bigger.
01:06:09.000 --> 01:06:13.750
Why I want to talk about this,
it's sort of a detail.
01:06:13.750 --> 01:06:20.800
But this equation is always
satisfied by these
01:06:20.800 --> 01:06:24.260
distribution function
like things.
01:06:24.260 --> 01:06:29.070
But these equations do not
necessarily solve for these
01:06:29.070 --> 01:06:30.920
quantities.
01:06:30.920 --> 01:06:33.320
How do I see that?
01:06:33.320 --> 01:06:42.010
Well, if I plug one in for x sub
ij of n, and f sub ij of n
01:06:42.010 --> 01:06:47.130
minus 1 for all i and all
j, what do I get?
01:06:47.130 --> 01:06:52.570
I get one that's equal to p
sub ij plus the sum over k
01:06:52.570 --> 01:06:55.520
unequal to j of p
sub ik times 1.
01:06:55.520 --> 01:06:59.150
So I get 1 equals 1.
01:06:59.150 --> 01:07:05.270
So a solution to this equation
is that all of the fij's are
01:07:05.270 --> 01:07:06.520
equal to 1.
01:07:08.690 --> 01:07:10.260
Is this disturbing?
01:07:10.260 --> 01:07:14.360
Well, no, it shouldn't be,
because all the time we can
01:07:14.360 --> 01:07:17.190
write equations for things, and
the equations don't have a
01:07:17.190 --> 01:07:18.690
unique solution.
01:07:18.690 --> 01:07:21.980
And these equations don't
have a unique solution.
01:07:21.980 --> 01:07:24.280
We never said they did.
01:07:24.280 --> 01:07:29.050
But there is a theorem in the
notes, which says that if you
01:07:29.050 --> 01:07:33.120
look at all the solutions to
this equation, and you take
01:07:33.120 --> 01:07:36.340
the smallest solution, that
the smallest solution
01:07:36.340 --> 01:07:37.850
is the right one.
01:07:37.850 --> 01:07:41.800
In other words, the smallest
solution is the solution you
01:07:41.800 --> 01:07:44.440
get from doing it
this other way.
01:07:51.980 --> 01:07:55.550
I mean, this solution always
works, because you can always
01:07:55.550 --> 01:07:58.110
solve for these quantities.
01:07:58.110 --> 01:07:59.850
And you don't have to--
01:07:59.850 --> 01:08:02.380
I mean, you're just using
iteration, so all these
01:08:02.380 --> 01:08:03.630
quantities exist.
01:08:07.100 --> 01:08:09.090
OK.
01:08:09.090 --> 01:08:15.430
Now finally, these equations
for going from state i to
01:08:15.430 --> 01:08:21.520
state j also work for going
from state j to state j.
01:08:21.520 --> 01:08:25.580
If the probability of going
from state j to state j
01:08:25.580 --> 01:08:29.350
eventually is equal to 1, that's
what we said recurrent
01:08:29.350 --> 01:08:31.300
ought to mean.
01:08:31.300 --> 01:08:34.790
And now we have a precise
way of saying it.
01:08:37.479 --> 01:08:42.060
If f sub jj to infinity is equal
to 1, an eventual return
01:08:42.060 --> 01:08:45.880
from state j occurs with
probability 1, and the
01:08:45.880 --> 01:08:50.160
sequence of returns is a
sequence of renewal epochs in
01:08:50.160 --> 01:08:52.340
a renewal process.
01:08:52.340 --> 01:08:54.819
Nice, huh?
01:08:54.819 --> 01:08:57.460
I mean, when we looked at finite
state in Markov chains,
01:08:57.460 --> 01:09:00.220
we just sort of said this
and we're done with it.
01:09:00.220 --> 01:09:02.479
Because with finite state
and Markov chains,
01:09:02.479 --> 01:09:04.990
what else can happen?
01:09:04.990 --> 01:09:07.990
You start in a particular
state.
01:09:07.990 --> 01:09:12.359
If it's a recurrent state, you
keep hitting that state, and
01:09:12.359 --> 01:09:13.970
you keep coming back.
01:09:13.970 --> 01:09:15.600
And then you hit it again.
01:09:15.600 --> 01:09:19.600
And the amount of time from
one hit to the next hit is
01:09:19.600 --> 01:09:25.010
independent, as the next hit
to the next yet hit.
01:09:25.010 --> 01:09:29.859
It was clear that you had
a renewal process there.
01:09:29.859 --> 01:09:32.729
Here it's still clear that you
have a renewal process, if you
01:09:32.729 --> 01:09:34.490
can define this random
variable.
01:09:34.490 --> 01:09:34.779
Yes?
01:09:34.779 --> 01:09:37.648
AUDIENCE: When you say the
smallest set you mean sum
01:09:37.648 --> 01:09:39.290
across all terms of the
set, and whichever
01:09:39.290 --> 01:09:40.970
gives you the smallest.
01:09:40.970 --> 01:09:42.760
PROFESSOR: No, I mean
each of the values
01:09:42.760 --> 01:09:44.669
being as small as possible.
01:09:44.669 --> 01:09:47.689
I mean, it turns out that the
solutions are monotonic in
01:09:47.689 --> 01:09:49.779
that sense.
01:09:49.779 --> 01:09:52.569
You can find some solutions
where these are big and these
01:09:52.569 --> 01:09:54.950
are small, and others where
these are little,
01:09:54.950 --> 01:09:56.200
and these are big.
01:09:59.620 --> 01:10:00.050
OK.
01:10:00.050 --> 01:10:05.520
So now we know what this
distribution function is.
01:10:05.520 --> 01:10:08.160
We know what a recurrent
state is.
01:10:08.160 --> 01:10:09.670
And what do we do with it?
01:10:13.110 --> 01:10:16.120
Well, we say there's a random
variable with this
01:10:16.120 --> 01:10:18.310
distribution function.
01:10:18.310 --> 01:10:19.740
We keep doing things
like this.
01:10:19.740 --> 01:10:22.500
We keep getting more
and more abstract.
01:10:22.500 --> 01:10:26.140
I mean, instead of saying here's
a random variable and
01:10:26.140 --> 01:10:29.270
here's what its distribution
function is, we say if this
01:10:29.270 --> 01:10:34.670
was a random variable, then
state j is recurrent.
01:10:34.670 --> 01:10:37.280
It has this distribution
function.
01:10:37.280 --> 01:10:40.930
The renewal process of returns
to j, then, has inter renewal
01:10:40.930 --> 01:10:44.460
intervals with this distribution
function.
01:10:44.460 --> 01:10:48.530
As soon as we have this renewal
process, we can state
01:10:48.530 --> 01:10:53.930
this lemma, which are things
that we proved when we were
01:10:53.930 --> 01:10:55.840
talking about renewal theory.
01:10:55.840 --> 01:10:59.590
And let me try to explain why
each of them is true.
01:10:59.590 --> 01:11:02.750
Let's start out by assuming
that state j is recurrent.
01:11:02.750 --> 01:11:07.260
In other words, you have this
random variable, which is the
01:11:07.260 --> 01:11:11.810
amount of time it takes
to get back to j.
01:11:11.810 --> 01:11:19.880
If you get back to j with
probability one, then ask the
01:11:19.880 --> 01:11:24.220
question, how long does it take
to get back to j for the
01:11:24.220 --> 01:11:26.770
second time?
01:11:26.770 --> 01:11:29.900
Well, you have a random
variable, which is the time
01:11:29.900 --> 01:11:33.750
that it takes to get from j
to j for the first time.
01:11:33.750 --> 01:11:36.120
You add this to a random
variable, which is the amount
01:11:36.120 --> 01:11:39.080
of time to get from j to
j the second time.
01:11:39.080 --> 01:11:42.240
You add two random variables
together, and you
01:11:42.240 --> 01:11:44.070
get a random variable.
01:11:44.070 --> 01:11:47.520
In other words, the second
return is sure if
01:11:47.520 --> 01:11:49.600
the first one is.
01:11:49.600 --> 01:11:53.150
The second return occurs with
probability 1 if the first
01:11:53.150 --> 01:11:54.740
return occurs with
probability--
01:11:54.740 --> 01:11:57.830
I mean, you have a very long
wait for the first return.
01:11:57.830 --> 01:12:00.400
But after that very long
wait, you just have a
01:12:00.400 --> 01:12:02.150
very long wait again.
01:12:02.150 --> 01:12:05.180
But it's going to happen
eventually.
01:12:05.180 --> 01:12:08.250
And the third return happens
eventually, and the fourth
01:12:08.250 --> 01:12:10.960
return happens eventually.
01:12:10.960 --> 01:12:16.790
So this says that as t goes
to infinity, the number of
01:12:16.790 --> 01:12:22.800
returns up to time t is
going to be infinite.
01:12:22.800 --> 01:12:24.050
Very small rate, perhaps.
01:12:26.550 --> 01:12:30.130
If I look at the expected value
of the number of returns
01:12:30.130 --> 01:12:37.090
up until time t, that's going to
be equal to infinity, also.
01:12:37.090 --> 01:12:41.340
I can't think of any easy
way of arguing that.
01:12:41.340 --> 01:12:49.510
If I look at the sum over all n
of the probability that I'm
01:12:49.510 --> 01:12:53.400
in this state j at time n,
and I add up all those
01:12:53.400 --> 01:12:56.420
probabilities, what do I get?
01:12:56.420 --> 01:12:59.040
When I add up all those
probabilities, I'm adding up
01:12:59.040 --> 01:13:04.870
the expectations of having
a renewal at time n.
01:13:04.870 --> 01:13:07.120
By adding that up for all n.
01:13:07.120 --> 01:13:11.060
And this, in fact, is exactly
the same thing as this.
01:13:11.060 --> 01:13:16.840
So if you believe that, you
have to believe this also.
01:13:16.840 --> 01:13:20.480
And then we just go back and
repeat the thing if these
01:13:20.480 --> 01:13:23.552
things are not random
variables.
01:13:23.552 --> 01:13:25.516
Yes?
01:13:25.516 --> 01:13:29.444
AUDIENCE: I don't understand
what you mean [INAUDIBLE]?
01:13:29.444 --> 01:13:33.294
I thought that we
put [INAUDIBLE]
01:13:33.294 --> 01:13:34.710
2 and 3?
01:13:34.710 --> 01:13:35.660
[INAUDIBLE]?
01:13:35.660 --> 01:13:38.500
PROFESSOR: And given
3, we proof 4.
01:13:38.500 --> 01:13:41.120
AUDIENCE: Oh, [INAUDIBLE].
01:13:41.120 --> 01:13:42.370
PROFESSOR: Yeah.
01:13:44.660 --> 01:13:47.362
And we don't have time
to prove that--
01:13:47.362 --> 01:13:48.565
AUDIENCE: [INAUDIBLE].
01:13:48.565 --> 01:13:48.920
PROFESSOR: Yeah.
01:13:48.920 --> 01:13:50.170
OK.
01:13:52.340 --> 01:13:56.140
But in fact, it's not hard to
say suppose one doesn't occur,
01:13:56.140 --> 01:13:58.030
then, should the other
stuff occur.
01:14:00.790 --> 01:14:07.090
But none of these imply that the
expected value of t sub jj
01:14:07.090 --> 01:14:09.440
is finite, or infinite.
01:14:09.440 --> 01:14:12.600
You can always have random
variables, which are random
01:14:12.600 --> 01:14:18.150
variables, namely this is an
integer value random variable.
01:14:18.150 --> 01:14:20.740
It always takes on some
integer value.
01:14:20.740 --> 01:14:26.100
But the expected value of that
value might be infinite.
01:14:26.100 --> 01:14:28.750
I mean, you've seen all sorts of
random variables like that.
01:14:28.750 --> 01:14:34.400
You have a random variable where
the probability of j is
01:14:34.400 --> 01:14:37.380
some constant divided
by j squared.
01:14:37.380 --> 01:14:42.065
Now, you multiply j by 1 over 2
squared, you sum it over j,
01:14:42.065 --> 01:14:44.070
and you've got infinity.
01:14:44.070 --> 01:14:50.550
OK, so you might have these
random variables, which have
01:14:50.550 --> 01:14:54.810
an expected return time,
which is infinite.
01:14:54.810 --> 01:15:00.360
That's exactly what happens when
you look at these back
01:15:00.360 --> 01:15:01.670
right at the beginning
of today.
01:15:05.990 --> 01:15:07.240
1, 2, 4.
01:15:09.710 --> 01:15:13.705
I'm going to look at this kind
of chain here, and I set p
01:15:13.705 --> 01:15:15.990
equal to one half.
01:15:15.990 --> 01:15:21.290
What we said was that you just
disperse here the probability
01:15:21.290 --> 01:15:24.980
that you're going to be in any
state after a long time goes
01:15:24.980 --> 01:15:30.060
to zero, but you have to
get back eventually.
01:15:30.060 --> 01:15:33.390
And if you look at that
condition carefully, and you
01:15:33.390 --> 01:15:36.450
put it together with all the
things we found out about
01:15:36.450 --> 01:15:39.420
Markov chains, you realize that
the expected time to get
01:15:39.420 --> 01:15:43.740
back has to be infinite
for this case.
01:15:43.740 --> 01:15:47.540
And the same for the next
example we looked at.
01:15:47.540 --> 01:15:51.390
If you look at p equals 1/2, you
don't have a steady state
01:15:51.390 --> 01:15:52.680
probability.
01:15:52.680 --> 01:15:55.620
Because you don't have a steady
state probability, the
01:15:55.620 --> 01:16:00.460
expected time to get back, the
expected recurrence time has
01:16:00.460 --> 01:16:04.470
an infinite expected value.
01:16:04.470 --> 01:16:10.020
The expected recurrence time
has to be 1 over the
01:16:10.020 --> 01:16:11.560
probability of the state.
01:16:11.560 --> 01:16:13.540
So the probability of
the state is 0.
01:16:13.540 --> 01:16:16.185
The expected return time
has to be infinite.
01:16:19.640 --> 01:16:20.890
So, where were we?
01:16:24.630 --> 01:16:28.110
Well, two states are in
the same class if they
01:16:28.110 --> 01:16:30.790
communicate.
01:16:30.790 --> 01:16:33.830
Same as for finite
state chains.
01:16:33.830 --> 01:16:36.496
That's the same argument we
gave, and you get from there
01:16:36.496 --> 01:16:38.320
to there, and there to there.
01:16:38.320 --> 01:16:40.840
And you get from here to there,
and here to here, then
01:16:40.840 --> 01:16:42.570
you get from here to there,
and here to there.
01:16:42.570 --> 01:16:44.640
OK.
01:16:44.640 --> 01:16:47.910
If states i and j are in the
same class, then either both
01:16:47.910 --> 01:16:52.040
are recurrent, or both
are transient.
01:16:52.040 --> 01:16:53.930
Which means not recurrent.
01:16:53.930 --> 01:16:58.570
If j is recurrent, then we've
already found that this sum is
01:16:58.570 --> 01:17:00.460
equal to infinity.
01:17:00.460 --> 01:17:06.040
And then, oh, I have to explain
this a little bit.
01:17:06.040 --> 01:17:10.345
What I'm trying to show you is
that if j is recurrent, then i
01:17:10.345 --> 01:17:11.820
has to be recurrent.
01:17:11.820 --> 01:17:14.270
And I know that j is
recurrent if this
01:17:14.270 --> 01:17:16.650
sum is equal to infinity.
01:17:16.650 --> 01:17:20.140
And if this sum is equal to
infinity, let's look at how we
01:17:20.140 --> 01:17:24.390
can get from i back
to i in n steps.
01:17:24.390 --> 01:17:28.700
Since i and j communicate,
there's some path of some
01:17:28.700 --> 01:17:34.340
number of steps, say m, which
guesses from i to j.
01:17:34.340 --> 01:17:38.860
There's also some path of say,
length l which gets us
01:17:38.860 --> 01:17:41.790
from j back to k.
01:17:41.790 --> 01:17:44.820
And if there's this path, and
there's this path, and then I
01:17:44.820 --> 01:17:50.670
sum this over all k, that I get
is a lower bound to that.
01:17:50.670 --> 01:17:54.060
I first go to j, and then I
spend an infinite number of
01:17:54.060 --> 01:17:58.200
states constantly coming
back to j.
01:17:58.200 --> 01:18:01.260
And then I finally
go back to i.
01:18:01.260 --> 01:18:08.140
And what that says is that this
sum of p sub i, i to the
01:18:08.140 --> 01:18:11.992
n, I mean, this is just one
set of paths which gets us
01:18:11.992 --> 01:18:13.690
from i to i in n steps.
01:18:23.600 --> 01:18:28.630
If state j is recurrent, then
t sub jj might or might not
01:18:28.630 --> 01:18:29.880
have a finite expectation.
01:18:39.560 --> 01:18:44.090
All I want to say here is that
if t sub jj, the time to get
01:18:44.090 --> 01:18:49.170
from state j back to state j has
an infinite expectation,
01:18:49.170 --> 01:18:51.100
then you call state j.
01:18:51.100 --> 01:18:54.160
No recurrent instead of
regular recurrent.
01:18:54.160 --> 01:18:58.100
As we were just saying, if that
expected recurrence time
01:18:58.100 --> 01:19:02.570
is infinite, then the steady
state probability is going to
01:19:02.570 --> 01:19:07.040
be zero, which says that if
something is no recurrent,
01:19:07.040 --> 01:19:10.530
it's going to be even a very,
very different way from when
01:19:10.530 --> 01:19:13.830
it's positive recurrent, which
is when the return time has a
01:19:13.830 --> 01:19:15.218
finite value.
01:19:15.218 --> 01:19:16.468
AUDIENCE: [INAUDIBLE]?
01:19:18.634 --> 01:19:20.590
PROFESSOR: Yes.
01:19:20.590 --> 01:19:23.460
Which means there isn't a steady
state probability.
01:19:23.460 --> 01:19:25.550
To have a steady state
probability, you want them all
01:19:25.550 --> 01:19:28.040
to add up to 1.
01:19:28.040 --> 01:19:29.530
So yes, they're all zero.
01:19:29.530 --> 01:19:33.100
And formally, there isn't a
steady state probability.
01:19:33.100 --> 01:19:35.220
OK, thank you.
01:19:35.220 --> 01:19:36.470
We will--