WEBVTT 00:00:00.040 --> 00:00:02.460 The following content is provided under a Creative 00:00:02.460 --> 00:00:03.870 Commons license. 00:00:03.870 --> 00:00:06.910 Your support will help MIT OpenCourseWare continue to 00:00:06.910 --> 00:00:10.560 offer high quality educational resources for free. 00:00:10.560 --> 00:00:13.460 To make a donation or view additional materials from 00:00:13.460 --> 00:00:19.290 hundreds of MIT courses, visit MIT OpenCourseWare at 00:00:19.290 --> 00:00:20.540 ocw.mit.edu. 00:00:22.170 --> 00:00:23.620 PROFESSOR: So let's get started. 00:00:23.620 --> 00:00:27.900 With the quiz out of the way we can now move ahead again. 00:00:30.560 --> 00:00:33.500 I want to talk a little bit about the 00:00:33.500 --> 00:00:36.350 major renewal theorem. 00:00:36.350 --> 00:00:39.740 Partly review, but partly it's something that we need because 00:00:39.740 --> 00:00:44.530 we're going on to talk about countable-state Markov chains, 00:00:44.530 --> 00:00:51.170 and almost all the analysis there is based on what happens 00:00:51.170 --> 00:00:55.640 when you're dealing with renewal processes. 00:00:55.640 --> 00:00:58.540 Especially discrete renewal processes that are the 00:00:58.540 --> 00:01:03.470 renewals when you have recurrences from one state to 00:01:03.470 --> 00:01:07.450 the same state at some later point. 00:01:07.450 --> 00:01:12.810 So, in order to do that, we also have to talk a little bit 00:01:12.810 --> 00:01:18.000 about age and duration at a given time instead of in terms 00:01:18.000 --> 00:01:25.110 of the sample average, because that's sort of essential also. 00:01:25.110 --> 00:01:27.630 And it gives you a nice interpretation of why these 00:01:27.630 --> 00:01:33.040 peculiar things happen about duration being so much longer 00:01:33.040 --> 00:01:37.880 than it seems like it should be, and things of that sort. 00:01:37.880 --> 00:01:40.800 We'll probably spend close to half the lecture dealing with 00:01:40.800 --> 00:01:44.080 that, and the other half dealing with countable-state 00:01:44.080 --> 00:01:45.330 Markov chains. 00:01:48.070 --> 00:01:51.010 We really have three major theorems dealing 00:01:51.010 --> 00:01:52.260 with renewal processes. 00:01:55.440 --> 00:01:59.350 One of them is a sample path time average, which we've 00:01:59.350 --> 00:02:01.960 called the strong law for renewals. 00:02:01.960 --> 00:02:07.450 It says that if you look at individual sample paths, it 00:02:07.450 --> 00:02:15.040 says that the limit as an individual sample path, m as 00:02:15.040 --> 00:02:19.640 the number of arrivals over that sample path if you look 00:02:19.640 --> 00:02:23.540 at the limit from 0 to infinity of the number of 00:02:23.540 --> 00:02:28.320 arrivals divided by time, that's 1 over x-bar. 00:02:28.320 --> 00:02:32.340 And the set of sample paths for which that's true has 00:02:32.340 --> 00:02:33.450 probability 1. 00:02:33.450 --> 00:02:37.660 That's what that statement is supposed to say. 00:02:37.660 --> 00:02:42.050 The next one is the elementary renewal theorem. 00:02:42.050 --> 00:02:47.200 When you look at it carefully, that doesn't say anything, or 00:02:47.200 --> 00:02:49.390 hardly anything. 00:02:49.390 --> 00:02:53.610 All it says is that if you look at the limit as t goes to 00:02:53.610 --> 00:02:57.400 infinity, and you also take the expected value, the 00:02:57.400 --> 00:03:03.350 expected value is the number of renewals over a period t 00:03:03.350 --> 00:03:04.280 divided by t. 00:03:04.280 --> 00:03:09.230 In other words, the rate of renewals expected value of 00:03:09.230 --> 00:03:11.310 that goes to 1 over x-bar. 00:03:11.310 --> 00:03:14.950 Also, this is also leading to the point of view that the 00:03:14.950 --> 00:03:19.860 rate of renewals is 1 over the expected value of the expected 00:03:19.860 --> 00:03:22.480 in a renewal time. 00:03:22.480 --> 00:03:30.210 Now, why doesn't that mean anything just by itself? 00:03:30.210 --> 00:03:34.850 Well, if you look at a set of random variables, which is, 00:03:34.850 --> 00:03:39.390 for example, [? zero ?] 00:03:39.390 --> 00:03:44.860 most of the time and a very large value with a very small 00:03:44.860 --> 00:03:45.630 probability. 00:03:45.630 --> 00:03:51.530 Suppose we look at a set of non IID random variables where 00:03:51.530 --> 00:04:12.380 x sub i is equal to 0, the probability 1 minus p, and is 00:04:12.380 --> 00:04:25.920 equal to some very large value, say 1 over p, with 00:04:25.920 --> 00:04:29.770 probability p. 00:04:32.670 --> 00:04:34.800 Let's make it 1 over p squared. 00:04:34.800 --> 00:04:38.140 Then, in that case, this doesn't tell you anything the 00:04:38.140 --> 00:04:43.290 expected value of x sub i if we look at x sub i as being n 00:04:43.290 --> 00:04:44.160 of t over t. 00:04:44.160 --> 00:04:49.020 The expected value of x sub i gets very large. 00:04:49.020 --> 00:04:55.570 There's a worse thing that can happen here though, which is 00:04:55.570 --> 00:04:59.330 suppose you have this situation where n of t 00:04:59.330 --> 00:05:02.200 fluctuates within. 00:05:02.200 --> 00:05:18.130 And n of t typically looks like t over x-bar plus the 00:05:18.130 --> 00:05:19.380 square root of t. 00:05:22.670 --> 00:05:28.970 Now, if you look at this, I mean this is expected 00:05:28.970 --> 00:05:30.330 value of n of t. 00:05:33.520 --> 00:05:38.300 And as t wanders around, expected value of n of t goes 00:05:38.300 --> 00:05:42.060 up linearly the way it's supposed to, but it fluctuates 00:05:42.060 --> 00:05:45.500 around that point with something like 00:05:45.500 --> 00:05:46.520 square root of t. 00:05:46.520 --> 00:05:51.660 If you divide by t, everything is fine, but if you look at n 00:05:51.660 --> 00:05:56.660 of t over some large period of time, it's going to move all 00:05:56.660 --> 00:05:58.630 over the place. 00:05:58.630 --> 00:06:02.460 So knowing that the expected value of n of t over t is 00:06:02.460 --> 00:06:06.290 equal to 1 over x-bar really doesn't tell you everything 00:06:06.290 --> 00:06:10.670 you'd like to know about this kind of process. 00:06:10.670 --> 00:06:13.770 And finally, we have Blackwell's theorem, which is 00:06:13.770 --> 00:06:16.420 getting closer to what we'd like to know. 00:06:16.420 --> 00:06:21.520 What I'm trying to argue here is that, in a sense, the major 00:06:21.520 --> 00:06:25.940 theorems, the things that you really want to know, there's a 00:06:25.940 --> 00:06:30.860 strong law and the strong law says with probability 1 all 00:06:30.860 --> 00:06:33.530 these paths behave in the same way. 00:06:33.530 --> 00:06:39.340 And Blackwell's theorem, if you take m of t as the 00:06:39.340 --> 00:06:42.140 expected value of n of t, that's the thing which might 00:06:42.140 --> 00:06:45.530 wander all over the place here. 00:06:45.530 --> 00:06:51.480 Blackwell says that if the inter-renewal interval is an 00:06:51.480 --> 00:06:58.210 arithmetic random variable, namely if it only takes on 00:06:58.210 --> 00:07:08.590 values that integers time some span called lambda then the 00:07:08.590 --> 00:07:15.830 limit of the expected value of n of t plus lambda, namely 1 00:07:15.830 --> 00:07:20.870 unit beyond where we start, minus the expected value of t 00:07:20.870 --> 00:07:22.360 is lambda over x-bar. 00:07:22.360 --> 00:07:29.100 It says you only move up each unit of time by a constant 00:07:29.100 --> 00:07:31.260 here divided by x-bar. 00:07:31.260 --> 00:07:35.620 If you look at this long term behavior, you're still moving 00:07:35.620 --> 00:07:38.520 up at a rate of 1 over x-bar. 00:07:38.520 --> 00:07:43.750 But since you can only have jumps at intervals of lambda, 00:07:43.750 --> 00:07:46.360 that's what causes that lambda there. 00:07:46.360 --> 00:07:48.900 Most of the time when we try to look at what's going on 00:07:48.900 --> 00:07:51.970 here, and for most of the examples that we want to look 00:07:51.970 --> 00:07:54.700 at, we'll just set lambda equal to 1. 00:07:54.700 --> 00:07:56.800 Especially when we look at Markov chains. 00:07:56.800 --> 00:08:01.050 For Markov chains you only get changes every step of the 00:08:01.050 --> 00:08:04.970 Markov chain, and you might as well visualize steps of a 00:08:04.970 --> 00:08:08.032 Markov chain as being at unit times 00:08:08.032 --> 00:08:09.259 AUDIENCE: I didn't quite catch the point 00:08:09.259 --> 00:08:10.509 of the first example. 00:08:12.942 --> 00:08:14.415 What was the point of that? 00:08:14.415 --> 00:08:18.850 You said the elementary renewal theorem [INAUDIBLE]. 00:08:18.850 --> 00:08:21.620 PROFESSOR: Oh, the point in this example is, you might not 00:08:21.620 --> 00:08:30.620 even have an expectation, but at the same time this kind of 00:08:30.620 --> 00:08:37.460 situation is a situation where n of t effectively moves up. 00:08:37.460 --> 00:08:40.990 Well, n of t over t effectively moves up at a nice 00:08:40.990 --> 00:08:46.380 regular way, and you have a strong law there, you have a 00:08:46.380 --> 00:08:50.410 weak law there, but you don't have the situation you want. 00:08:50.410 --> 00:08:54.290 Namely, looking at an expected value does not always tell you 00:08:54.290 --> 00:08:57.170 everything you'd like to know. 00:08:57.170 --> 00:09:00.850 I'm saying there's more to life than expected values. 00:09:00.850 --> 00:09:04.390 And this one says the other thing that if you look over 00:09:04.390 --> 00:09:06.980 time you're going to have things wobble around quite a 00:09:06.980 --> 00:09:12.960 bit, and Blackwell's theorem says, yes that wobbling around 00:09:12.960 --> 00:09:16.645 can occur over time, but it doesn't happen very fast. 00:09:20.010 --> 00:09:21.720 The second one here is kind of funny. 00:09:25.830 --> 00:09:29.410 This is probably why this is called somebody's theorem 00:09:29.410 --> 00:09:32.650 instead of some lemma that everybody's known since the 00:09:32.650 --> 00:09:33.710 17th century. 00:09:33.710 --> 00:09:37.780 Blackwell was still doing research. 00:09:37.780 --> 00:09:42.220 About 10 years ago, I heard him give a lecture in the math 00:09:42.220 --> 00:09:44.030 department here. 00:09:44.030 --> 00:09:50.620 He was not ancient at that time, so this was probably 00:09:50.620 --> 00:09:53.960 then sometime around the '50s or '60s, back when Blackwell 00:09:53.960 --> 00:09:55.620 did a lot of work on stochastic 00:09:55.620 --> 00:09:57.260 processes was being done. 00:09:57.260 --> 00:10:00.680 The reason why this result is not trivial is because 00:10:00.680 --> 00:10:03.870 of this part here. 00:10:03.870 --> 00:10:09.520 If you have a renewal process where some renewals occur, say 00:10:09.520 --> 00:10:13.440 with interval 1, and some renewals occur with interval 00:10:13.440 --> 00:10:16.730 pi is a good example of it. 00:10:16.730 --> 00:10:20.200 Then as t gets larger and larger, the set of times at 00:10:20.200 --> 00:10:24.250 which renewals can occur becomes more and more dense. 00:10:24.250 --> 00:10:27.770 But along with it becoming more and more dense, the jumps 00:10:27.770 --> 00:10:30.870 you get at each one of those times gets smaller and 00:10:30.870 --> 00:10:36.050 smaller, and pretty soon n of t, this expected value of n of 00:10:36.050 --> 00:10:41.070 t, is looking like something which, if you don't have your 00:10:41.070 --> 00:10:44.590 glasses on, it looks like it's going up exactly the way it 00:10:44.590 --> 00:10:48.320 should be going up, but if you put your glasses on you see an 00:10:48.320 --> 00:10:50.950 enormous amount of fine structure there. 00:10:50.950 --> 00:10:53.200 And the fine structure never goes away. 00:10:53.200 --> 00:10:55.845 We'll talk more about that in the next slide. 00:11:01.840 --> 00:11:06.140 You can really look at this in a much simpler way, it's just 00:11:06.140 --> 00:11:08.810 that people like to look at the expected number of 00:11:08.810 --> 00:11:12.380 renewals at different times. 00:11:12.380 --> 00:11:17.540 Since renewals can only occur at time separated by lambda, 00:11:17.540 --> 00:11:20.580 and since you can't have two renewals at a time, the only 00:11:20.580 --> 00:11:24.430 question is, do you get a renewal at m lambda or don't 00:11:24.430 --> 00:11:27.450 you got a renewal at m lambda? 00:11:27.450 --> 00:11:31.600 And therefore this limit here, a limit of m of t plus lambda 00:11:31.600 --> 00:11:35.590 minus m of t, is really the question of whether you've 00:11:35.590 --> 00:11:38.450 gotten a renewal at t plus lambda. 00:11:38.450 --> 00:11:42.520 So, you can rewrite this condition as the limit of the 00:11:42.520 --> 00:11:46.180 probability of a renewal at m lambda as equal 00:11:46.180 --> 00:11:48.420 to lambda over x-bar. 00:11:48.420 --> 00:11:52.410 What happens with the scaling here? 00:11:52.410 --> 00:11:55.153 I mean, is this scaling right? 00:12:03.400 --> 00:12:05.140 Well, the expected time between 00:12:05.140 --> 00:12:08.260 renewals is 1 over x-bar. 00:12:08.260 --> 00:12:15.000 If I take this renewal process and I scale it, measuring it 00:12:15.000 --> 00:12:17.430 in milliseconds instead of seconds, 00:12:17.430 --> 00:12:20.140 what's going to happen? 00:12:20.140 --> 00:12:24.800 1 over x-bar is going to change by a factor of 1,000. 00:12:24.800 --> 00:12:27.760 You really want lambda to change by a factor of 1,000, 00:12:27.760 --> 00:12:32.140 because the probability of a jump at one of these possible 00:12:32.140 --> 00:12:35.910 places for a jump is still the same. 00:12:35.910 --> 00:12:39.310 So you need a lambda over the x-bar here. 00:12:39.310 --> 00:12:42.510 If you model an arithmetic renewal process as a Markov 00:12:42.510 --> 00:12:47.450 chain, starting in renewal state 0, this essentially says 00:12:47.450 --> 00:12:52.610 that p sub 0, 0 the probability of going from 0 to 00:12:52.610 --> 00:12:56.840 0 and in steps is some constant. 00:12:56.840 --> 00:12:59.810 I mean, calling pi 0, which is just what we've always called 00:12:59.810 --> 00:13:04.820 it, and pi 0 has to be, in this case, lambda over x-bar 00:13:04.820 --> 00:13:07.670 or 1 over x-bar. 00:13:07.670 --> 00:13:11.370 But what it's saying is that this reaches a constant. 00:13:11.370 --> 00:13:13.570 You know, that's the hard thing to prove. 00:13:13.570 --> 00:13:16.850 It's not hard to find this number. 00:13:16.850 --> 00:13:21.300 It's hard to prove that it does reach a limit, and that's 00:13:21.300 --> 00:13:24.730 the same thing that you're trying to prove here, so this 00:13:24.730 --> 00:13:27.760 and this are really saying the same thing. 00:13:27.760 --> 00:13:30.740 That the probability of renewal at m lambda is lambda 00:13:30.740 --> 00:13:38.900 over x-bar, and expected value of renewals at a given time is 00:13:38.900 --> 00:13:43.560 also 1 over x-bar. 00:13:43.560 --> 00:13:47.260 So, that's really the best you could hope for. 00:13:50.090 --> 00:13:55.120 If you look at the non-arithmetic case, I think 00:13:55.120 --> 00:13:58.120 one way of understanding it is to take the results that we 00:13:58.120 --> 00:14:03.180 had before, which we stated as part of Blackwell's theorem, 00:14:03.180 --> 00:14:05.380 which is this one. 00:14:05.380 --> 00:14:10.320 Divide both sides by delta and see what happens. 00:14:10.320 --> 00:14:15.190 If you divide m of t plus delta minus m of t by delta, 00:14:15.190 --> 00:14:17.520 it looks like you're trying to go to a limit and get a 00:14:17.520 --> 00:14:18.800 derivative. 00:14:18.800 --> 00:14:20.050 We know we can't get a derivative. 00:14:22.780 --> 00:14:25.950 So, what's going on then? 00:14:25.950 --> 00:14:34.070 It says for any delta that you want the limit as t goes to 00:14:34.070 --> 00:14:40.010 infinity of this ratio has to be 1 over x-bar. 00:14:40.010 --> 00:14:44.500 So, it says that if you take the limit as delta goes to 0 00:14:44.500 --> 00:14:48.960 of this quantity here, you still get 1 over x-bar. 00:14:48.960 --> 00:14:52.530 But this is a good example of the case where you really 00:14:52.530 --> 00:14:56.350 can't interchange these two limits. 00:14:56.350 --> 00:14:59.190 This is not a mathematical fine point. 00:14:59.190 --> 00:15:03.200 I mean, this is something that really cuts to the grain of 00:15:03.200 --> 00:15:06.630 what renewal processes are all about. 00:15:06.630 --> 00:15:12.240 So, you really ought to think through on your own why this 00:15:12.240 --> 00:15:17.320 makes sense when you take the limit as delta goes to 0 after 00:15:17.320 --> 00:15:21.350 you take this limit, whereas if you try to interchange the 00:15:21.350 --> 00:15:25.130 limits then you'd be trying to say you're taking the limit as 00:15:25.130 --> 00:15:28.070 t approaches infinity of a derivative here, and there 00:15:28.070 --> 00:15:29.870 isn't any derivative, so there isn't any 00:15:29.870 --> 00:15:31.500 limit, so nothing works. 00:15:34.150 --> 00:15:39.790 Let's look a little bit at age and duration at a particular 00:15:39.790 --> 00:15:41.670 value of t. 00:15:41.670 --> 00:15:45.200 Because we looked at age and duration only in terms of 00:15:45.200 --> 00:15:49.590 sample paths, and we went to the limit and we got very 00:15:49.590 --> 00:15:50.860 surprising results. 00:15:50.860 --> 00:15:58.370 We found out that the expected age was the expected value of 00:15:58.370 --> 00:16:01.830 the inter-renewal interval squared divided by the 00:16:01.830 --> 00:16:05.560 expected value of the inter-renewal interval all 00:16:05.560 --> 00:16:10.900 divided by 2, which didn't seem to make any sense. 00:16:10.900 --> 00:16:14.650 Because the inter-renewal time was just this random variable 00:16:14.650 --> 00:16:18.790 x, the expected inter-renewal time is x-bar. 00:16:18.790 --> 00:16:21.430 And yet you have this enormous duration. 00:16:21.430 --> 00:16:24.370 And we sort of motivated this in a number of ways. 00:16:24.370 --> 00:16:31.740 We drew some pictures and all of that, but it didn't really 00:16:31.740 --> 00:16:33.680 come together. 00:16:33.680 --> 00:16:37.030 When we look at this way I think it will come 00:16:37.030 --> 00:16:38.660 together for you. 00:16:38.660 --> 00:16:43.140 So let's assume an arithmetic renewal process will span 1. 00:16:45.760 --> 00:16:50.290 I partly want to look at an arithmetic process because 00:16:50.290 --> 00:16:53.130 it's much easier mathematically, and partly 00:16:53.130 --> 00:16:57.160 because that's the kind of thing we'll be interested in 00:16:57.160 --> 00:17:01.890 going to Markov chains with a countable number of states. 00:17:01.890 --> 00:17:09.810 You're looking at an integer value of t, z of t which is 00:17:09.810 --> 00:17:18.050 the age of time t is how long it's been since the last 00:17:18.050 --> 00:17:19.319 renewal occurred. 00:17:19.319 --> 00:17:26.000 So the age at time t is t minus s of 2, in this case, 00:17:26.000 --> 00:17:30.060 and in general it's t minus s sub n. 00:17:30.060 --> 00:17:33.170 So that's what the age is. 00:17:33.170 --> 00:17:41.570 The duration x tilde of t, is going to be the interval from 00:17:41.570 --> 00:17:47.570 this last renewal up to the next renewal after t. 00:17:47.570 --> 00:17:49.850 So, I've written this out here. 00:17:49.850 --> 00:17:57.690 x tilde of t is the renewal time for 00:17:57.690 --> 00:18:00.850 the n of t-th renewal. 00:18:00.850 --> 00:18:07.300 n of t is this value here. 00:18:07.300 --> 00:18:11.350 So what we're doing is we're looking at how long it takes 00:18:11.350 --> 00:18:13.080 to get from here to here. 00:18:13.080 --> 00:18:16.520 If I tell you that this interval starts here, this 00:18:16.520 --> 00:18:20.590 distribution here, we'll have the distribution of x of t. 00:18:20.590 --> 00:18:23.970 If all I tell you is we're looking around t for the 00:18:23.970 --> 00:18:27.820 previous interval in the next interval, then 00:18:27.820 --> 00:18:29.010 it's something different. 00:18:29.010 --> 00:18:30.980 How do we make sense out of this? 00:18:30.980 --> 00:18:33.720 Well, this is the picture that will make sense out it for 00:18:33.720 --> 00:18:38.290 you, so I hope this makes sense. 00:18:38.290 --> 00:18:42.530 If you look at an integer value of t, the z of t this 00:18:42.530 --> 00:18:46.090 age is going to be some integer greater 00:18:46.090 --> 00:18:47.340 than or equal to 0. 00:18:47.340 --> 00:18:50.470 Is it possible for the age to be 0? 00:18:50.470 --> 00:18:54.480 Yes, of course it is, because t is some integer value. 00:18:54.480 --> 00:18:57.940 An arrival could have just come in at time t, and then 00:18:57.940 --> 00:18:59.190 the age is 0. 00:19:03.940 --> 00:19:08.780 The time from 1 renewal until the next has to be at least 1, 00:19:08.780 --> 00:19:12.310 because you only get renewals in at integer times, and you 00:19:12.310 --> 00:19:14.890 can have two renewals at the same time. 00:19:14.890 --> 00:19:19.200 So x tilde of t has to be bigger than 1. 00:19:19.200 --> 00:19:22.690 How do we express this in a different way? 00:19:22.690 --> 00:19:26.590 Well, let's let q sub j be the probability this is an 00:19:26.590 --> 00:19:29.130 arrival at time j. 00:19:29.130 --> 00:19:33.430 If you want to write that down with an equation, it's the sum 00:19:33.430 --> 00:19:37.910 over all n greater than or equal to 1 of the probability 00:19:37.910 --> 00:19:41.710 that the n-th arrival occurs at time j. 00:19:41.710 --> 00:19:52.160 In other words, q sub j is the probability that the first 00:19:52.160 --> 00:19:55.600 arrival occurs at time j, plus the probability that the 00:19:55.600 --> 00:19:58.570 second arrival occurs at time j, plus the probability a 00:19:58.570 --> 00:20:00.420 third arrival occurs at time j. 00:20:00.420 --> 00:20:03.060 Those are all disjoint events, you can't have two 00:20:03.060 --> 00:20:05.660 of them be the same. 00:20:05.660 --> 00:20:10.940 So this q sub j is just the probability that there is an 00:20:10.940 --> 00:20:12.920 arrival at time j. 00:20:12.920 --> 00:20:14.170 What's that a function of? 00:20:20.360 --> 00:20:23.180 It's a function of the arrivals that 00:20:23.180 --> 00:20:25.675 occur before time j. 00:20:28.500 --> 00:20:30.110 And it's independent. 00:20:30.110 --> 00:20:32.670 That's how long the next arrival takes. 00:20:32.670 --> 00:20:38.120 If I tell you, yes, there was an arrival here conditional on 00:20:38.120 --> 00:20:40.600 the fact that there was an arrival here, 00:20:40.600 --> 00:20:43.090 which arrival is it? 00:20:43.090 --> 00:20:45.430 I'm not talking about t or anything else, I'm just 00:20:45.430 --> 00:20:48.170 saying, suppose we know there's an arrival here. 00:20:48.170 --> 00:20:50.830 What you would look at would be the previous arrivals, the 00:20:50.830 --> 00:20:54.040 previous inter-renewal intervals, and in terms of 00:20:54.040 --> 00:20:58.220 that you would sort out what that probability is. 00:20:58.220 --> 00:21:04.370 Then if there's an arrival here, what's the probability 00:21:04.370 --> 00:21:09.810 that x tilde of t is equal to this particular value here? 00:21:09.810 --> 00:21:13.150 Well, now this is where the argument gets tricky. 00:21:13.150 --> 00:21:17.690 q sub j is a probability of arrival of time j. 00:21:17.690 --> 00:21:23.960 What I maintain is the joint probability that z of t is 00:21:23.960 --> 00:21:28.020 equal to i and this duration here is equal to k. 00:21:28.020 --> 00:21:33.910 In other words, this is equal to i here, this is equal to k. 00:21:33.910 --> 00:21:38.950 The probability of that this q of t minus i, in other words, 00:21:38.950 --> 00:21:44.180 the probability that this is an arrival at time t minus i, 00:21:44.180 --> 00:21:50.290 and the probability that this inter-renewal interval here 00:21:50.290 --> 00:21:55.220 has duration k where the restriction is that k has to 00:21:55.220 --> 00:21:57.860 be bigger than i. 00:21:57.860 --> 00:22:00.960 And that's what's fishy about this. 00:22:00.960 --> 00:22:09.480 But it's perfectly correct, just so long as I stick to 00:22:09.480 --> 00:22:15.980 values of i and k, where k is bigger than i. 00:22:15.980 --> 00:22:20.050 i is the age here and x hat is t, x 00:22:20.050 --> 00:22:23.570 tilde of t is the duration. 00:22:23.570 --> 00:22:28.030 And I can rewrite that joint probability as a probability 00:22:28.030 --> 00:22:31.710 that we get an arrival here, and that the probability the 00:22:31.710 --> 00:22:39.060 next arrival takes this time x tilde of t. 00:22:39.060 --> 00:22:44.080 Now, what we've done with this is to replace the idea of 00:22:44.080 --> 00:22:47.520 looking at duration with the idea of looking at an 00:22:47.520 --> 00:22:50.190 inter-renewal interval. 00:22:50.190 --> 00:22:55.400 In other words, this probability here, this is not 00:22:55.400 --> 00:23:01.050 the pmf of x tilde, this is the pmf of x itself. 00:23:01.050 --> 00:23:04.700 This is the thing we hope we understand at this point. 00:23:04.700 --> 00:23:06.610 This is what you learned about on the first day of 00:23:06.610 --> 00:23:10.650 probability theory, when you started taking 6041 or 00:23:10.650 --> 00:23:11.890 whatever you took. 00:23:11.890 --> 00:23:14.730 You started learning about random variables, and if they 00:23:14.730 --> 00:23:19.500 were discrete random variables they had pmf's, and if you had 00:23:19.500 --> 00:23:23.150 a sequence of IID random variables, this 00:23:23.150 --> 00:23:25.710 was the pmf you had. 00:23:25.710 --> 00:23:30.050 So this joint probability here is really these very simple 00:23:30.050 --> 00:23:33.460 things that you already understand. 00:23:33.460 --> 00:23:38.190 But it's only equal to this for i less than or equal to t, 00:23:38.190 --> 00:23:43.290 0 less than i, less or equal to t, and k greater than i. 00:23:43.290 --> 00:23:48.030 This restriction here that how far back you go to the last 00:23:48.030 --> 00:23:52.470 arrival can't be any more than t, it's really sort of a 00:23:52.470 --> 00:23:53.590 technical restriction. 00:23:53.590 --> 00:23:55.760 I mean, you need it to be accurate, but 00:23:55.760 --> 00:23:57.330 it's not very important. 00:23:57.330 --> 00:24:01.110 The important thing is that k has to be bigger than i 00:24:01.110 --> 00:24:07.060 because otherwise you get this arrival here and it's not 00:24:07.060 --> 00:24:10.810 beyond t and therefore it's not the interval that 00:24:10.810 --> 00:24:13.990 surrounds t, it's some other interval. 00:24:13.990 --> 00:24:19.510 So, that tells you what you need to know about this. 00:24:22.350 --> 00:24:28.660 So, the joint probability is z of t, and x tilde of t is 00:24:28.660 --> 00:24:35.110 equal to i and k, is this conventional probability here. 00:24:35.110 --> 00:24:37.830 So what we know, there's a q sub i as the probability of an 00:24:37.830 --> 00:24:42.350 arrival of j that's equal to the expected value of an 00:24:42.350 --> 00:24:46.240 arrival at j and is equal to the expected number of 00:24:46.240 --> 00:24:49.380 arrivals that have occurred. 00:24:49.380 --> 00:24:52.500 Excuse me, that i there should be a j and this 00:24:52.500 --> 00:24:55.400 i should be a j. 00:24:55.400 --> 00:24:58.680 Oh wait, this is q sub i. 00:24:58.680 --> 00:25:02.830 This should be an i and this should be an i. 00:25:02.830 --> 00:25:05.500 If I'm looking at it on my computer and I don't have my 00:25:05.500 --> 00:25:09.160 glasses cleaned I can't tell the difference between them. 00:25:09.160 --> 00:25:12.790 q sub i is the probability of an arrival at i. 00:25:12.790 --> 00:25:17.620 The expectation of the number of arrivals at i is equal to 00:25:17.620 --> 00:25:20.390 the expected number of arrivals at i minus expected 00:25:20.390 --> 00:25:22.860 number of arrivals at i minus 1. 00:25:22.860 --> 00:25:31.260 So Blackwell says, what i is 00:25:31.260 --> 00:25:33.110 asymptotically when i gets large. 00:25:33.110 --> 00:25:36.095 He says that the limit as t goes to infinity. 00:25:41.880 --> 00:25:45.120 I think this thing's a little weak. 00:25:45.120 --> 00:25:46.630 In fact, it's terribly weak. 00:25:51.750 --> 00:25:56.440 So the thing that Blackwell says is that q sub i goes to a 00:25:56.440 --> 00:25:58.720 constant when i gets very large. 00:26:01.330 --> 00:26:03.580 Namely he says that the probability that you get an 00:26:03.580 --> 00:26:07.520 arrival at time i when i is very large 00:26:07.520 --> 00:26:09.040 is just some constant. 00:26:09.040 --> 00:26:11.180 It doesn't depend on i anymore. 00:26:11.180 --> 00:26:13.945 It wobbles around for a while, and then it becomes constant. 00:26:16.850 --> 00:26:24.810 This limit here is then 1 over x-bar, which is what q sub i 00:26:24.810 --> 00:26:29.590 is, times the probability of k. 00:26:29.590 --> 00:26:34.630 Now, that's very weird because what it says is that this 00:26:34.630 --> 00:26:39.730 probability does not depend on the age at all. 00:26:39.730 --> 00:26:42.840 It's just a function of the duration. 00:26:42.840 --> 00:26:46.250 As a function of the duration, it's just p sub x of 00:26:46.250 --> 00:26:47.500 k divided by x-bar. 00:26:50.090 --> 00:26:55.250 If I go back to what we were looking at before, what this 00:26:55.250 --> 00:27:00.810 says is if I take this interval here and shift it 00:27:00.810 --> 00:27:06.030 around, those pairs of points will have exactly the same 00:27:06.030 --> 00:27:07.650 probability asymptotically. 00:27:12.830 --> 00:27:17.610 Now, if you remember what we talked about this very vague 00:27:17.610 --> 00:27:22.270 idea called random incidents, if you look at a sample path 00:27:22.270 --> 00:27:26.820 and then you throw a dart at the sample path and you say, 00:27:26.820 --> 00:27:31.990 what's the duration beyond t, what's the duration before t, 00:27:31.990 --> 00:27:34.250 this is doing the same thing. 00:27:34.250 --> 00:27:37.940 But it's doing it in a very exact and precise way. 00:27:37.940 --> 00:27:41.330 So, this is really the mathematical way to look at 00:27:41.330 --> 00:27:44.250 this random instance idea. 00:27:44.250 --> 00:27:58.210 And what it's saying, if we go back to where we were, that's 00:27:58.210 --> 00:28:03.860 telling us the joint probability of age and 00:28:03.860 --> 00:28:10.750 duration is just a function of the inter-renewal interval. 00:28:10.750 --> 00:28:14.590 It doesn't depend on what i is at all. 00:28:14.590 --> 00:28:17.390 It doesn't depend on how long it's been since the last 00:28:17.390 --> 00:28:20.185 renewal, it only depends on the size of the 00:28:20.185 --> 00:28:22.310 inter-renewal interval. 00:28:22.310 --> 00:28:29.010 So then we say, OK why don't we try to find the pmf of age? 00:28:29.010 --> 00:28:29.930 How do we do that? 00:28:29.930 --> 00:28:33.640 Well, we have this joint distribution of z and x with 00:28:33.640 --> 00:28:37.210 these constraints on it. k has to be bigger than i to make 00:28:37.210 --> 00:28:41.630 sure the inter-renewal interval covers t. 00:28:41.630 --> 00:28:45.520 And you look at that formula there, and you say, OK i 00:28:45.520 --> 00:28:54.430 travels from 0 up to k minus 1, and k is going to travel 00:28:54.430 --> 00:28:58.500 from i plus 1 all the way up to infinity. 00:28:58.500 --> 00:29:03.750 So, if I try to look at what the probability of the age is, 00:29:03.750 --> 00:29:09.062 it's going to be the sum of k equals i plus 1 up to infinity 00:29:09.062 --> 00:29:11.280 of the joint probability. 00:29:11.280 --> 00:29:18.170 Because if I fix what the age is I'm just looking at all 00:29:18.170 --> 00:29:22.980 possible durations from i plus 1 all the way up to infinity. 00:29:22.980 --> 00:29:27.770 So, the marginal for z of t is this complimentary 00:29:27.770 --> 00:29:30.160 distribution function evaluated at 00:29:30.160 --> 00:29:33.510 i divided by x-bar. 00:29:33.510 --> 00:29:36.340 That's a little hard to visualize. 00:29:36.340 --> 00:29:39.690 But if we look at the duration, it's a whole lot 00:29:39.690 --> 00:29:42.540 easier to visualize and see what's going on. 00:29:42.540 --> 00:29:46.510 If you want to take the marginal of the duration the 00:29:46.510 --> 00:29:52.650 pmf the duration is equal to k, what is it? 00:29:52.650 --> 00:29:56.410 We have to average out over z of t, which is the age. 00:29:56.410 --> 00:30:02.690 The age can be anything from 0 up to k minus 1 if we're 00:30:02.690 --> 00:30:05.140 looking at a particular value of k. 00:30:05.140 --> 00:30:11.940 What that is again is this diagram here, and x-hat of t 00:30:11.940 --> 00:30:17.760 will have a particular value of k here, here, here, here, 00:30:17.760 --> 00:30:21.130 all the way up to here. 00:30:21.130 --> 00:30:24.480 So what we're doing is adding up all those values, knowing 00:30:24.480 --> 00:30:27.870 exactly what the idea of random instance was doing, but 00:30:27.870 --> 00:30:30.260 doing it in a nice clean way. 00:30:35.850 --> 00:30:39.630 Back to the ranch, it says that the limit is t goes to 00:30:39.630 --> 00:30:47.690 infinity of x tilde is going to be k times pmf of 00:30:47.690 --> 00:30:50.840 x divided by x-bar. 00:30:50.840 --> 00:30:54.020 You need to divide by x-bar so that this is a probability 00:30:54.020 --> 00:30:57.750 mass function, and we actually had the x-bar there all along. 00:30:57.750 --> 00:31:02.230 But if we sum this up over k, what do we get? 00:31:02.230 --> 00:31:06.310 We get k times the pmf of x, which is the 00:31:06.310 --> 00:31:08.600 expected value of x. 00:31:08.600 --> 00:31:12.010 So, that's all very nice. 00:31:12.010 --> 00:31:16.690 But what happens if we try to find the expected value of 00:31:16.690 --> 00:31:18.540 this duration function here? 00:31:22.100 --> 00:31:23.750 That's really no harder, the expected 00:31:23.750 --> 00:31:26.450 value of the duration. 00:31:26.450 --> 00:31:29.370 Here's one of the few places where we use pmf's to do 00:31:29.370 --> 00:31:31.050 everything. 00:31:31.050 --> 00:31:36.890 It's a sum from k equals 1 to infinity of k, which is the k 00:31:36.890 --> 00:31:45.920 we had before, k times px of k over x-bar. 00:31:45.920 --> 00:31:50.240 So it's k times kpx of k divided by x-bar. 00:31:50.240 --> 00:31:56.570 That's k squared times the probability mass function of x 00:31:56.570 --> 00:32:00.460 divided by x-bar, which is the expected value of x squared 00:32:00.460 --> 00:32:02.970 divided by the expected value of x. 00:32:08.610 --> 00:32:12.770 Now, that sounds a little weird also, but I think it's a 00:32:12.770 --> 00:32:20.300 whole lot less weird than the argument using sample paths. 00:32:20.300 --> 00:32:24.110 I mean, you can look at this and you can track down every 00:32:24.110 --> 00:32:27.560 little bit of it, and you can be very sure 00:32:27.560 --> 00:32:29.150 of what it's saying. 00:32:29.150 --> 00:32:33.200 You can find the expected age in sort of the same way, and 00:32:33.200 --> 00:32:37.300 it's done in the text, and it comes out to be expected value 00:32:37.300 --> 00:32:42.730 of x squared divided by 2 times the expected value of x, 00:32:42.730 --> 00:32:49.170 which is exactly what the argument was for if you're 00:32:49.170 --> 00:32:52.510 looking at the sample path point. 00:32:52.510 --> 00:32:55.787 But you lose a 1/2. 00:32:55.787 --> 00:32:57.510 AUDIENCE: When you look at this [INAUDIBLE] 00:32:57.510 --> 00:32:59.104 it was for non-arithmetic distributions. 00:32:59.104 --> 00:33:01.040 And here it's for arithmetic. 00:33:01.040 --> 00:33:04.912 So it's always true that the [INAUDIBLE], even for 00:33:04.912 --> 00:33:06.162 arithmetic? 00:33:08.230 --> 00:33:08.510 PROFESSOR: Yes. 00:33:08.510 --> 00:33:13.890 The previous argument we went through using sample paths was 00:33:13.890 --> 00:33:17.330 for arithmetic distribution, as non-arithmetic was for any 00:33:17.330 --> 00:33:19.810 distribution at all. 00:33:19.810 --> 00:33:22.920 This argument here, you have to distinguish between 00:33:22.920 --> 00:33:26.880 arithmetic and non-arithmetic, and you have to distinguish 00:33:26.880 --> 00:33:30.930 because they act in very different ways. 00:33:30.930 --> 00:33:34.700 I mean, if you read the text and you read the section on 00:33:34.700 --> 00:33:38.810 non-arithmetic random variables, you wind up with a 00:33:38.810 --> 00:33:43.340 lot of very tedious stuff that's going on, a lot of 00:33:43.340 --> 00:33:46.230 really having to understand what [INAUDIBLE] integration 00:33:46.230 --> 00:33:49.000 is all about. 00:33:49.000 --> 00:33:51.750 I mean, if you put in densities it's all fine, but 00:33:51.750 --> 00:33:54.740 if you try to do it for these weird distributions which are 00:33:54.740 --> 00:33:58.810 discrete but non-arithmetic, you really have a lot of 00:33:58.810 --> 00:34:02.205 sweating to do to make sure that any of this makes sense. 00:34:05.720 --> 00:34:07.300 Which is why I'm doing it this way. 00:34:07.300 --> 00:34:09.790 But, when we do it this way we get this extra 00:34:09.790 --> 00:34:11.880 factor of 1/2 here. 00:34:11.880 --> 00:34:14.000 And where does the factor of 1/2 come from? 00:34:16.880 --> 00:34:20.719 Well, the factor of 1/2 comes from the fact that all of this 00:34:20.719 --> 00:34:27.280 argument was looking at a t, which has an integer value. 00:34:27.280 --> 00:34:31.350 Because we've made t the integer, the age could be 0. 00:34:35.409 --> 00:34:39.380 If we make t non-integer, then the age is going to be 00:34:39.380 --> 00:34:42.840 something between 0 and 1. 00:34:42.840 --> 00:34:50.409 And in fact, if we look at what's happening, what we're 00:34:50.409 --> 00:34:58.760 going to find is that the age of your integer values into 00:34:58.760 --> 00:35:04.230 the 6th, into the 6th plus one. 00:35:04.230 --> 00:35:06.590 And you know from the homework that this might have to be 10 00:35:06.590 --> 00:35:09.620 to the 20th, but it doesn't make any difference. 00:35:13.360 --> 00:35:20.380 And now the age here can be 0, so the average age is going to 00:35:20.380 --> 00:35:24.100 be some value here. 00:35:24.100 --> 00:35:27.130 And as you look at larger and larger t's, as you go from 00:35:27.130 --> 00:35:31.325 this integer to this integer, the age is going to increase. 00:35:37.230 --> 00:35:40.710 And then at this point there might be an 00:35:40.710 --> 00:35:42.040 arrival at this point. 00:35:42.040 --> 00:35:45.760 This is the probability of an arrival at this point. 00:35:45.760 --> 00:35:47.015 Then it goes up again. 00:35:49.720 --> 00:35:53.730 It goes up exactly the same way. 00:35:53.730 --> 00:35:56.180 It goes down. 00:35:56.180 --> 00:35:59.870 And the value it has at the integer is its lowest value, 00:35:59.870 --> 00:36:02.730 because we're assuming that when you look at the age 00:36:02.730 --> 00:36:06.080 you're looking at an age which could be 0 if you look at this 00:36:06.080 --> 00:36:09.390 particular integer value. 00:36:09.390 --> 00:36:12.330 So, where's the problem? 00:36:12.330 --> 00:36:16.220 We take the sample average over time, and what we're 00:36:16.220 --> 00:36:19.170 doing is we're averaging over time here. 00:36:19.170 --> 00:36:23.890 So, you wind up with this point, then you crawl up, then 00:36:23.890 --> 00:36:27.790 you go down again, you crawl up, you go down again. 00:36:27.790 --> 00:36:31.340 And this average, which is the thing we found before for the 00:36:31.340 --> 00:36:36.720 sample path average, is exactly 1/2 larger than what 00:36:36.720 --> 00:36:39.200 we found here. 00:36:39.200 --> 00:36:43.200 So, miraculously these two numbers jive, which I find 00:36:43.200 --> 00:36:45.210 amazing after all this work. 00:36:45.210 --> 00:36:46.460 But they do. 00:36:48.480 --> 00:36:53.480 And the fact that you have the 1/2 there is sort of a check 00:36:53.480 --> 00:36:55.830 on the fact that people have done the work right, because 00:36:55.830 --> 00:36:59.340 nobody would ever imagine that was there until they actually 00:36:59.340 --> 00:37:01.450 went through it and found it. 00:37:05.750 --> 00:37:09.870 That, I think, explains why you get these peculiar results 00:37:09.870 --> 00:37:11.815 for duration and for age. 00:37:11.815 --> 00:37:13.065 At least, I hope it does. 00:37:15.950 --> 00:37:18.755 Let's go on to countable-state Markov chains. 00:37:22.480 --> 00:37:27.050 And the big change that occurs when you go to countable-state 00:37:27.050 --> 00:37:32.130 chains is what you mean by a recurring class. 00:37:32.130 --> 00:37:36.580 Before with finite state Markov chains, we just blandly 00:37:36.580 --> 00:37:41.210 defined a recurrent state of the state which had the 00:37:41.210 --> 00:37:44.760 property that wherever you could go from that state, 00:37:44.760 --> 00:37:47.370 there was always some way to get back. 00:37:47.370 --> 00:37:50.530 And since you had a finite number of states, if there was 00:37:50.530 --> 00:37:53.220 some place you could go, you're always going to get 00:37:53.220 --> 00:37:56.260 back to eventually, because there was always some path 00:37:56.260 --> 00:37:59.700 that had some probability, and you keep repeating the 00:37:59.700 --> 00:38:02.500 possibility of doing that. 00:38:02.500 --> 00:38:05.190 So, you didn't have to worry about decision. 00:38:05.190 --> 00:38:09.860 Here you do have to worry about it. 00:38:09.860 --> 00:38:16.850 It's particularly funny if you look at a Markov chain model 00:38:16.850 --> 00:38:20.140 of a Bernoulli process. 00:38:20.140 --> 00:38:23.070 So, a Markov chain model of a Bernoulli process, this is a 00:38:23.070 --> 00:38:28.730 countable-state process, you start out at state 0. 00:38:28.730 --> 00:38:32.490 You flip your coin, your loaded coin, which comes up 1, 00:38:32.490 --> 00:38:37.750 so the probability p and tails with probability q, and if it 00:38:37.750 --> 00:38:41.430 comes up heads you go to state 1. 00:38:41.430 --> 00:38:45.180 If it comes up tails, you go to state minus 1. 00:38:45.180 --> 00:38:56.270 So this state here is really the sum of the x of i's. 00:38:56.270 --> 00:39:03.910 In other words, it's the number of successes minus the 00:39:03.910 --> 00:39:06.530 number of failures. 00:39:06.530 --> 00:39:09.360 So, we're looking at that sum and we're looking at what 00:39:09.360 --> 00:39:12.540 happens as time gets larger and larger. 00:39:12.540 --> 00:39:15.990 So, you go wandering around here, moving up with 00:39:15.990 --> 00:39:20.110 probability p, moving down the probability q, and you sort of 00:39:20.110 --> 00:39:23.060 get the idea that this is going to diffuse 00:39:23.060 --> 00:39:26.090 after a long time. 00:39:26.090 --> 00:39:29.610 How do you do that mathematically? 00:39:29.610 --> 00:39:32.600 And the text writes this out carefully. 00:39:32.600 --> 00:39:37.010 I don't want to do that here, because I think it's important 00:39:37.010 --> 00:39:41.690 for you in doing exercises and things like that to start to 00:39:41.690 --> 00:39:45.580 see these arguments automatically. 00:39:45.580 --> 00:39:52.940 So, the thing that's going to happen as n gets large is that 00:39:52.940 --> 00:40:00.810 the variance of this state is going to be n times the 00:40:00.810 --> 00:40:06.050 variance of a single up or down random variable. 00:40:06.050 --> 00:40:11.090 You calculate that as 1 minus p minus q squared, and if p 00:40:11.090 --> 00:40:16.060 and q are both strictly between 0 and 1, that variance 00:40:16.060 --> 00:40:19.170 is always positive. 00:40:19.170 --> 00:40:21.460 It can't be negative, you can't have negative variance. 00:40:21.460 --> 00:40:25.970 But the important thing is it keeps increasing within. 00:40:25.970 --> 00:40:30.690 So, it says that if you try to draw a diagram, if you try to 00:40:30.690 --> 00:40:38.280 draw pmf of the probability that s sub n is equal to 0, 1, 00:40:38.280 --> 00:40:42.400 2, 3, 4, and so forth, what it's going to do is n gets 00:40:42.400 --> 00:40:43.460 very large. 00:40:43.460 --> 00:40:48.490 Because this pmf is going to keep scaling outward, and it's 00:40:48.490 --> 00:40:51.770 going to scale outwards with the square root of n. 00:40:54.520 --> 00:40:56.660 And we already know that it starts to look Gaussian. 00:41:00.190 --> 00:41:06.550 And in particular what I mean by looking Gaussian is that 00:41:06.550 --> 00:41:09.360 it's going to be a quantized version of the Gaussian, 00:41:09.360 --> 00:41:15.720 because each time you increase by 1 there's a probability 00:41:15.720 --> 00:41:18.770 that s sub n equals that increased value. 00:41:18.770 --> 00:41:23.310 So you're spreading out the individual values at each 00:41:23.310 --> 00:41:28.590 integer, have to spread, have to come down. 00:41:28.590 --> 00:41:30.820 They can't do anything else. 00:41:30.820 --> 00:41:33.360 If you're spreading a distribution out and it's an 00:41:33.360 --> 00:41:38.460 integer distribution, you can't keep these values the 00:41:38.460 --> 00:41:42.080 same as they were before or you would have a total 00:41:42.080 --> 00:41:45.440 probability, which would be growing with the 00:41:45.440 --> 00:41:47.330 square root of n. 00:41:47.330 --> 00:41:50.890 And the probability that s sub n is something is equal to 1. 00:41:50.890 --> 00:41:55.320 So as you spread out you have this Gaussian distribution 00:41:55.320 --> 00:41:58.140 where the variance is growing within and where the 00:41:58.140 --> 00:42:02.460 probability of each individual value is going down as 1 over 00:42:02.460 --> 00:42:05.080 the square root of the n. 00:42:05.080 --> 00:42:08.520 Now, this is the sort of argument which I hope can 00:42:08.520 --> 00:42:11.860 become automatic for you, because this is the kind of 00:42:11.860 --> 00:42:17.100 thing you see in problems and it's a big mess to analyze it. 00:42:17.100 --> 00:42:19.270 You have to go through a lot of work, you have to be very 00:42:19.270 --> 00:42:22.660 careful about how you're scaling things. 00:42:22.660 --> 00:42:25.860 And if you just look at this saying, what's going to happen 00:42:25.860 --> 00:42:30.360 here as n gets large is that this going to be like a 00:42:30.360 --> 00:42:34.260 Gaussian distribution, like a quantized Gaussian 00:42:34.260 --> 00:42:37.260 distribution, and if it goes out this way, 00:42:37.260 --> 00:42:40.090 it's got to go down. 00:42:40.090 --> 00:42:43.170 You can't go out without going down. 00:42:43.170 --> 00:42:47.510 So, it says that no matter what p is and no matter what q 00:42:47.510 --> 00:42:53.200 is, so long as neither of them are 0, this thing is going to 00:42:53.200 --> 00:42:56.460 spread out, and the probability that you're in any 00:42:56.460 --> 00:43:02.030 particular state after a long time is going to 0. 00:43:02.030 --> 00:43:05.410 Now, that's not like the behavior of finite state 00:43:05.410 --> 00:43:06.115 Markov chains. 00:43:06.115 --> 00:43:08.740 Because in the finite state Markov chains, you have 00:43:08.740 --> 00:43:12.330 recurring classes, you have steady state probabilities for 00:43:12.330 --> 00:43:14.070 those recurring classes. 00:43:14.070 --> 00:43:18.720 Here you don't have any steady state probability distribution 00:43:18.720 --> 00:43:20.950 for this situation. 00:43:20.950 --> 00:43:26.630 Because in a steady state every state has probability 0, 00:43:26.630 --> 00:43:30.960 and those probabilities don't add to 1, so you just can't 00:43:30.960 --> 00:43:34.420 deal with this in any sensible way. 00:43:34.420 --> 00:43:37.700 So, here we have a countable-state Markov chain, 00:43:37.700 --> 00:43:42.630 which doesn't behave at all like the finite state Markov 00:43:42.630 --> 00:43:45.220 chains we've looked at before. 00:43:45.220 --> 00:43:47.790 What's the period of this chain, by the way? 00:43:47.790 --> 00:43:52.020 I said, here it's equal to 2, why is it equal to 2? 00:43:52.020 --> 00:43:57.020 Well, you start out at an even number, s sub n is equal to 0, 00:43:57.020 --> 00:44:01.940 after one transition s sub n is odd, after two transitions 00:44:01.940 --> 00:44:05.603 it's even again, so it keeps oscillating between even and 00:44:05.603 --> 00:44:08.750 odd, which means there's a period of two. 00:44:08.750 --> 00:44:13.170 So, we have a situation where all states communicate. 00:44:13.170 --> 00:44:16.410 The definition of communicate is the same as it was before. 00:44:16.410 --> 00:44:18.990 There's a path they get from here to there, and there's a 00:44:18.990 --> 00:44:22.520 path to get back again. 00:44:22.520 --> 00:44:26.150 And classes are the same as they were before, a set of 00:44:26.150 --> 00:44:30.960 states which all communicate with each other, are all in 00:44:30.960 --> 00:44:36.250 the same class, and if i and j communicate, then if j and k 00:44:36.250 --> 00:44:39.290 communicate, then there's a path to get from i to j and a 00:44:39.290 --> 00:44:42.600 path to get from j to k, to the path to get from i to k, 00:44:42.600 --> 00:44:44.780 to the path to get back again. 00:44:44.780 --> 00:44:55.713 So if i and j communicate and j and k communicate, then i 00:44:55.713 --> 00:44:58.310 and k communicate also, which is why you get 00:44:58.310 --> 00:45:01.400 classes out of this. 00:45:01.400 --> 00:45:03.140 So, we have classes. 00:45:03.140 --> 00:45:07.410 What we don't have is anything relating to steady state 00:45:07.410 --> 00:45:10.330 probabilities, necessarily. 00:45:10.330 --> 00:45:13.480 So what we do? 00:45:13.480 --> 00:45:19.260 Well, another example, that's called a birth-death chain, we 00:45:19.260 --> 00:45:23.270 will see this much more often than the ugly thing I had in 00:45:23.270 --> 00:45:25.020 the last slide. 00:45:25.020 --> 00:45:27.660 The ugly thing I had in the last slide is really much 00:45:27.660 --> 00:45:31.870 simpler, but it's harder to analyze. 00:45:31.870 --> 00:45:33.120 It's gruesome. 00:45:35.380 --> 00:45:37.860 Well, as a matter of fact, we probably ought to talk about 00:45:37.860 --> 00:45:40.950 this a little further. 00:45:40.950 --> 00:45:43.030 Because this does something. 00:45:43.030 --> 00:45:50.420 If you pick p equal to 1/2, this thing is going to expand 00:45:50.420 --> 00:45:55.140 the center of it for any n is going to stay at 0, so the 00:45:55.140 --> 00:45:57.020 probability of each state is going to 00:45:57.020 --> 00:46:00.480 get smaller and smaller. 00:46:00.480 --> 00:46:05.610 What's the probability when you start at 0 that you ever 00:46:05.610 --> 00:46:07.030 get back to 0 again? 00:46:11.680 --> 00:46:13.780 Now, that's not an easy question, it's a question that 00:46:13.780 --> 00:46:16.830 takes a good deal of analysis and a good deal of head 00:46:16.830 --> 00:46:18.200 scratching. 00:46:18.200 --> 00:46:21.720 But the answer is 1. 00:46:21.720 --> 00:46:24.740 There's a 0 steady state probability of being in state 00:46:24.740 --> 00:46:29.050 0, but you wander away and you eventually get back. 00:46:29.050 --> 00:46:32.450 There's always a path to come back, and no matter how far 00:46:32.450 --> 00:46:36.020 away you get it's just as easy to get back as it was to get 00:46:36.020 --> 00:46:39.350 out there, so it's certainly plausible that you ought to 00:46:39.350 --> 00:46:42.870 get back with probability 1. 00:46:42.870 --> 00:46:46.660 And we will probably prove that at some point. 00:46:46.660 --> 00:46:52.070 I'm not going to prove it today because it would just be 00:46:52.070 --> 00:46:53.570 too much to prove for today. 00:46:56.490 --> 00:47:00.060 And, also, you will need to have done 00:47:00.060 --> 00:47:02.750 a few of the exercises. 00:47:02.750 --> 00:47:07.660 The homework set for this week is kind of easy, because 00:47:07.660 --> 00:47:11.010 you're probably all exhausted after studying for the quiz. 00:47:11.010 --> 00:47:15.200 I hope the quiz was easy, but even if it was easy you still 00:47:15.200 --> 00:47:16.720 had to study for it, and that's the thing 00:47:16.720 --> 00:47:19.380 that takes the time. 00:47:19.380 --> 00:47:26.310 So this week we won't do too much, but you will get some 00:47:26.310 --> 00:47:28.640 experience working with a set of ideas. 00:47:32.140 --> 00:47:35.470 We want to go to this next thing, which is called a 00:47:35.470 --> 00:47:38.210 birth-death chain. 00:47:38.210 --> 00:47:41.460 There are only non-negative states here, 0, 1, 2. 00:47:41.460 --> 00:47:45.210 It looks exactly the same as the previous chain, except you 00:47:45.210 --> 00:47:46.130 can't go negative. 00:47:46.130 --> 00:47:50.560 Whenever you go down to 0 you bounce around and you then can 00:47:50.560 --> 00:47:52.890 go up again and come back again as 00:47:52.890 --> 00:47:54.140 sort of like an accordion. 00:47:57.570 --> 00:48:02.270 If p is less than 1/2, you can sort of imagine what's going 00:48:02.270 --> 00:48:02.880 to happen here. 00:48:02.880 --> 00:48:07.350 Every time you go up there's a force pulling you back that's 00:48:07.350 --> 00:48:10.520 bigger than the force going up, so you can imagine that 00:48:10.520 --> 00:48:13.960 you're going to stay clustered pretty close to 0. 00:48:13.960 --> 00:48:17.460 If p is greater than 1/2, you're going to go off into 00:48:17.460 --> 00:48:21.410 the wild blue yonder, and you're never going to come 00:48:21.410 --> 00:48:28.890 back eventually, but if p is equal to 1/2 then it's kind of 00:48:28.890 --> 00:48:31.410 hard to see what's going to happen again. 00:48:31.410 --> 00:48:34.360 It's the same as the situation before. 00:48:34.360 --> 00:48:40.600 You will come back with probability 1 if p is 1/2, but 00:48:40.600 --> 00:48:43.980 you won't have any steady state probability, which is a 00:48:43.980 --> 00:48:48.220 strange case, which we'll talk about as we go on. 00:48:48.220 --> 00:48:50.980 You looked at the truncated case of this in the homework, 00:48:50.980 --> 00:48:55.050 namely you looked at a case of what happened if you just 00:48:55.050 --> 00:49:00.090 truncated this chain of some value n and you found the 00:49:00.090 --> 00:49:02.310 steady state probabilities. 00:49:02.310 --> 00:49:07.100 And what you found was that if p is equal to 1/2, the steady 00:49:07.100 --> 00:49:10.230 state probabilities are uniform. 00:49:10.230 --> 00:49:14.760 If p is greater than 1/2, the states over in this end have 00:49:14.760 --> 00:49:18.300 high probabilities, they travel down geometrically to 00:49:18.300 --> 00:49:20.760 the states at this end. 00:49:20.760 --> 00:49:26.220 If p is less than 1/2, the states are highly probable and 00:49:26.220 --> 00:49:28.760 these state go down geometrically 00:49:28.760 --> 00:49:29.880 as you go out here. 00:49:29.880 --> 00:49:33.700 What do you think happens if you take this truncated chain 00:49:33.700 --> 00:49:38.310 and then start moving n further and further out? 00:49:38.310 --> 00:49:43.470 Well, if you have the nice case where the probabilities 00:49:43.470 --> 00:49:48.930 are mostly clustered around 0, then as you keep moving it out 00:49:48.930 --> 00:49:50.430 it doesn't make any difference. 00:49:50.430 --> 00:49:53.070 In fact, you look at the answer you've got and see what 00:49:53.070 --> 00:49:56.990 happens in the limit as you take into infinity. 00:49:56.990 --> 00:50:01.970 But, if you look at the case where p is greater than 1/2, 00:50:01.970 --> 00:50:05.070 then everything is clustered up at the right n and every 00:50:05.070 --> 00:50:09.690 time you increase the number of states by 1, blah. 00:50:09.690 --> 00:50:12.240 Everything goes to hell. 00:50:12.240 --> 00:50:17.080 Everything goes to hell unless you realize that you just move 00:50:17.080 --> 00:50:19.820 everything up by 1, and then you have the same case that 00:50:19.820 --> 00:50:21.320 you had before. 00:50:21.320 --> 00:50:23.910 So, you just keep moving things up. 00:50:23.910 --> 00:50:28.750 But there isn't any steady state, and as advertised 00:50:28.750 --> 00:50:30.140 things go to infinity. 00:50:30.140 --> 00:50:34.150 If you look at the case where p is equal to 1/2, as you 00:50:34.150 --> 00:50:37.530 increase the number of states what happens is all the states 00:50:37.530 --> 00:50:39.190 get less and less likely. 00:50:39.190 --> 00:50:42.660 You keep wandering around in an aimless fashion, and 00:50:42.660 --> 00:50:44.170 nothing very interesting happens. 00:50:49.680 --> 00:50:55.010 What we want to do, just to be able to define recurrence, to 00:50:55.010 --> 00:50:59.680 mean that given that you start off in some state i, there's a 00:50:59.680 --> 00:51:03.490 future return to state i with probability 1. 00:51:03.490 --> 00:51:06.490 That's what recurrence should mean, that's what recurrence 00:51:06.490 --> 00:51:08.320 means in English. 00:51:08.320 --> 00:51:10.630 Recurrence means you come back. 00:51:10.630 --> 00:51:15.080 Since it's probabilistic, you have to say, well we don't 00:51:15.080 --> 00:51:17.840 know when we're going to get back, but we are going to get 00:51:17.840 --> 00:51:20.300 back eventually. 00:51:20.300 --> 00:51:23.030 It's what you say to a friend you don't want to see. 00:51:23.030 --> 00:51:25.820 I'll see you sometime. 00:51:25.820 --> 00:51:28.900 And then it might be an infinite expected time. 00:51:28.900 --> 00:51:32.130 But at least you said you're going to make it back. 00:51:32.130 --> 00:51:35.710 We will see the birth-death chain above is recurrent in 00:51:35.710 --> 00:51:40.970 this sense if p is less than 1/2, and it's not recurrent of 00:51:40.970 --> 00:51:45.500 p is greater than 1/2, and we're clearly going to have to 00:51:45.500 --> 00:51:48.020 struggle a little bit to find out what it is that 00:51:48.020 --> 00:51:50.320 p is equal to 1/2. 00:51:50.320 --> 00:51:53.500 And that's a strange case and we'll call non-recurrent when 00:51:53.500 --> 00:51:56.260 we get to that point. 00:51:56.260 --> 00:51:59.220 We're going to use renewal theory to study these 00:51:59.220 --> 00:52:05.200 recurrent chains, which is why we did renewal theory first. 00:52:05.200 --> 00:52:08.460 But first we have to understand first passage times 00:52:08.460 --> 00:52:09.560 a little better. 00:52:09.560 --> 00:52:12.080 We looked at first passage times a little bit when we 00:52:12.080 --> 00:52:14.130 were dealing with Markov chains. 00:52:14.130 --> 00:52:16.490 We looked at first passage time by looking at the 00:52:16.490 --> 00:52:20.040 expected first passage time to get from one state to another 00:52:20.040 --> 00:52:24.140 state, and we found a nice clean way of doing that. 00:52:24.140 --> 00:52:27.150 We'll see how that relates to this, but here instead of 00:52:27.150 --> 00:52:30.850 looking at the expected value we want to find the 00:52:30.850 --> 00:52:35.230 probability that you have a return after some particular 00:52:35.230 --> 00:52:37.624 period of time. 00:52:37.624 --> 00:52:38.995 AUDIENCE: So why did [INAUDIBLE]? 00:52:41.740 --> 00:52:46.640 PROFESSOR: If p is greater than 1/2, what's going to 00:52:46.640 --> 00:52:50.630 happen is you keep wandering further and 00:52:50.630 --> 00:52:51.680 further off to the right. 00:52:51.680 --> 00:52:53.400 You can come back. 00:52:53.400 --> 00:52:56.790 There's a certain probability that no matter how far you get 00:52:56.790 --> 00:52:59.280 out there, there's a probability 00:52:59.280 --> 00:53:01.640 that you can get back. 00:53:01.640 --> 00:53:05.090 But there's also a probability that you won't get back, that 00:53:05.090 --> 00:53:07.960 you keep getting bigger and bigger. 00:53:07.960 --> 00:53:10.450 And this is not obvious. 00:53:10.450 --> 00:53:14.195 But it's something that we're going to sort out as we go on. 00:53:14.195 --> 00:53:18.140 But it certainly is plausible that there's a probability, a 00:53:18.140 --> 00:53:22.460 positive probability that you'll never return. 00:53:22.460 --> 00:53:26.230 Because the further you go, the harder it is to get back. 00:53:26.230 --> 00:53:28.670 And the drift is always to the right. 00:53:28.670 --> 00:53:32.630 And since the drift is always to the right, the further you 00:53:32.630 --> 00:53:38.160 get away, the less probable it gets that you ever get back. 00:53:38.160 --> 00:53:40.880 And we will analyze this. 00:53:40.880 --> 00:53:43.085 I mean, for this case, it's not very hard to analyze. 00:53:47.370 --> 00:53:48.620 Where were we? 00:53:50.770 --> 00:53:51.310 OK. 00:53:51.310 --> 00:53:55.640 The first pass each time probability, we're going to 00:53:55.640 --> 00:53:58.960 call it f sub ij of n. 00:53:58.960 --> 00:54:02.440 This is the probability given that you're in 00:54:02.440 --> 00:54:04.745 state i at time zero. 00:54:04.745 --> 00:54:08.960 It's the probability that you reach state j for the first 00:54:08.960 --> 00:54:10.480 time a time n. 00:54:10.480 --> 00:54:15.340 It's not the probability that you're in state j at time n, 00:54:15.340 --> 00:54:21.020 which is what we call p sub i j to the n. 00:54:21.020 --> 00:54:23.440 It's just the probability that you get there 00:54:23.440 --> 00:54:24.870 for the first time. 00:54:24.870 --> 00:54:26.120 At time n. 00:54:28.780 --> 00:54:34.120 If you remember, quick comment about notation. 00:54:34.120 --> 00:54:49.170 We called p sub ij is the probability that xj, xn equals 00:54:49.170 --> 00:54:53.990 j, given that x0 equals i. 00:54:53.990 --> 00:54:55.240 And f sub ij. 00:54:57.760 --> 00:55:02.820 And now the n is in parentheses as the probability 00:55:02.820 --> 00:55:15.910 that xn is equal to j, given that x0 equals i, and x1 x2, 00:55:15.910 --> 00:55:21.320 so forth are unequal to j. 00:55:21.320 --> 00:55:24.100 Now you see why I get confused with i's and j's. 00:55:28.180 --> 00:55:31.200 The reason I'm using parentheses here and using a 00:55:31.200 --> 00:55:34.580 subscript here is when we're dealing with finite state 00:55:34.580 --> 00:55:39.200 Markov chains, it was just so convenient to view this as the 00:55:39.200 --> 00:55:45.130 ij component of the matrix p taken to the n-th power. 00:55:45.130 --> 00:55:49.430 And this is to remind you that this is a matrix taken to the 00:55:49.430 --> 00:55:50.110 n-th power. 00:55:50.110 --> 00:55:52.170 And the you take the ij element. 00:55:52.170 --> 00:55:54.310 There isn't any matrix multiplication here. 00:55:54.310 --> 00:55:57.550 This is partly because we're dealing with countable state 00:55:57.550 --> 00:55:59.210 and Markov chains here. 00:55:59.210 --> 00:56:03.330 But partly also because this is an uglier thing that we're 00:56:03.330 --> 00:56:04.740 dealing with. 00:56:04.740 --> 00:56:07.960 But you can still work this out. 00:56:07.960 --> 00:56:13.190 What's the probability that you will be in state j for the 00:56:13.190 --> 00:56:17.630 first time at n, given you're back in i? 00:56:17.630 --> 00:56:22.380 It's really the probability that the sum of the 00:56:22.380 --> 00:56:25.780 probability that on the first transition, you 00:56:25.780 --> 00:56:27.780 move from i to k. 00:56:27.780 --> 00:56:30.870 Some k unequal to j, because if we're equal to j, we'd 00:56:30.870 --> 00:56:32.650 already be there, and we wouldn't be looking at 00:56:32.650 --> 00:56:34.320 anything after that. 00:56:34.320 --> 00:56:37.260 So it's the probability we move from i to k. 00:56:37.260 --> 00:56:42.430 And then we have n minus 1 transitions to get from k back 00:56:42.430 --> 00:56:44.040 to j for the first time. 00:56:47.060 --> 00:56:51.650 If you think you understand this, most things like this, 00:56:51.650 --> 00:56:53.180 you can write them either way. 00:56:53.180 --> 00:56:56.450 You can look at the first transition followed by n minus 00:56:56.450 --> 00:57:00.860 1 transitions after it, or you can look at the n minus 1 00:57:00.860 --> 00:57:03.880 transitions first, followed by the last transition. 00:57:03.880 --> 00:57:05.650 Here you can't do that. 00:57:05.650 --> 00:57:08.147 And you should look at it, and figure out why. 00:57:08.147 --> 00:57:08.941 Yes? 00:57:08.941 --> 00:57:09.340 AUDIENCE: I'm sorry. 00:57:09.340 --> 00:57:12.852 I just was wondering if it's the same to say that all of 00:57:12.852 --> 00:57:16.545 the [INAUDIBLE] are given, like you would in [INAUDIBLE]? 00:57:16.545 --> 00:57:18.180 And [INAUDIBLE] 00:57:18.180 --> 00:57:20.546 all of them given just [INAUDIBLE]? 00:57:20.546 --> 00:57:24.900 PROFESSOR: It's the probability that all of the 00:57:24.900 --> 00:57:29.140 x1, x2, x3 and so forth are not equal to j. 00:57:29.140 --> 00:57:31.370 AUDIENCE: Because here and there, you did [INAUDIBLE] 00:57:31.370 --> 00:57:33.384 exactly the same question. 00:57:33.384 --> 00:57:35.610 In here, [INAUDIBLE] 00:57:35.610 --> 00:57:36.610 given [INAUDIBLE]. 00:57:36.610 --> 00:57:39.770 And they're [INAUDIBLE] the same? 00:57:39.770 --> 00:57:42.810 PROFESSOR: This is the same as that, yes. 00:57:42.810 --> 00:57:45.840 Except it's not totally obvious why it is, and I'm 00:57:45.840 --> 00:57:49.460 trying to explain why it is. 00:57:49.460 --> 00:57:52.820 I mean, you look at the first transition you take. 00:57:52.820 --> 00:57:55.320 You start out in state i. 00:57:55.320 --> 00:57:59.690 The next place you go, if it's state j, you're all through. 00:57:59.690 --> 00:58:02.340 And then you've gotten to state j in one step, and 00:58:02.340 --> 00:58:03.940 that's the end of it. 00:58:03.940 --> 00:58:06.870 But if you don't get to state j in one step, you get to some 00:58:06.870 --> 00:58:08.510 other state k. 00:58:08.510 --> 00:58:12.260 And then the question you ask is what's the probability 00:58:12.260 --> 00:58:16.430 starting from state k that you will reach j for the first 00:58:16.430 --> 00:58:19.070 time in n minus 1 steps? 00:58:23.170 --> 00:58:25.130 If you try to match up these two equations, 00:58:25.130 --> 00:58:27.890 it's not at all obvious. 00:58:27.890 --> 00:58:32.100 If you look at what this first transition probability means, 00:58:32.100 --> 00:58:34.820 then it's easy to get this from that. 00:58:34.820 --> 00:58:36.660 And it's easy to get this from that. 00:58:39.640 --> 00:58:42.300 But anyway, that's what it is. 00:58:42.300 --> 00:58:46.990 f sub ij of 1 is equal to pij, and therefore this is equal to 00:58:46.990 --> 00:58:50.150 n greater than one. 00:58:50.150 --> 00:58:53.530 And you can use this recursion, and you can go 00:58:53.530 --> 00:58:57.290 through and calculate all of these n-th order 00:58:57.290 --> 00:59:00.600 probabilities, as some [INAUDIBLE] is going to point 00:59:00.600 --> 00:59:02.160 out in just a minute. 00:59:02.160 --> 00:59:03.830 It takes a while to do that for an 00:59:03.830 --> 00:59:06.580 infinite number of states. 00:59:06.580 --> 00:59:09.492 But we assume there's some nice formula for doing it. 00:59:14.230 --> 00:59:16.680 I mean, this is a formula you could in 00:59:16.680 --> 00:59:20.140 principal, compute it. 00:59:20.140 --> 00:59:23.320 OK, so here's the same formula again. 00:59:23.320 --> 00:59:29.420 I want to relate that to the probabilities of being in 00:59:29.420 --> 00:59:35.240 state j at time n, given the you were in state i at time 0. 00:59:35.240 --> 00:59:38.540 Which by Chapman and Kolmogorov is 00:59:38.540 --> 00:59:40.630 this equation here. 00:59:40.630 --> 00:59:41.750 That's the same. 00:59:41.750 --> 00:59:43.830 You go first to state k, and then from 00:59:43.830 --> 00:59:47.130 kj, n minus one steps. 00:59:47.130 --> 00:59:50.390 Here you're summing over all k, and you're summing over all 00:59:50.390 --> 00:59:55.240 k because in fact, if you get to state j in the first step, 00:59:55.240 --> 00:59:57.910 you can still get to state j again after n 00:59:57.910 --> 00:59:59.760 minus one more steps. 00:59:59.760 --> 01:00:02.800 Here you're only interested in the first time you 01:00:02.800 --> 01:00:04.210 get to state j. 01:00:04.210 --> 01:00:08.590 Here you're interested in any time you get to state j. 01:00:08.590 --> 01:00:12.910 I bring this up because remember what you did when you 01:00:12.910 --> 01:00:19.450 solve that problem related to rewards with a Markov chain? 01:00:19.450 --> 01:00:22.610 If you can think back to that. 01:00:22.610 --> 01:00:27.750 The way to solve the reward problem, trying to find the 01:00:27.750 --> 01:00:32.790 first pass each time from sum i to sum j was to just take 01:00:32.790 --> 01:00:37.830 all the outputs from state j and remove them. 01:00:37.830 --> 01:00:40.540 If you look at this formula, you'll see that that's exactly 01:00:40.540 --> 01:00:44.330 what we've done mathematically by summing only over 01:00:44.330 --> 01:00:46.400 k unequal to j. 01:00:46.400 --> 01:00:50.930 Every time we get to j, we terminate the whole thing, and 01:00:50.930 --> 01:00:52.360 we don't proceed any further. 01:00:52.360 --> 01:00:56.770 So this is just a mathematical way of saying just a Markov 01:00:56.770 --> 01:01:02.010 chain by ripping all of the outputs out of state j, and 01:01:02.010 --> 01:01:05.190 putting a self loop in state j. 01:01:05.190 --> 01:01:07.620 OK, so this is really saying the same sort of thing that 01:01:07.620 --> 01:01:10.570 that was saying, except that was only giving us expected 01:01:10.570 --> 01:01:14.510 value, and this is giving us the whole thing. 01:01:14.510 --> 01:01:20.550 Now, the next thing is we would like to find what looks 01:01:20.550 --> 01:01:23.230 like a distribution function for the same thing. 01:01:25.960 --> 01:01:28.020 This is the probability of reaching j 01:01:28.020 --> 01:01:30.810 by time n or before. 01:01:30.810 --> 01:01:34.560 And the probability that you reach state j by time n or 01:01:34.560 --> 01:01:39.910 before it's just the sum of the probabilities that you 01:01:39.910 --> 01:01:44.150 reach state j for the first time at some time m less than 01:01:44.150 --> 01:01:46.980 or equal to n. 01:01:46.980 --> 01:01:51.560 It's a probability of reaching j by time n or before. 01:01:51.560 --> 01:01:55.800 If this limit now is equal to 1, it means that with 01:01:55.800 --> 01:01:58.890 probability 1, you eventually get there. 01:01:58.890 --> 01:01:59.884 Yes? 01:01:59.884 --> 01:02:00.872 AUDIENCE: I'm sorry, Professor Gallager. 01:02:00.872 --> 01:02:04.824 I'm [INAUDIBLE] definition of fij of n. 01:02:04.824 --> 01:02:06.800 So is it the thing you wrote on the board, or is it the 01:02:06.800 --> 01:02:07.294 thing in the notes? 01:02:07.294 --> 01:02:08.776 Like, I don't see why they're the same. 01:02:08.776 --> 01:02:11.450 Because the thing in the notes, you're only given that 01:02:11.450 --> 01:02:13.600 x0 is equal to i. 01:02:13.600 --> 01:02:15.962 But the thing on the board, you're given that x0 is equal 01:02:15.962 --> 01:02:18.704 to i, and you're given that x1, x2, none of them 01:02:18.704 --> 01:02:20.320 are equal to j. 01:02:20.320 --> 01:02:22.142 PROFESSOR: Oh, I'm sorry. 01:02:22.142 --> 01:02:23.392 This is-- 01:02:25.934 --> 01:02:27.943 you are absolutely right. 01:02:34.750 --> 01:02:39.675 And then given x0 equals i. 01:02:42.860 --> 01:02:44.110 Does that make sense now? 01:02:48.670 --> 01:02:52.240 I can't see in my mind if that's equal to this. 01:02:52.240 --> 01:02:56.350 But if I sit there quietly and look at it for five minutes, I 01:02:56.350 --> 01:02:58.445 realize why this is equal to that. 01:02:58.445 --> 01:03:00.780 And that's why I write it down wrong half the time. 01:03:06.500 --> 01:03:17.050 So, if this limit is equal to 1, it means I can define a 01:03:17.050 --> 01:03:21.110 random variable, which is the amount of time that it takes 01:03:21.110 --> 01:03:26.690 to get to state j for the first time. 01:03:26.690 --> 01:03:30.170 And that random variable is a non defective random variable, 01:03:30.170 --> 01:03:37.160 because I always get to state j eventually, 01:03:37.160 --> 01:03:38.410 starting from state i. 01:03:42.200 --> 01:03:46.930 Now, what was awkward about this was the fact that I had 01:03:46.930 --> 01:03:49.340 to go through all these probabilities before I could 01:03:49.340 --> 01:03:56.470 say let t sub ij be a random variable, and let that random 01:03:56.470 --> 01:03:59.860 variable be the number of steps that it takes to get to 01:03:59.860 --> 01:04:02.760 state j, starting in state i. 01:04:02.760 --> 01:04:05.370 And I couldn't do that because it wasn't clear there was a 01:04:05.370 --> 01:04:06.880 random variable. 01:04:06.880 --> 01:04:10.740 If I said it was as effective random variable, then it would 01:04:10.740 --> 01:04:13.050 be clear that there had to be some sort of 01:04:13.050 --> 01:04:14.570 defective random variable. 01:04:14.570 --> 01:04:16.950 But then I'd have to deal with the question of what do I 01:04:16.950 --> 01:04:19.150 really mean by defective random variable? 01:04:19.150 --> 01:04:22.180 Incidentally, the notes does not define what a defective 01:04:22.180 --> 01:04:24.170 random variable is. 01:04:24.170 --> 01:04:30.430 If you have a non-negative thing that might be a random 01:04:30.430 --> 01:04:36.700 variable, the definition of a defective random variable is 01:04:36.700 --> 01:04:43.370 that all these probabilities exist, but the limit is not 01:04:43.370 --> 01:04:44.200 equal to 1. 01:04:44.200 --> 01:04:48.940 In other words, sometimes the thing never happens. 01:04:48.940 --> 01:04:54.620 So that you can either look at this as you have a thing like 01:04:54.620 --> 01:04:58.150 a random variable, but it matched a lot of sample points 01:04:58.150 --> 01:04:59.670 into infinity. 01:04:59.670 --> 01:05:01.830 Or you can view it as it maps a lot of 01:05:01.830 --> 01:05:03.660 sample points into nothing. 01:05:03.660 --> 01:05:07.840 But you still have a distribution function for it. 01:05:07.840 --> 01:05:09.090 OK. 01:05:10.890 --> 01:05:14.800 But anyway, now we can talk about a random variable, which 01:05:14.800 --> 01:05:20.950 is the time to get from state i to state j, if in fact, it's 01:05:20.950 --> 01:05:23.700 certain that we're going to get there. 01:05:23.700 --> 01:05:31.240 Now, if you start out with a definition of this 01:05:31.240 --> 01:05:38.380 distribution function here, and you play with this formula 01:05:38.380 --> 01:05:42.170 here, you play with that formula, and that formula a 01:05:42.170 --> 01:05:47.010 little bit, you can rewrite the formula for this 01:05:47.010 --> 01:05:51.520 distribution function like thing in the following way. 01:05:51.520 --> 01:05:55.660 This is only different from the thing we wrote before by 01:05:55.660 --> 01:05:57.520 the presence of pij here. 01:05:57.520 --> 01:06:02.430 Otherwise, little fij of n is equal to just a sum over here 01:06:02.430 --> 01:06:03.740 without that. 01:06:03.740 --> 01:06:09.000 With this, you keep adding up, and it keeps getting bigger. 01:06:09.000 --> 01:06:13.750 Why I want to talk about this, it's sort of a detail. 01:06:13.750 --> 01:06:20.800 But this equation is always satisfied by these 01:06:20.800 --> 01:06:24.260 distribution function like things. 01:06:24.260 --> 01:06:29.070 But these equations do not necessarily solve for these 01:06:29.070 --> 01:06:30.920 quantities. 01:06:30.920 --> 01:06:33.320 How do I see that? 01:06:33.320 --> 01:06:42.010 Well, if I plug one in for x sub ij of n, and f sub ij of n 01:06:42.010 --> 01:06:47.130 minus 1 for all i and all j, what do I get? 01:06:47.130 --> 01:06:52.570 I get one that's equal to p sub ij plus the sum over k 01:06:52.570 --> 01:06:55.520 unequal to j of p sub ik times 1. 01:06:55.520 --> 01:06:59.150 So I get 1 equals 1. 01:06:59.150 --> 01:07:05.270 So a solution to this equation is that all of the fij's are 01:07:05.270 --> 01:07:06.520 equal to 1. 01:07:08.690 --> 01:07:10.260 Is this disturbing? 01:07:10.260 --> 01:07:14.360 Well, no, it shouldn't be, because all the time we can 01:07:14.360 --> 01:07:17.190 write equations for things, and the equations don't have a 01:07:17.190 --> 01:07:18.690 unique solution. 01:07:18.690 --> 01:07:21.980 And these equations don't have a unique solution. 01:07:21.980 --> 01:07:24.280 We never said they did. 01:07:24.280 --> 01:07:29.050 But there is a theorem in the notes, which says that if you 01:07:29.050 --> 01:07:33.120 look at all the solutions to this equation, and you take 01:07:33.120 --> 01:07:36.340 the smallest solution, that the smallest solution 01:07:36.340 --> 01:07:37.850 is the right one. 01:07:37.850 --> 01:07:41.800 In other words, the smallest solution is the solution you 01:07:41.800 --> 01:07:44.440 get from doing it this other way. 01:07:51.980 --> 01:07:55.550 I mean, this solution always works, because you can always 01:07:55.550 --> 01:07:58.110 solve for these quantities. 01:07:58.110 --> 01:07:59.850 And you don't have to-- 01:07:59.850 --> 01:08:02.380 I mean, you're just using iteration, so all these 01:08:02.380 --> 01:08:03.630 quantities exist. 01:08:07.100 --> 01:08:09.090 OK. 01:08:09.090 --> 01:08:15.430 Now finally, these equations for going from state i to 01:08:15.430 --> 01:08:21.520 state j also work for going from state j to state j. 01:08:21.520 --> 01:08:25.580 If the probability of going from state j to state j 01:08:25.580 --> 01:08:29.350 eventually is equal to 1, that's what we said recurrent 01:08:29.350 --> 01:08:31.300 ought to mean. 01:08:31.300 --> 01:08:34.790 And now we have a precise way of saying it. 01:08:37.479 --> 01:08:42.060 If f sub jj to infinity is equal to 1, an eventual return 01:08:42.060 --> 01:08:45.880 from state j occurs with probability 1, and the 01:08:45.880 --> 01:08:50.160 sequence of returns is a sequence of renewal epochs in 01:08:50.160 --> 01:08:52.340 a renewal process. 01:08:52.340 --> 01:08:54.819 Nice, huh? 01:08:54.819 --> 01:08:57.460 I mean, when we looked at finite state in Markov chains, 01:08:57.460 --> 01:09:00.220 we just sort of said this and we're done with it. 01:09:00.220 --> 01:09:02.479 Because with finite state and Markov chains, 01:09:02.479 --> 01:09:04.990 what else can happen? 01:09:04.990 --> 01:09:07.990 You start in a particular state. 01:09:07.990 --> 01:09:12.359 If it's a recurrent state, you keep hitting that state, and 01:09:12.359 --> 01:09:13.970 you keep coming back. 01:09:13.970 --> 01:09:15.600 And then you hit it again. 01:09:15.600 --> 01:09:19.600 And the amount of time from one hit to the next hit is 01:09:19.600 --> 01:09:25.010 independent, as the next hit to the next yet hit. 01:09:25.010 --> 01:09:29.859 It was clear that you had a renewal process there. 01:09:29.859 --> 01:09:32.729 Here it's still clear that you have a renewal process, if you 01:09:32.729 --> 01:09:34.490 can define this random variable. 01:09:34.490 --> 01:09:34.779 Yes? 01:09:34.779 --> 01:09:37.648 AUDIENCE: When you say the smallest set you mean sum 01:09:37.648 --> 01:09:39.290 across all terms of the set, and whichever 01:09:39.290 --> 01:09:40.970 gives you the smallest. 01:09:40.970 --> 01:09:42.760 PROFESSOR: No, I mean each of the values 01:09:42.760 --> 01:09:44.669 being as small as possible. 01:09:44.669 --> 01:09:47.689 I mean, it turns out that the solutions are monotonic in 01:09:47.689 --> 01:09:49.779 that sense. 01:09:49.779 --> 01:09:52.569 You can find some solutions where these are big and these 01:09:52.569 --> 01:09:54.950 are small, and others where these are little, 01:09:54.950 --> 01:09:56.200 and these are big. 01:09:59.620 --> 01:10:00.050 OK. 01:10:00.050 --> 01:10:05.520 So now we know what this distribution function is. 01:10:05.520 --> 01:10:08.160 We know what a recurrent state is. 01:10:08.160 --> 01:10:09.670 And what do we do with it? 01:10:13.110 --> 01:10:16.120 Well, we say there's a random variable with this 01:10:16.120 --> 01:10:18.310 distribution function. 01:10:18.310 --> 01:10:19.740 We keep doing things like this. 01:10:19.740 --> 01:10:22.500 We keep getting more and more abstract. 01:10:22.500 --> 01:10:26.140 I mean, instead of saying here's a random variable and 01:10:26.140 --> 01:10:29.270 here's what its distribution function is, we say if this 01:10:29.270 --> 01:10:34.670 was a random variable, then state j is recurrent. 01:10:34.670 --> 01:10:37.280 It has this distribution function. 01:10:37.280 --> 01:10:40.930 The renewal process of returns to j, then, has inter renewal 01:10:40.930 --> 01:10:44.460 intervals with this distribution function. 01:10:44.460 --> 01:10:48.530 As soon as we have this renewal process, we can state 01:10:48.530 --> 01:10:53.930 this lemma, which are things that we proved when we were 01:10:53.930 --> 01:10:55.840 talking about renewal theory. 01:10:55.840 --> 01:10:59.590 And let me try to explain why each of them is true. 01:10:59.590 --> 01:11:02.750 Let's start out by assuming that state j is recurrent. 01:11:02.750 --> 01:11:07.260 In other words, you have this random variable, which is the 01:11:07.260 --> 01:11:11.810 amount of time it takes to get back to j. 01:11:11.810 --> 01:11:19.880 If you get back to j with probability one, then ask the 01:11:19.880 --> 01:11:24.220 question, how long does it take to get back to j for the 01:11:24.220 --> 01:11:26.770 second time? 01:11:26.770 --> 01:11:29.900 Well, you have a random variable, which is the time 01:11:29.900 --> 01:11:33.750 that it takes to get from j to j for the first time. 01:11:33.750 --> 01:11:36.120 You add this to a random variable, which is the amount 01:11:36.120 --> 01:11:39.080 of time to get from j to j the second time. 01:11:39.080 --> 01:11:42.240 You add two random variables together, and you 01:11:42.240 --> 01:11:44.070 get a random variable. 01:11:44.070 --> 01:11:47.520 In other words, the second return is sure if 01:11:47.520 --> 01:11:49.600 the first one is. 01:11:49.600 --> 01:11:53.150 The second return occurs with probability 1 if the first 01:11:53.150 --> 01:11:54.740 return occurs with probability-- 01:11:54.740 --> 01:11:57.830 I mean, you have a very long wait for the first return. 01:11:57.830 --> 01:12:00.400 But after that very long wait, you just have a 01:12:00.400 --> 01:12:02.150 very long wait again. 01:12:02.150 --> 01:12:05.180 But it's going to happen eventually. 01:12:05.180 --> 01:12:08.250 And the third return happens eventually, and the fourth 01:12:08.250 --> 01:12:10.960 return happens eventually. 01:12:10.960 --> 01:12:16.790 So this says that as t goes to infinity, the number of 01:12:16.790 --> 01:12:22.800 returns up to time t is going to be infinite. 01:12:22.800 --> 01:12:24.050 Very small rate, perhaps. 01:12:26.550 --> 01:12:30.130 If I look at the expected value of the number of returns 01:12:30.130 --> 01:12:37.090 up until time t, that's going to be equal to infinity, also. 01:12:37.090 --> 01:12:41.340 I can't think of any easy way of arguing that. 01:12:41.340 --> 01:12:49.510 If I look at the sum over all n of the probability that I'm 01:12:49.510 --> 01:12:53.400 in this state j at time n, and I add up all those 01:12:53.400 --> 01:12:56.420 probabilities, what do I get? 01:12:56.420 --> 01:12:59.040 When I add up all those probabilities, I'm adding up 01:12:59.040 --> 01:13:04.870 the expectations of having a renewal at time n. 01:13:04.870 --> 01:13:07.120 By adding that up for all n. 01:13:07.120 --> 01:13:11.060 And this, in fact, is exactly the same thing as this. 01:13:11.060 --> 01:13:16.840 So if you believe that, you have to believe this also. 01:13:16.840 --> 01:13:20.480 And then we just go back and repeat the thing if these 01:13:20.480 --> 01:13:23.552 things are not random variables. 01:13:23.552 --> 01:13:25.516 Yes? 01:13:25.516 --> 01:13:29.444 AUDIENCE: I don't understand what you mean [INAUDIBLE]? 01:13:29.444 --> 01:13:33.294 I thought that we put [INAUDIBLE] 01:13:33.294 --> 01:13:34.710 2 and 3? 01:13:34.710 --> 01:13:35.660 [INAUDIBLE]? 01:13:35.660 --> 01:13:38.500 PROFESSOR: And given 3, we proof 4. 01:13:38.500 --> 01:13:41.120 AUDIENCE: Oh, [INAUDIBLE]. 01:13:41.120 --> 01:13:42.370 PROFESSOR: Yeah. 01:13:44.660 --> 01:13:47.362 And we don't have time to prove that-- 01:13:47.362 --> 01:13:48.565 AUDIENCE: [INAUDIBLE]. 01:13:48.565 --> 01:13:48.920 PROFESSOR: Yeah. 01:13:48.920 --> 01:13:50.170 OK. 01:13:52.340 --> 01:13:56.140 But in fact, it's not hard to say suppose one doesn't occur, 01:13:56.140 --> 01:13:58.030 then, should the other stuff occur. 01:14:00.790 --> 01:14:07.090 But none of these imply that the expected value of t sub jj 01:14:07.090 --> 01:14:09.440 is finite, or infinite. 01:14:09.440 --> 01:14:12.600 You can always have random variables, which are random 01:14:12.600 --> 01:14:18.150 variables, namely this is an integer value random variable. 01:14:18.150 --> 01:14:20.740 It always takes on some integer value. 01:14:20.740 --> 01:14:26.100 But the expected value of that value might be infinite. 01:14:26.100 --> 01:14:28.750 I mean, you've seen all sorts of random variables like that. 01:14:28.750 --> 01:14:34.400 You have a random variable where the probability of j is 01:14:34.400 --> 01:14:37.380 some constant divided by j squared. 01:14:37.380 --> 01:14:42.065 Now, you multiply j by 1 over 2 squared, you sum it over j, 01:14:42.065 --> 01:14:44.070 and you've got infinity. 01:14:44.070 --> 01:14:50.550 OK, so you might have these random variables, which have 01:14:50.550 --> 01:14:54.810 an expected return time, which is infinite. 01:14:54.810 --> 01:15:00.360 That's exactly what happens when you look at these back 01:15:00.360 --> 01:15:01.670 right at the beginning of today. 01:15:05.990 --> 01:15:07.240 1, 2, 4. 01:15:09.710 --> 01:15:13.705 I'm going to look at this kind of chain here, and I set p 01:15:13.705 --> 01:15:15.990 equal to one half. 01:15:15.990 --> 01:15:21.290 What we said was that you just disperse here the probability 01:15:21.290 --> 01:15:24.980 that you're going to be in any state after a long time goes 01:15:24.980 --> 01:15:30.060 to zero, but you have to get back eventually. 01:15:30.060 --> 01:15:33.390 And if you look at that condition carefully, and you 01:15:33.390 --> 01:15:36.450 put it together with all the things we found out about 01:15:36.450 --> 01:15:39.420 Markov chains, you realize that the expected time to get 01:15:39.420 --> 01:15:43.740 back has to be infinite for this case. 01:15:43.740 --> 01:15:47.540 And the same for the next example we looked at. 01:15:47.540 --> 01:15:51.390 If you look at p equals 1/2, you don't have a steady state 01:15:51.390 --> 01:15:52.680 probability. 01:15:52.680 --> 01:15:55.620 Because you don't have a steady state probability, the 01:15:55.620 --> 01:16:00.460 expected time to get back, the expected recurrence time has 01:16:00.460 --> 01:16:04.470 an infinite expected value. 01:16:04.470 --> 01:16:10.020 The expected recurrence time has to be 1 over the 01:16:10.020 --> 01:16:11.560 probability of the state. 01:16:11.560 --> 01:16:13.540 So the probability of the state is 0. 01:16:13.540 --> 01:16:16.185 The expected return time has to be infinite. 01:16:19.640 --> 01:16:20.890 So, where were we? 01:16:24.630 --> 01:16:28.110 Well, two states are in the same class if they 01:16:28.110 --> 01:16:30.790 communicate. 01:16:30.790 --> 01:16:33.830 Same as for finite state chains. 01:16:33.830 --> 01:16:36.496 That's the same argument we gave, and you get from there 01:16:36.496 --> 01:16:38.320 to there, and there to there. 01:16:38.320 --> 01:16:40.840 And you get from here to there, and here to here, then 01:16:40.840 --> 01:16:42.570 you get from here to there, and here to there. 01:16:42.570 --> 01:16:44.640 OK. 01:16:44.640 --> 01:16:47.910 If states i and j are in the same class, then either both 01:16:47.910 --> 01:16:52.040 are recurrent, or both are transient. 01:16:52.040 --> 01:16:53.930 Which means not recurrent. 01:16:53.930 --> 01:16:58.570 If j is recurrent, then we've already found that this sum is 01:16:58.570 --> 01:17:00.460 equal to infinity. 01:17:00.460 --> 01:17:06.040 And then, oh, I have to explain this a little bit. 01:17:06.040 --> 01:17:10.345 What I'm trying to show you is that if j is recurrent, then i 01:17:10.345 --> 01:17:11.820 has to be recurrent. 01:17:11.820 --> 01:17:14.270 And I know that j is recurrent if this 01:17:14.270 --> 01:17:16.650 sum is equal to infinity. 01:17:16.650 --> 01:17:20.140 And if this sum is equal to infinity, let's look at how we 01:17:20.140 --> 01:17:24.390 can get from i back to i in n steps. 01:17:24.390 --> 01:17:28.700 Since i and j communicate, there's some path of some 01:17:28.700 --> 01:17:34.340 number of steps, say m, which guesses from i to j. 01:17:34.340 --> 01:17:38.860 There's also some path of say, length l which gets us 01:17:38.860 --> 01:17:41.790 from j back to k. 01:17:41.790 --> 01:17:44.820 And if there's this path, and there's this path, and then I 01:17:44.820 --> 01:17:50.670 sum this over all k, that I get is a lower bound to that. 01:17:50.670 --> 01:17:54.060 I first go to j, and then I spend an infinite number of 01:17:54.060 --> 01:17:58.200 states constantly coming back to j. 01:17:58.200 --> 01:18:01.260 And then I finally go back to i. 01:18:01.260 --> 01:18:08.140 And what that says is that this sum of p sub i, i to the 01:18:08.140 --> 01:18:11.992 n, I mean, this is just one set of paths which gets us 01:18:11.992 --> 01:18:13.690 from i to i in n steps. 01:18:23.600 --> 01:18:28.630 If state j is recurrent, then t sub jj might or might not 01:18:28.630 --> 01:18:29.880 have a finite expectation. 01:18:39.560 --> 01:18:44.090 All I want to say here is that if t sub jj, the time to get 01:18:44.090 --> 01:18:49.170 from state j back to state j has an infinite expectation, 01:18:49.170 --> 01:18:51.100 then you call state j. 01:18:51.100 --> 01:18:54.160 No recurrent instead of regular recurrent. 01:18:54.160 --> 01:18:58.100 As we were just saying, if that expected recurrence time 01:18:58.100 --> 01:19:02.570 is infinite, then the steady state probability is going to 01:19:02.570 --> 01:19:07.040 be zero, which says that if something is no recurrent, 01:19:07.040 --> 01:19:10.530 it's going to be even a very, very different way from when 01:19:10.530 --> 01:19:13.830 it's positive recurrent, which is when the return time has a 01:19:13.830 --> 01:19:15.218 finite value. 01:19:15.218 --> 01:19:16.468 AUDIENCE: [INAUDIBLE]? 01:19:18.634 --> 01:19:20.590 PROFESSOR: Yes. 01:19:20.590 --> 01:19:23.460 Which means there isn't a steady state probability. 01:19:23.460 --> 01:19:25.550 To have a steady state probability, you want them all 01:19:25.550 --> 01:19:28.040 to add up to 1. 01:19:28.040 --> 01:19:29.530 So yes, they're all zero. 01:19:29.530 --> 01:19:33.100 And formally, there isn't a steady state probability. 01:19:33.100 --> 01:19:35.220 OK, thank you. 01:19:35.220 --> 01:19:36.470 We will--