WEBVTT 00:00:00.040 --> 00:00:02.460 The following content is provided under a Creative 00:00:02.460 --> 00:00:03.870 Commons license. 00:00:03.870 --> 00:00:06.910 Your support will help MIT OpenCourseWare continue to 00:00:06.910 --> 00:00:08.700 offer high-quality, educational 00:00:08.700 --> 00:00:10.560 resources for free. 00:00:10.560 --> 00:00:13.460 To make a donation or view additional materials from 00:00:13.460 --> 00:00:19.290 hundreds of MIT courses, visit MIT OpenCourseWare at 00:00:19.290 --> 00:00:21.764 ocw.mit.edu. 00:00:21.764 --> 00:00:22.590 PROFESSOR: All right. 00:00:22.590 --> 00:00:26.070 So today, we're going to start by taking stock of what we 00:00:26.070 --> 00:00:27.670 discussed last time, review the 00:00:27.670 --> 00:00:29.810 definition of Markov chains. 00:00:29.810 --> 00:00:32.400 And then most of the lecture, we're going to concentrate on 00:00:32.400 --> 00:00:34.520 their steady-state behavior. 00:00:34.520 --> 00:00:38.860 Meaning, we're going to look at what does a Markov chain do 00:00:38.860 --> 00:00:40.820 if it has run for a long time. 00:00:40.820 --> 00:00:42.860 What can we say about the probabilities of 00:00:42.860 --> 00:00:44.610 the different states? 00:00:44.610 --> 00:00:47.390 So what I would like to repeat is a statement I made last 00:00:47.390 --> 00:00:49.730 time that Markov chains is a very, very 00:00:49.730 --> 00:00:51.910 useful class of models. 00:00:51.910 --> 00:00:56.820 Pretty much anything in the real world can be 00:00:56.820 --> 00:01:01.380 approximately modeled by a Markov chain provided that you 00:01:01.380 --> 00:01:05.019 set your states in the proper way. 00:01:05.019 --> 00:01:06.890 So we're going to see some examples. 00:01:06.890 --> 00:01:09.310 You're going to see more examples in the problems 00:01:09.310 --> 00:01:11.140 you're going to do in homework and recitation. 00:01:13.880 --> 00:01:15.810 On the other hand, we're not going to go 00:01:15.810 --> 00:01:17.830 too deep into examples. 00:01:17.830 --> 00:01:20.310 Rather, we're going to develop the general methodology. 00:02:11.440 --> 00:02:12.690 OK. 00:02:19.310 --> 00:02:19.730 All right. 00:02:19.730 --> 00:02:22.280 Markov models can be pretty general. 00:02:22.280 --> 00:02:24.670 They can run in continuous or discrete time. 00:02:24.670 --> 00:02:27.540 They can have continuous or discrete state spaces. 00:02:27.540 --> 00:02:30.280 In this class, we're going to stick just to the case where 00:02:30.280 --> 00:02:33.880 the state space is discrete and time is discrete because 00:02:33.880 --> 00:02:35.570 this is the simplest case. 00:02:35.570 --> 00:02:38.760 And also, it's the one where you build your intuition 00:02:38.760 --> 00:02:40.830 before going to more general cases 00:02:40.830 --> 00:02:42.840 perhaps in other classes. 00:02:42.840 --> 00:02:47.810 So the state is discrete and finite. 00:02:47.810 --> 00:02:50.130 There's a finite number of states. 00:02:50.130 --> 00:02:53.540 At any point in time, the process is sitting on one of 00:02:53.540 --> 00:02:54.580 those states. 00:02:54.580 --> 00:03:00.520 Time is discrete, so at each unit of time, somebody 00:03:00.520 --> 00:03:03.000 whistles and then the state jumps. 00:03:03.000 --> 00:03:06.110 And when it jumps, it can either land in the same place, 00:03:06.110 --> 00:03:08.180 or it can land somewhere else. 00:03:08.180 --> 00:03:10.740 And the evolution of the process is described by 00:03:10.740 --> 00:03:13.130 transition probabilities. 00:03:13.130 --> 00:03:16.740 Pij is the probability that the next state is j given that 00:03:16.740 --> 00:03:18.630 the current state is i. 00:03:18.630 --> 00:03:21.640 And the most important property that the Markov chain 00:03:21.640 --> 00:03:24.690 has, the definition of a Markov chain or Markov 00:03:24.690 --> 00:03:28.880 process, is that this probability, Pij, is the same 00:03:28.880 --> 00:03:31.490 every time that you land at state i -- 00:03:31.490 --> 00:03:36.320 no matter how you got there and also no matter 00:03:36.320 --> 00:03:38.000 what time it is. 00:03:38.000 --> 00:03:41.650 So the model we have is time homogeneous, which basically 00:03:41.650 --> 00:03:44.630 means that those transition probabilities are the same at 00:03:44.630 --> 00:03:45.720 every time. 00:03:45.720 --> 00:03:49.320 So the model is time invariant in that sense. 00:03:49.320 --> 00:03:52.670 So we're interested in what the chain or the process is 00:03:52.670 --> 00:03:54.780 going to do in the longer run. 00:03:54.780 --> 00:03:57.060 So we're interested, let's say, in the probability that 00:03:57.060 --> 00:04:00.760 starting at a certain state, n times steps later, we find 00:04:00.760 --> 00:04:03.760 ourselves at some particular state j. 00:04:03.760 --> 00:04:06.350 Fortunately, we can calculate those probabilities 00:04:06.350 --> 00:04:07.660 recursively. 00:04:07.660 --> 00:04:12.990 Of course, at the first time 1, the probability of being 1 00:04:12.990 --> 00:04:16.950 time later at state j given that we are right now at state 00:04:16.950 --> 00:04:20.910 i, by definition, this is just the transition probabilities. 00:04:20.910 --> 00:04:26.000 So by knowing these, we can start a recursion that tells 00:04:26.000 --> 00:04:27.730 us the transition probabilities 00:04:27.730 --> 00:04:31.430 for more than n steps. 00:04:31.430 --> 00:04:32.865 This recursion, it's a formula. 00:04:32.865 --> 00:04:34.000 It's always true. 00:04:34.000 --> 00:04:36.860 You can copy it or memorize it. 00:04:36.860 --> 00:04:40.460 But there is a big idea behind that formula that you should 00:04:40.460 --> 00:04:41.480 keep in mind. 00:04:41.480 --> 00:04:43.750 And basically, the divide and conquer idea. 00:04:43.750 --> 00:04:47.530 It's an application of the total probability law. 00:04:47.530 --> 00:04:49.080 So let's fix i. 00:04:49.080 --> 00:04:52.360 The probability that you find yourself at state j, you break 00:04:52.360 --> 00:04:55.770 it up into the probabilities of the different ways that you 00:04:55.770 --> 00:04:57.850 can get to state j. 00:04:57.850 --> 00:04:59.240 What are those different ways? 00:04:59.240 --> 00:05:02.460 The different ways are the different states k at which 00:05:02.460 --> 00:05:05.220 you might find yourself the previous time. 00:05:05.220 --> 00:05:08.400 So with some probability, with this probability, you find 00:05:08.400 --> 00:05:11.070 yourself at state k the previous time. 00:05:11.070 --> 00:05:13.350 And then with probability Pkj, you make a 00:05:13.350 --> 00:05:15.030 transition to state j. 00:05:15.030 --> 00:05:19.000 So this is a possible scenario that takes you to state j 00:05:19.000 --> 00:05:21.090 after n transitions. 00:05:21.090 --> 00:05:25.350 And by summing over all the k's, then we have considered 00:05:25.350 --> 00:05:28.430 all the possible scenarios. 00:05:28.430 --> 00:05:31.590 Now, before we move to the more serious stuff, let's do a 00:05:31.590 --> 00:05:38.420 little bit of warm up to get a handle on how we use 00:05:38.420 --> 00:05:42.170 transition probabilities to calculate more general 00:05:42.170 --> 00:05:45.250 probabilities, then talk about some structural properties of 00:05:45.250 --> 00:05:47.980 Markov chains, and then eventually get to the main 00:05:47.980 --> 00:05:51.310 business of today, which is a steady-state behavior. 00:05:51.310 --> 00:05:56.380 So somebody gives you this chain, and our convention is 00:05:56.380 --> 00:06:00.340 that those arcs that are not shown here corresponds to 0 00:06:00.340 --> 00:06:01.510 probabilities. 00:06:01.510 --> 00:06:05.760 And each one of the arcs that's shown has a non-zero 00:06:05.760 --> 00:06:09.060 probability, and somebody gives it to us. 00:06:09.060 --> 00:06:11.650 Suppose that the chain starts at state 1. 00:06:11.650 --> 00:06:15.720 We want to calculate the probability that it follows 00:06:15.720 --> 00:06:16.830 this particular path. 00:06:16.830 --> 00:06:20.620 That is, it goes to 2, then to 6, then to 7. 00:06:20.620 --> 00:06:22.920 How do we calculate the probability of a particular 00:06:22.920 --> 00:06:24.280 trajectory? 00:06:24.280 --> 00:06:26.700 Well, this is the probability-- 00:06:26.700 --> 00:06:30.980 so it's the probability of the trajectory from 1 that you go 00:06:30.980 --> 00:06:34.480 to 2, then to 6, then to 7. 00:06:34.480 --> 00:06:38.020 So the probability of this trajectory is we use the 00:06:38.020 --> 00:06:39.270 multiplication rule. 00:06:39.270 --> 00:06:42.140 The probability of several things happening is the 00:06:42.140 --> 00:06:44.700 probability that the first thing happens, which is a 00:06:44.700 --> 00:06:47.140 transition from 1 to 2. 00:06:47.140 --> 00:06:53.190 And then given that we are at state 2, we multiply with a 00:06:53.190 --> 00:06:57.120 conditional probability that the next event happens. 00:06:57.120 --> 00:07:02.370 That is, that X2 is equal to 6 given that right now, we are 00:07:02.370 --> 00:07:03.570 at state 1. 00:07:03.570 --> 00:07:07.760 And that conditional probability is just P26. 00:07:07.760 --> 00:07:09.840 And notice that this conditional probability 00:07:09.840 --> 00:07:13.380 applies no matter how we got to state 2. 00:07:13.380 --> 00:07:15.340 This is the Markov assumption. 00:07:15.340 --> 00:07:17.970 So we don't care about the fact that we came in in a 00:07:17.970 --> 00:07:19.120 particular way. 00:07:19.120 --> 00:07:22.370 Given that we came in here, this probability P26, that the 00:07:22.370 --> 00:07:24.640 next transition takes us to 6. 00:07:24.640 --> 00:07:28.620 And then given that all that stuff happened, so given that 00:07:28.620 --> 00:07:32.060 right now, we are at state 6, we need to multiply with a 00:07:32.060 --> 00:07:34.680 conditional probability that the next transition takes us 00:07:34.680 --> 00:07:36.020 to state 7. 00:07:36.020 --> 00:07:39.920 And this is just the P67. 00:07:39.920 --> 00:07:44.140 So to find the probability of following a specific 00:07:44.140 --> 00:07:49.280 trajectory, you just multiply the transition probabilities 00:07:49.280 --> 00:07:50.885 along the particular trajectory. 00:07:54.490 --> 00:07:58.150 Now, if you want to calculate something else, such as for 00:07:58.150 --> 00:08:02.100 example, the probability that 4 time steps later, I find 00:08:02.100 --> 00:08:06.250 myself at state 7 given that they started, let's say, at 00:08:06.250 --> 00:08:07.680 this state. 00:08:07.680 --> 00:08:09.920 How do you calculate this probability? 00:08:09.920 --> 00:08:15.350 One way is to use the recursion for the Rijs that we 00:08:15.350 --> 00:08:17.990 know that it is always valid. 00:08:17.990 --> 00:08:20.970 But for short and simple examples, and with a small 00:08:20.970 --> 00:08:24.890 time horizon, perhaps you can do this in a brute force way. 00:08:24.890 --> 00:08:27.100 What would be the brute force way? 00:08:27.100 --> 00:08:30.410 This is the event that 4 time steps later, I find 00:08:30.410 --> 00:08:32.210 myself at state 7. 00:08:32.210 --> 00:08:36.350 This event can happen in various ways. 00:08:36.350 --> 00:08:41.070 So we can take stock of all the different ways, and write 00:08:41.070 --> 00:08:42.760 down their probabilities. 00:08:42.760 --> 00:08:44.800 So starting from 2. 00:08:44.800 --> 00:08:49.870 One possibility is to follow this trajectory, 1 transition, 00:08:49.870 --> 00:08:53.790 2 transitions, 3 transitions, 4 transitions. 00:08:53.790 --> 00:08:55.810 And that takes me to state 7. 00:08:55.810 --> 00:08:57.760 What's the probability of this trajectory? 00:08:57.760 --> 00:09:05.160 It's P26 times P67 times P76 and then times P67. 00:09:05.160 --> 00:09:08.000 So this is a probability of a particular trajectory that 00:09:08.000 --> 00:09:11.040 takes you to state 7 after 4 time steps. 00:09:11.040 --> 00:09:14.790 But there's other trajectories as well. 00:09:14.790 --> 00:09:16.300 What could be it? 00:09:16.300 --> 00:09:23.470 I might start from state 2, go to state 6, stay at state 6, 00:09:23.470 --> 00:09:26.450 stay at state 6 once more. 00:09:26.450 --> 00:09:31.670 And then from state 6, go to state 7. 00:09:31.670 --> 00:09:36.800 And so there must be one more. 00:09:36.800 --> 00:09:38.470 What's the other one? 00:09:38.470 --> 00:09:45.620 I guess I could go 1, 2, 6, 7. 00:09:45.620 --> 00:09:45.690 OK. 00:09:45.690 --> 00:09:48.012 That's the other trajectory. 00:09:48.012 --> 00:10:01.850 Plus P21 times P12 times P26 and times P67. 00:10:01.850 --> 00:10:05.580 So the transition probability, the overall probability of 00:10:05.580 --> 00:10:09.490 finding ourselves at state 7, is broken down as the sum of 00:10:09.490 --> 00:10:12.580 the probabilities of all the different ways that I can get 00:10:12.580 --> 00:10:16.000 to state 7 in exactly 4 steps. 00:10:16.000 --> 00:10:19.460 So we could always do that without knowing much about 00:10:19.460 --> 00:10:21.750 Markov chains or the general formula for the 00:10:21.750 --> 00:10:24.490 Rij's that we had. 00:10:24.490 --> 00:10:26.370 What's the trouble with this procedure? 00:10:26.370 --> 00:10:29.190 The trouble with this procedure is that the number 00:10:29.190 --> 00:10:34.450 of possible trajectories becomes quite large if this 00:10:34.450 --> 00:10:37.550 index is a little bigger. 00:10:37.550 --> 00:10:41.510 If this 4 was 100, and you ask how many different 00:10:41.510 --> 00:10:45.240 trajectories of length 100 are there to take me from here to 00:10:45.240 --> 00:10:48.170 there, that number of trajectories would be huge. 00:10:48.170 --> 00:10:51.120 It grows exponentially with the time horizon. 00:10:51.120 --> 00:10:55.110 And this kind of calculation would be impossible. 00:10:55.110 --> 00:10:58.310 The basic equation, the recursion that have for the 00:10:58.310 --> 00:11:01.650 Rij's is basically a clever way of organizing this 00:11:01.650 --> 00:11:04.870 computation so that the amount of computation that you do is 00:11:04.870 --> 00:11:06.960 not exponential in the time horizon. 00:11:06.960 --> 00:11:11.100 Rather, it's sort of linear with the time horizon. 00:11:11.100 --> 00:11:14.590 For each time step you need in the time horizon, you just 00:11:14.590 --> 00:11:17.260 keep repeating the same iteration over and over. 00:11:20.460 --> 00:11:20.565 OK. 00:11:20.565 --> 00:11:24.940 Now, the other thing that we discussed last time, briefly, 00:11:24.940 --> 00:11:28.530 was a classification of the different states of the Markov 00:11:28.530 --> 00:11:31.510 chain in two different types. 00:11:31.510 --> 00:11:37.020 A Markov chain, in general, has states that are recurrent, 00:11:37.020 --> 00:11:39.670 which means that from a recurrent state, I can go 00:11:39.670 --> 00:11:41.000 somewhere else. 00:11:41.000 --> 00:11:44.910 But from that somewhere else, there's always some way of 00:11:44.910 --> 00:11:46.040 coming back. 00:11:46.040 --> 00:11:49.910 So if you have a chain of this form, no matter where you go, 00:11:49.910 --> 00:11:52.500 no matter where you start, you can always come 00:11:52.500 --> 00:11:54.540 back where you started. 00:11:54.540 --> 00:11:56.800 States of this kind are called recurrent. 00:11:56.800 --> 00:12:00.320 On the other hand, if you have a few states all this kind, a 00:12:00.320 --> 00:12:04.560 transition of this type, then these states are transient in 00:12:04.560 --> 00:12:07.560 the sense that from those states, it's possible to go 00:12:07.560 --> 00:12:11.520 somewhere else from which place there's no way to come 00:12:11.520 --> 00:12:13.580 back where you started. 00:12:13.580 --> 00:12:18.370 The general structure of a Markov chain is basically a 00:12:18.370 --> 00:12:20.660 collection of transient states. 00:12:20.660 --> 00:12:23.960 You're certain that you are going to leave the transient 00:12:23.960 --> 00:12:27.370 states eventually. 00:12:27.370 --> 00:12:30.300 And after you leave the transient states, you enter 00:12:30.300 --> 00:12:33.570 into a class of states in which you are trapped. 00:12:33.570 --> 00:12:35.890 You are trapped if you get inside here. 00:12:35.890 --> 00:12:39.040 You are trapped if you get inside there. 00:12:39.040 --> 00:12:41.270 This is a recurrent class of states. 00:12:41.270 --> 00:12:43.210 From any state, you can get to any other 00:12:43.210 --> 00:12:44.380 state within this class. 00:12:44.380 --> 00:12:46.100 That's another recurrent class. 00:12:46.100 --> 00:12:49.380 From any state inside here, you can get anywhere else 00:12:49.380 --> 00:12:50.680 inside that class. 00:12:50.680 --> 00:12:52.930 But these 2 classes, you do not communicate. 00:12:52.930 --> 00:12:56.310 If you start here, there's no way to get there. 00:12:56.310 --> 00:12:59.820 If you have 2 recurrent classes, then it's clear that 00:12:59.820 --> 00:13:02.310 the initial conditions of your Markov chain 00:13:02.310 --> 00:13:04.130 matter in the long run. 00:13:04.130 --> 00:13:07.450 If you start here, you will be stuck inside here for the long 00:13:07.450 --> 00:13:09.140 run and similarly about here. 00:13:09.140 --> 00:13:11.680 So the initial conditions do make a difference. 00:13:11.680 --> 00:13:14.720 On the other hand, if this class was not here and you 00:13:14.720 --> 00:13:17.300 only had that class, what would happen to the chain? 00:13:17.300 --> 00:13:18.480 Let's say you start here. 00:13:18.480 --> 00:13:19.490 You move around. 00:13:19.490 --> 00:13:21.730 At some point, you make that transition. 00:13:21.730 --> 00:13:23.260 You get stuck in here. 00:13:23.260 --> 00:13:26.410 And inside here, you keep circulating, because of the 00:13:26.410 --> 00:13:30.210 randomness, you keep visiting all states over and over. 00:13:30.210 --> 00:13:35.800 And hopefully or possibly, in the long run, it doesn't 00:13:35.800 --> 00:13:39.610 matter exactly what time it is or where you started, but the 00:13:39.610 --> 00:13:43.860 probability of being at that particular state is the same 00:13:43.860 --> 00:13:46.310 no matter what the initial condition was. 00:13:46.310 --> 00:13:48.820 So with a single recurrent class, we hope that the 00:13:48.820 --> 00:13:50.750 initial conditions do not matter. 00:13:50.750 --> 00:13:55.780 With 2 or more recurrent classes, initial conditions 00:13:55.780 --> 00:13:58.660 will definitely matter. 00:13:58.660 --> 00:14:03.620 So how many recurrent classes we have is something that has 00:14:03.620 --> 00:14:06.790 to do with the long-term behavior of the chain and the 00:14:06.790 --> 00:14:09.790 extent to which initial conditions matter. 00:14:09.790 --> 00:14:16.000 Another way that initial conditions may matter is if a 00:14:16.000 --> 00:14:19.360 chain has a periodic structure. 00:14:19.360 --> 00:14:21.990 There are many ways of defining periodicity. 00:14:21.990 --> 00:14:25.240 The one that I find sort of the most intuitive and with 00:14:25.240 --> 00:14:27.410 the least amount of mathematical 00:14:27.410 --> 00:14:29.670 symbols is the following. 00:14:29.670 --> 00:14:34.630 The state space of a chain is said to be periodic if you can 00:14:34.630 --> 00:14:39.510 lump the states into a number of clusters called 00:14:39.510 --> 00:14:42.550 d clusters or groups. 00:14:42.550 --> 00:14:45.980 And the transition diagram has the property that from a 00:14:45.980 --> 00:14:48.860 cluster, you always make a transition 00:14:48.860 --> 00:14:50.870 into the next cluster. 00:14:50.870 --> 00:14:52.860 So here d is equal to 2. 00:14:52.860 --> 00:14:55.570 We have two subsets of the state space. 00:14:55.570 --> 00:14:58.130 Whenever we're here, next time we'll be there. 00:14:58.130 --> 00:15:01.080 Whenever we're here, next time we will be there. 00:15:01.080 --> 00:15:03.830 So this chain has a periodic structure. 00:15:03.830 --> 00:15:05.880 There may be still some randomness. 00:15:05.880 --> 00:15:10.270 When I jump from here to here, the state to which I jump may 00:15:10.270 --> 00:15:14.490 be random, but I'm sure that I'm going to be inside here. 00:15:14.490 --> 00:15:17.610 And then next time, I will be sure that I'm inside here. 00:15:17.610 --> 00:15:20.310 This would be a structure of a diagram in which we have a 00:15:20.310 --> 00:15:21.410 period of 3. 00:15:21.410 --> 00:15:25.540 If you start in this lump, you know that the next time, you 00:15:25.540 --> 00:15:27.480 would be in a state inside here. 00:15:27.480 --> 00:15:30.830 Next time, you'll be in a state inside here, and so on. 00:15:30.830 --> 00:15:35.220 So these chains certainly have a periodic structure. 00:15:35.220 --> 00:15:37.860 And that periodicity gets maintained. 00:15:37.860 --> 00:15:42.500 If I start, let's say, at this lump, at even times, 00:15:42.500 --> 00:15:44.500 I'm sure I'm here. 00:15:44.500 --> 00:15:47.660 At odd times, I'm sure I am here. 00:15:47.660 --> 00:15:51.600 So the exact time does matter in determining the 00:15:51.600 --> 00:15:54.660 probabilities of the different states. 00:15:54.660 --> 00:15:57.140 And in particular, the probability of being at the 00:15:57.140 --> 00:16:00.500 particular state cannot convert to a state value. 00:16:00.500 --> 00:16:03.410 The probability of being at the state inside here is going 00:16:03.410 --> 00:16:06.300 to be 0 for all times. 00:16:06.300 --> 00:16:08.290 In general, it's going to be some positive 00:16:08.290 --> 00:16:10.160 number for even times. 00:16:10.160 --> 00:16:13.810 So it goes 0 positive, zero, positive, 0 positive. 00:16:13.810 --> 00:16:15.370 Doesn't settle to anything. 00:16:15.370 --> 00:16:19.860 So when we have periodicity, we do not expect the states 00:16:19.860 --> 00:16:22.600 probabilities to converge to something, but rather, we 00:16:22.600 --> 00:16:24.580 expect them to oscillate. 00:16:24.580 --> 00:16:26.830 Now, how can we tell whether a Markov chain 00:16:26.830 --> 00:16:29.920 is periodic or not? 00:16:29.920 --> 00:16:33.900 There are systematic ways of doing it, but usually with the 00:16:33.900 --> 00:16:36.560 types of examples we see in this class, we just eyeball 00:16:36.560 --> 00:16:39.890 the chain, and we tell whether it's periodic or not. 00:16:39.890 --> 00:16:45.240 So is this chain down here, is it the periodic one or not? 00:16:45.240 --> 00:16:48.870 How many people think it's periodic? 00:16:48.870 --> 00:16:50.680 No one. 00:16:50.680 --> 00:16:51.200 One. 00:16:51.200 --> 00:16:54.070 How many people think it's not periodic? 00:16:54.070 --> 00:16:54.560 OK. 00:16:54.560 --> 00:16:56.270 Not periodic? 00:16:56.270 --> 00:16:57.570 Let's see. 00:16:57.570 --> 00:16:59.490 Let me do some drawing here. 00:17:03.230 --> 00:17:04.119 OK. 00:17:04.119 --> 00:17:05.369 Is it periodic? 00:17:07.856 --> 00:17:09.140 It is. 00:17:09.140 --> 00:17:14.180 From a red state, you can only get to a white state. 00:17:14.180 --> 00:17:17.849 And from a white state, you can only get to a red state. 00:17:17.849 --> 00:17:20.660 So this chain, even though it's not apparent from the 00:17:20.660 --> 00:17:24.589 picture, actually has this structure. 00:17:24.589 --> 00:17:28.600 We can group the states into red states and white states. 00:17:28.600 --> 00:17:32.810 And from reds, we always go to a white, and from a white, we 00:17:32.810 --> 00:17:34.500 always go to a red. 00:17:34.500 --> 00:17:36.540 So this tells you that sometimes 00:17:36.540 --> 00:17:38.680 eyeballing is not as easy. 00:17:38.680 --> 00:17:40.810 If you have lots and lots of states, you might have some 00:17:40.810 --> 00:17:43.280 trouble doing this exercise. 00:17:43.280 --> 00:17:47.230 On the other hand, something very useful to know. 00:17:47.230 --> 00:17:50.270 Sometimes it's extremely easy to tell that the 00:17:50.270 --> 00:17:52.360 chain is not periodic. 00:17:52.360 --> 00:17:53.770 What's that case? 00:17:53.770 --> 00:17:58.600 Suppose that your chain has a self-transition somewhere. 00:17:58.600 --> 00:18:01.670 Then automatically, you know that your 00:18:01.670 --> 00:18:04.660 chain is not periodic. 00:18:04.660 --> 00:18:08.560 So remember, the definition of periodicity requires that if 00:18:08.560 --> 00:18:12.230 you are in a certain group of states, next time, you will be 00:18:12.230 --> 00:18:14.380 in a different group. 00:18:14.380 --> 00:18:16.450 But if you have self-transitions, that 00:18:16.450 --> 00:18:17.600 property is not true. 00:18:17.600 --> 00:18:20.790 If you have a possible self-transition, it's possible 00:18:20.790 --> 00:18:24.530 that you stay inside your own group for the next time step. 00:18:24.530 --> 00:18:29.560 So whenever you have a self-transition, this implies 00:18:29.560 --> 00:18:31.440 that the chain is not periodic. 00:18:34.280 --> 00:18:39.240 And usually that's the simplest and easy way that we 00:18:39.240 --> 00:18:44.210 can tell most of the time that the chain is not periodic. 00:18:44.210 --> 00:18:49.110 So now, we come to the big topic of today, the central 00:18:49.110 --> 00:18:53.080 topic, which is the question about what does the chain do 00:18:53.080 --> 00:18:55.550 in the long run. 00:18:55.550 --> 00:19:00.060 The question we are asking and which we motivated last time 00:19:00.060 --> 00:19:02.510 by looking at an example. 00:19:02.510 --> 00:19:05.790 It's something that did happen in our example of last time. 00:19:05.790 --> 00:19:08.000 So we're asking whether this happens for 00:19:08.000 --> 00:19:09.440 every Markov chain. 00:19:09.440 --> 00:19:12.400 We're asking the question whether the probability of 00:19:12.400 --> 00:19:18.250 being at state j at some time n settles to a 00:19:18.250 --> 00:19:20.350 steady-state value. 00:19:20.350 --> 00:19:22.900 Let's call it pi sub j. 00:19:22.900 --> 00:19:26.960 That these were asking whether this quantity has a limit as n 00:19:26.960 --> 00:19:29.400 goes to infinity, so that we can talk about the 00:19:29.400 --> 00:19:32.300 steady-state probability of state j. 00:19:32.300 --> 00:19:36.060 And furthermore, we asked whether the steady-state 00:19:36.060 --> 00:19:38.900 probability of that state does not depend 00:19:38.900 --> 00:19:40.800 on the initial state. 00:19:40.800 --> 00:19:44.110 In other words, after the chain runs for a long, long 00:19:44.110 --> 00:19:48.620 time, it doesn't matter exactly what time it is, and 00:19:48.620 --> 00:19:51.990 it doesn't matter where the chain started from. 00:19:51.990 --> 00:19:54.860 You can tell me the probability that the state is 00:19:54.860 --> 00:19:58.700 a particular j is approximately the steady-state 00:19:58.700 --> 00:20:00.450 probability pi sub j. 00:20:00.450 --> 00:20:03.470 It doesn't matter exactly what time it is as long as you tell 00:20:03.470 --> 00:20:06.900 me that a lot of time has elapsed so 00:20:06.900 --> 00:20:09.520 that n is a big number. 00:20:09.520 --> 00:20:11.150 So this is the question. 00:20:11.150 --> 00:20:14.210 We have seen examples, and we understand that this is not 00:20:14.210 --> 00:20:16.600 going to be the case always. 00:20:16.600 --> 00:20:19.880 For example, as I just discussed, if we have 2 00:20:19.880 --> 00:20:23.850 recurrent classes, where we start does matter. 00:20:23.850 --> 00:20:28.060 The probability pi(j) of being in that state j is going to be 00:20:28.060 --> 00:20:32.650 0 if we start here, but it would be something positive if 00:20:32.650 --> 00:20:34.710 we were to start in that lump. 00:20:34.710 --> 00:20:37.650 So the initial state does matter if we have multiple 00:20:37.650 --> 00:20:39.690 recurrent classes. 00:20:39.690 --> 00:20:45.590 But if we have only a single class of recurrent states from 00:20:45.590 --> 00:20:48.780 each one of which you can get to any other one, then we 00:20:48.780 --> 00:20:49.980 don't have that problem. 00:20:49.980 --> 00:20:53.010 Then we expect initial conditions to be forgotten. 00:20:53.010 --> 00:20:55.100 So that's one condition that we need. 00:20:58.960 --> 00:21:01.670 And then the other condition that we need is that the chain 00:21:01.670 --> 00:21:02.930 is not periodic. 00:21:02.930 --> 00:21:07.580 If the chain is periodic, then these Rij's do not converge. 00:21:07.580 --> 00:21:09.510 They keep oscillating. 00:21:09.510 --> 00:21:13.190 If we do not have periodicity, then there is hope that we 00:21:13.190 --> 00:21:16.070 will get the convergence that we need. 00:21:16.070 --> 00:21:19.210 It turns out this is the big theory of Markov chains-- the 00:21:19.210 --> 00:21:20.900 steady-state convergence theorem. 00:21:20.900 --> 00:21:26.470 It turns out that yes, the rijs do converge to a 00:21:26.470 --> 00:21:29.510 steady-state limit, which we call a steady-state 00:21:29.510 --> 00:21:35.180 probability as long as these two conditions are satisfied. 00:21:35.180 --> 00:21:37.500 We're not going to prove this theorem. 00:21:37.500 --> 00:21:41.790 If you're really interested, the end of chapter exercises 00:21:41.790 --> 00:21:45.670 basically walk you through a proof of this result, but it's 00:21:45.670 --> 00:21:49.470 probably a little too much for doing it in this class. 00:21:49.470 --> 00:21:52.140 What is the intuitive idea behind this theorem? 00:21:52.140 --> 00:21:52.860 Let's see. 00:21:52.860 --> 00:21:56.830 Let's think intuitively as to why the initial 00:21:56.830 --> 00:21:59.010 state doesn't matter. 00:21:59.010 --> 00:22:02.870 Think of two copies of the chain that starts at different 00:22:02.870 --> 00:22:06.430 initial states, and the state moves randomly. 00:22:06.430 --> 00:22:09.390 As the state moves randomly starting from the two initial 00:22:09.390 --> 00:22:11.900 states a random trajectory. 00:22:11.900 --> 00:22:15.610 as long as you have a single recurrent class at some point, 00:22:15.610 --> 00:22:19.170 and you don't have periodicity at some point, those states, 00:22:19.170 --> 00:22:22.610 those two trajectories, are going to collide. 00:22:22.610 --> 00:22:25.650 Just because there's enough randomness there. 00:22:25.650 --> 00:22:28.830 Even though we started from different places, the state is 00:22:28.830 --> 00:22:30.490 going to be the same. 00:22:30.490 --> 00:22:33.540 After the state becomes the same, then the future of these 00:22:33.540 --> 00:22:37.100 trajectories, probabilistically, is the same 00:22:37.100 --> 00:22:39.800 because they both started at the same state. 00:22:39.800 --> 00:22:42.540 So this means that the initial conditions 00:22:42.540 --> 00:22:45.330 stopped having any influence. 00:22:45.330 --> 00:22:50.040 That's sort of the high-level idea of why the initial state 00:22:50.040 --> 00:22:50.820 gets forgotten. 00:22:50.820 --> 00:22:53.780 Even if you started at different initial states, at 00:22:53.780 --> 00:22:57.030 some time, you may find yourself to be in the same 00:22:57.030 --> 00:22:59.120 state as the other trajectory. 00:22:59.120 --> 00:23:05.370 And once that happens, your initial conditions cannot have 00:23:05.370 --> 00:23:08.210 any effect into the future. 00:23:08.210 --> 00:23:09.230 All right. 00:23:09.230 --> 00:23:15.650 So let's see how we might calculate those steady-state 00:23:15.650 --> 00:23:16.870 probabilities. 00:23:16.870 --> 00:23:19.850 The way we calculate the steady-state probabilities is 00:23:19.850 --> 00:23:24.230 by taking this recursion, which is always true for the 00:23:24.230 --> 00:23:27.150 end-step transition probabilities, and take the 00:23:27.150 --> 00:23:29.400 limit of both sides. 00:23:29.400 --> 00:23:32.970 The limit of this side is the steady-state probability of 00:23:32.970 --> 00:23:36.010 state j, which is pi sub j. 00:23:36.010 --> 00:23:38.140 The limit of this side, we put the 00:23:38.140 --> 00:23:40.120 limit inside the summation. 00:23:40.120 --> 00:23:44.330 Now, as n goes to infinity, n - also goes to infinity. 00:23:44.330 --> 00:23:48.530 So this Rik is going to be the steady-state probability of 00:23:48.530 --> 00:23:51.150 state k starting from state i. 00:23:51.150 --> 00:23:53.170 Now where we started doesn't matter. 00:23:53.170 --> 00:23:54.620 So this is just the steady-state 00:23:54.620 --> 00:23:56.290 probability of state k. 00:23:56.290 --> 00:24:00.000 So this term converges to that one, and this gives us an 00:24:00.000 --> 00:24:03.010 equation that's satisfied by the steady-state 00:24:03.010 --> 00:24:04.030 probabilities. 00:24:04.030 --> 00:24:06.150 Actually, it's not one equation. 00:24:06.150 --> 00:24:10.580 We get one equation for each one of the j's. 00:24:10.580 --> 00:24:13.220 So if we have 10 possible states, we're going to get the 00:24:13.220 --> 00:24:15.580 system of 10 linear equations. 00:24:15.580 --> 00:24:18.840 In the unknowns, pi(1) up to pi(10). 00:24:18.840 --> 00:24:19.210 OK. 00:24:19.210 --> 00:24:20.800 10 unknowns, 10 equations. 00:24:20.800 --> 00:24:23.150 You might think that we are in business. 00:24:23.150 --> 00:24:27.646 But actually, this system of equations is singular. 00:24:27.646 --> 00:24:30.180 0 is a possible solution of this system. 00:24:30.180 --> 00:24:32.900 If you plug pi equal to zero everywhere, the 00:24:32.900 --> 00:24:33.900 equations are satisfied. 00:24:33.900 --> 00:24:37.550 It does not have a unique solution, so maybe we need one 00:24:37.550 --> 00:24:40.580 more condition to get the uniquely solvable system of 00:24:40.580 --> 00:24:41.900 linear equations. 00:24:41.900 --> 00:24:44.350 It turns out that this system of equations 00:24:44.350 --> 00:24:46.150 has a unique solution. 00:24:46.150 --> 00:24:48.980 If you impose an additional condition, which is pretty 00:24:48.980 --> 00:24:52.160 natural, the pi(j)'s are the probabilities of the different 00:24:52.160 --> 00:24:54.960 states, so they should add to 1. 00:24:54.960 --> 00:24:58.410 So you want this one equation to the mix. 00:24:58.410 --> 00:25:05.490 And once you do that, then this system of equations is 00:25:05.490 --> 00:25:07.340 going to have a unique solution. 00:25:07.340 --> 00:25:09.550 And so we can find the steady-state probabilities of 00:25:09.550 --> 00:25:12.980 the Markov chain by just solving these linear 00:25:12.980 --> 00:25:16.130 equations, which is numerically straightforward. 00:25:16.130 --> 00:25:18.790 Now, these equations are quite important. 00:25:18.790 --> 00:25:23.240 I mean, they're the central point in the Markov chain. 00:25:23.240 --> 00:25:24.220 They have a name. 00:25:24.220 --> 00:25:27.030 They're called the balance equations. 00:25:27.030 --> 00:25:31.260 And it's worth interpreting them in a 00:25:31.260 --> 00:25:33.450 somewhat different way. 00:25:33.450 --> 00:25:37.030 So intuitively, one can sometimes think of 00:25:37.030 --> 00:25:39.290 probabilities as frequencies. 00:25:39.290 --> 00:25:45.780 For example, if I toss an unbiased coin, probability 1/2 00:25:45.780 --> 00:25:49.600 of heads, you could also say that if I keep flipping that 00:25:49.600 --> 00:25:52.510 coin, in the long run, 1/2 of the time, I'm 00:25:52.510 --> 00:25:54.500 going to see heads. 00:25:54.500 --> 00:25:58.910 Similarly, let's try an interpretation of this pi(j), 00:25:58.910 --> 00:26:02.500 the steady-state probability, the long-term probability of 00:26:02.500 --> 00:26:04.980 finding myself at state j. 00:26:04.980 --> 00:26:08.440 Let's try to interpret it as the frequency with which I 00:26:08.440 --> 00:26:12.620 find myself at state j if I run a very, very long 00:26:12.620 --> 00:26:14.940 trajectory over that Markov chain. 00:26:14.940 --> 00:26:18.400 So the trajectory moves around, visits states. 00:26:18.400 --> 00:26:22.470 It visits the different states with different frequencies. 00:26:22.470 --> 00:26:27.380 And let's think of the probability that you are at a 00:26:27.380 --> 00:26:32.270 certain state as being sort of the same as the frequency of 00:26:32.270 --> 00:26:34.420 visiting that state. 00:26:34.420 --> 00:26:37.290 This turns out to be a correct statement. 00:26:37.290 --> 00:26:41.040 If you were more rigorous, you would have to prove it. 00:26:41.040 --> 00:26:44.390 But it's an interpretation which is valid and which gives 00:26:44.390 --> 00:26:48.560 us a lot of intuition about what these equation is saying. 00:26:48.560 --> 00:26:50.170 So let's think as follows. 00:26:50.170 --> 00:26:54.240 Let's focus on a particular state j, and think of 00:26:54.240 --> 00:27:00.660 transitions into the state j versus transitions out of the 00:27:00.660 --> 00:27:05.410 state j, or transitions into j versus transitions 00:27:05.410 --> 00:27:07.080 starting from j. 00:27:07.080 --> 00:27:10.010 So transition starting from that includes a 00:27:10.010 --> 00:27:11.260 self-transition. 00:27:14.980 --> 00:27:15.110 Ok. 00:27:15.110 --> 00:27:18.300 So how often do we get a transition, if we interpret 00:27:18.300 --> 00:27:21.230 the pi(j)'s as frequencies, how often do we get a 00:27:21.230 --> 00:27:23.110 transition into j? 00:27:23.110 --> 00:27:25.870 Here's how we think about it. 00:27:25.870 --> 00:27:31.110 A fraction pi(1) of the time, we're going to be at state 1. 00:27:31.110 --> 00:27:35.070 Whenever we are at state 1, there's going to be a 00:27:35.070 --> 00:27:40.550 probability, P1j, that we make a transition of this kind. 00:27:40.550 --> 00:27:44.730 So out of the times that we're at state 1, there's a 00:27:44.730 --> 00:27:50.141 frequency, P1j with which the next transition is into j. 00:27:53.230 --> 00:27:57.870 So out of the overall number of transitions that happen at 00:27:57.870 --> 00:28:01.820 the trajectory, what fraction of those transitions is 00:28:01.820 --> 00:28:03.700 exactly of that kind? 00:28:03.700 --> 00:28:06.710 That fraction of transitions is the fraction of time that 00:28:06.710 --> 00:28:11.940 you find yourself at 1 times the fraction with which out of 00:28:11.940 --> 00:28:15.700 one you happen to visit next state j. 00:28:15.700 --> 00:28:19.300 So we interpreted this number as the frequency of 00:28:19.300 --> 00:28:21.610 transitions of this kind. 00:28:21.610 --> 00:28:24.780 At any given time, our chain can do transitions of 00:28:24.780 --> 00:28:28.470 different kinds, transitions of the general form from some 00:28:28.470 --> 00:28:30.670 k, I go to some l. 00:28:30.670 --> 00:28:33.600 So we try to do some accounting. 00:28:33.600 --> 00:28:37.740 How often does a transition of each particular kind happen? 00:28:37.740 --> 00:28:40.950 And this is the frequency with which transitions of that 00:28:40.950 --> 00:28:42.970 particular kind happens. 00:28:42.970 --> 00:28:44.610 Now, what's the total frequency of 00:28:44.610 --> 00:28:46.870 transitions into state j? 00:28:46.870 --> 00:28:49.880 Transitions into state j can happen by having a transition 00:28:49.880 --> 00:28:54.510 from 1 to j, from 2 to j, or from state m to j. 00:28:54.510 --> 00:28:58.590 So to find the total frequency with which we would observe 00:28:58.590 --> 00:29:03.960 transitions into j is going to be this particular sum. 00:29:03.960 --> 00:29:09.090 Now, you are at state j if and only if the last transition 00:29:09.090 --> 00:29:11.440 was into state j. 00:29:11.440 --> 00:29:16.230 So the frequency with which you are at j is the frequency 00:29:16.230 --> 00:29:20.100 with which transitions into j happen. 00:29:20.100 --> 00:29:24.020 So this equation expresses exactly that statement. 00:29:24.020 --> 00:29:27.740 The probability of being at state j is the sum of the 00:29:27.740 --> 00:29:32.440 probabilities that the last transition was into state j. 00:29:32.440 --> 00:29:35.120 Or in terms of frequencies, the frequency with which you 00:29:35.120 --> 00:29:39.170 find yourself at state j is the sum of the frequencies of 00:29:39.170 --> 00:29:42.580 all the possible transition types that take you 00:29:42.580 --> 00:29:45.360 inside state j. 00:29:45.360 --> 00:29:47.770 So that's a useful intuition to have, and we're going to 00:29:47.770 --> 00:29:52.360 see an example a little later that it gives us short cuts 00:29:52.360 --> 00:29:55.240 into analyzing Markov chains. 00:29:55.240 --> 00:29:58.270 But before we move, let's revisit the 00:29:58.270 --> 00:30:01.450 example from last time. 00:30:01.450 --> 00:30:03.560 And let us write down the balance 00:30:03.560 --> 00:30:06.090 equations for this example. 00:30:06.090 --> 00:30:09.380 So the steady-state probability that I find myself 00:30:09.380 --> 00:30:16.370 at state 1 is the probability that the previous time I was 00:30:16.370 --> 00:30:21.550 at state 1 and I made a self-transition-- 00:30:21.550 --> 00:30:25.540 So the probability that I was here last time and I made a 00:30:25.540 --> 00:30:28.290 transition of this kind, plus the probability that the last 00:30:28.290 --> 00:30:32.220 time I was here and I made a transition of that kind. 00:30:32.220 --> 00:30:36.100 So plus pi(2) times 0.2. 00:30:36.100 --> 00:30:43.060 And similarly, for the other states, the steady-state 00:30:43.060 --> 00:30:46.250 probably that I find myself at state 2 is the probability 00:30:46.250 --> 00:30:50.650 that last time I was at state 1 and I made a transition into 00:30:50.650 --> 00:30:53.780 state 2, plus the probability that the last time I was at 00:30:53.780 --> 00:30:57.750 state 2 and I made the transition into state 1. 00:30:57.750 --> 00:30:59.910 Now, these are two equations and two 00:30:59.910 --> 00:31:02.040 unknowns, pi(1) and pi(2). 00:31:02.040 --> 00:31:06.150 But you notice that both of these equations tell you the 00:31:06.150 --> 00:31:07.170 same thing. 00:31:07.170 --> 00:31:12.226 They tell you that 0.5pi(1) equals 0.2pi(2). 00:31:17.770 --> 00:31:21.820 Either of these equations tell you exactly this if you move 00:31:21.820 --> 00:31:22.910 terms around. 00:31:22.910 --> 00:31:25.450 So these two equations are not really two equations. 00:31:25.450 --> 00:31:27.010 It's just one equation. 00:31:27.010 --> 00:31:30.520 They are linearly dependent equations, and in order to 00:31:30.520 --> 00:31:33.320 solve the problem, we need the additional condition that 00:31:33.320 --> 00:31:36.300 pi(1) + pi(2) is equal to 1. 00:31:36.300 --> 00:31:38.420 Now, we have our system of two equations, 00:31:38.420 --> 00:31:39.880 which you can solve. 00:31:39.880 --> 00:31:45.030 And once you solve it, you find that pi(1) is 2/7 and 00:31:45.030 --> 00:31:48.610 pi(2) is 5/7. 00:31:48.610 --> 00:31:52.880 So these are the steady state probabilities of the two 00:31:52.880 --> 00:31:54.130 different states. 00:31:56.600 --> 00:32:01.770 If we start this chain, at some state, let's say state 1, 00:32:01.770 --> 00:32:07.050 and we let it run for a long, long time, the chain settles 00:32:07.050 --> 00:32:08.580 into steady state. 00:32:08.580 --> 00:32:09.440 What does that mean? 00:32:09.440 --> 00:32:12.470 It does not mean that the state itself 00:32:12.470 --> 00:32:13.960 enters steady state. 00:32:13.960 --> 00:32:17.390 The state will keep jumping around forever and ever. 00:32:17.390 --> 00:32:21.040 It will keep visiting both states once in a while. 00:32:21.040 --> 00:32:23.250 So the jumping never ceases. 00:32:23.250 --> 00:32:25.790 The thing that gets into steady state is the 00:32:25.790 --> 00:32:30.180 probability of finding yourself at state 1. 00:32:30.180 --> 00:32:34.020 So the probability that you find yourself at state 1 at 00:32:34.020 --> 00:32:37.640 time one trillion is approximately 2/7. 00:32:37.640 --> 00:32:40.990 The probability you find yourself at state 1 at time 00:32:40.990 --> 00:32:45.590 two trillions is again, approximately 2/7. 00:32:45.590 --> 00:32:48.640 So the probability of being in that state settles into a 00:32:48.640 --> 00:32:52.270 steady value. 00:32:52.270 --> 00:32:55.630 That's what the steady-state convergence means. 00:32:55.630 --> 00:32:58.370 It's convergence of probabilities, not convergence 00:32:58.370 --> 00:33:00.680 of the process itself. 00:33:00.680 --> 00:33:04.750 And again, the two main things that are happening in this 00:33:04.750 --> 00:33:07.940 example, and more generally, when we have a single class 00:33:07.940 --> 00:33:10.650 and no periodicity, is that the initial 00:33:10.650 --> 00:33:12.600 state does not matter. 00:33:12.600 --> 00:33:15.980 There's enough randomness here so that no matter where you 00:33:15.980 --> 00:33:19.720 start, the randomness kind of washes out any memory of where 00:33:19.720 --> 00:33:20.780 you started. 00:33:20.780 --> 00:33:23.270 And also in this example, clearly, we do not have 00:33:23.270 --> 00:33:27.500 periodicity because we have self arcs. 00:33:27.500 --> 00:33:30.960 And this, in particular, implies that the exact time 00:33:30.960 --> 00:33:32.210 does not matter. 00:33:35.740 --> 00:33:41.510 So now, we're going to spend the rest of our time by 00:33:41.510 --> 00:33:45.330 looking into a special class of chains that's a little 00:33:45.330 --> 00:33:47.450 easier to deal with, but still, it's 00:33:47.450 --> 00:33:49.436 an important class. 00:33:49.436 --> 00:33:52.010 So what's the moral from here? 00:33:52.010 --> 00:33:55.680 This was a simple example with two states, and we could find 00:33:55.680 --> 00:33:59.440 the steady-state probabilities by solving a simple system of 00:33:59.440 --> 00:34:01.320 two-by-two equations. 00:34:01.320 --> 00:34:05.690 If you have a chain with 100 states, it's no problem for a 00:34:05.690 --> 00:34:09.120 computer to solve a system of 100-by-100 equations. 00:34:09.120 --> 00:34:12.800 But you can certainly not do it by hand, and usually, you 00:34:12.800 --> 00:34:15.560 cannot get any closed-form formulas, so you do not 00:34:15.560 --> 00:34:17.969 necessarily get a lot of insight. 00:34:17.969 --> 00:34:21.330 So one looks for special structures or models that 00:34:21.330 --> 00:34:25.610 maybe give you a little more insight or maybe lead you to 00:34:25.610 --> 00:34:27.690 closed-form formulas. 00:34:27.690 --> 00:34:32.130 And an interesting subclass of Markov chains in which all of 00:34:32.130 --> 00:34:35.080 these nice things do happen, is the class 00:34:35.080 --> 00:34:39.080 of birth/death processes. 00:34:39.080 --> 00:34:41.500 So what's a birth/death process? 00:34:41.500 --> 00:34:44.510 It's a Markov chain who's diagram looks 00:34:44.510 --> 00:34:46.810 basically like this. 00:34:46.810 --> 00:34:52.020 So the states of the Markov chain start from 0 and go up 00:34:52.020 --> 00:34:54.460 to some finite integer m. 00:34:54.460 --> 00:34:57.400 What's special about this chain is that if you are at a 00:34:57.400 --> 00:35:02.710 certain state, next time you can either go up by 1, you can 00:35:02.710 --> 00:35:06.640 go down by 1, or you can stay in place. 00:35:06.640 --> 00:35:09.820 So it's like keeping track of some population 00:35:09.820 --> 00:35:11.310 at any given time. 00:35:11.310 --> 00:35:13.820 One person gets born, or one person 00:35:13.820 --> 00:35:15.680 dies, or nothing happens. 00:35:15.680 --> 00:35:19.150 Again, we're not accounting for twins here. 00:35:19.150 --> 00:35:24.430 So we're given this structure, and we are given the 00:35:24.430 --> 00:35:27.660 transition probabilities, the probabilities associated with 00:35:27.660 --> 00:35:29.630 transitions of the different types. 00:35:29.630 --> 00:35:32.880 So we use P's for the upward transitions, Q's for the 00:35:32.880 --> 00:35:34.550 downward transitions. 00:35:34.550 --> 00:35:37.830 An example of a chain of this kind was the supermarket 00:35:37.830 --> 00:35:40.410 counter model that we discussed last time. 00:35:40.410 --> 00:35:45.080 That is, a customer arrives, so this increments 00:35:45.080 --> 00:35:46.310 the state by 1. 00:35:46.310 --> 00:35:49.670 Or a customer finishes service, in which case, the 00:35:49.670 --> 00:35:53.020 state gets decremented by 1, or nothing happens in which 00:35:53.020 --> 00:35:55.500 you stay in place, and so on. 00:35:55.500 --> 00:35:59.880 In the supermarket model, these P's inside here were all 00:35:59.880 --> 00:36:03.780 taken to be equal because we assume that the arrival rate 00:36:03.780 --> 00:36:07.400 was sort of constant at each time slot. 00:36:07.400 --> 00:36:10.960 But you can generalize a little bit by assuming that 00:36:10.960 --> 00:36:15.330 these transition probabilities P1 here, P2 there, and so on 00:36:15.330 --> 00:36:18.190 may be different from state to state. 00:36:18.190 --> 00:36:21.750 So in general, from state i, there's going to be a 00:36:21.750 --> 00:36:24.400 transition probability Pi that the next 00:36:24.400 --> 00:36:26.210 transition is upwards. 00:36:26.210 --> 00:36:29.820 And there's going to be a probability Qi that the next 00:36:29.820 --> 00:36:31.910 transition is downwards. 00:36:31.910 --> 00:36:35.310 And so from that state, the probability that the next 00:36:35.310 --> 00:36:37.640 transition is downwards is going to be Q_(i+1). 00:36:40.930 --> 00:36:43.650 So this is the structure of our chain. 00:36:43.650 --> 00:36:47.580 As I said, it's a crude model of what happens at the 00:36:47.580 --> 00:36:53.820 supermarket counter but it's also a good model for lots of 00:36:53.820 --> 00:36:55.370 types of service systems. 00:36:55.370 --> 00:36:59.400 Again, you have a server somewhere that has a buffer. 00:36:59.400 --> 00:37:00.970 Jobs come into the buffer. 00:37:00.970 --> 00:37:02.460 So the buffer builds up. 00:37:02.460 --> 00:37:06.880 The server processes jobs, so the buffer keeps going down. 00:37:06.880 --> 00:37:10.470 And the state of the chain would be the number of jobs 00:37:10.470 --> 00:37:12.570 that you have inside your buffer. 00:37:12.570 --> 00:37:18.690 Or you could be thinking about active phone calls out of a 00:37:18.690 --> 00:37:19.680 certain city. 00:37:19.680 --> 00:37:22.520 Each time that the phone call is placed, the number of 00:37:22.520 --> 00:37:24.090 active phone calls goes up by 1. 00:37:24.090 --> 00:37:28.360 Each time that the phone call stops happening, is 00:37:28.360 --> 00:37:31.790 terminated, then the count goes down by 1. 00:37:31.790 --> 00:37:34.710 So it's for processes of this kind that a model with this 00:37:34.710 --> 00:37:36.890 structure is going to show up. 00:37:36.890 --> 00:37:39.240 And they do show up in many, many models. 00:37:39.240 --> 00:37:43.730 Or you can think about the number of people in a certain 00:37:43.730 --> 00:37:45.690 population that have a disease. 00:37:45.690 --> 00:37:51.010 So 1 more person gets the flu, the count goes up. 00:37:51.010 --> 00:37:55.320 1 more person gets healed, the count goes down. 00:37:55.320 --> 00:37:58.350 And these probabilities in such an epidemic model would 00:37:58.350 --> 00:38:02.270 certainly depend on the current state. 00:38:02.270 --> 00:38:06.280 If lots of people already have the flu, the probability that 00:38:06.280 --> 00:38:10.010 another person catches it would be pretty high. 00:38:10.010 --> 00:38:13.800 Whereas, if no one has the flu, then the probability that 00:38:13.800 --> 00:38:16.820 you get a transition where someone catches the flu, that 00:38:16.820 --> 00:38:18.960 probability would be pretty small. 00:38:18.960 --> 00:38:26.990 So the transition rates, the incidence of new people who 00:38:26.990 --> 00:38:30.010 have the disease definitely depends on how many people 00:38:30.010 --> 00:38:31.560 already have the disease. 00:38:31.560 --> 00:38:34.970 And that motivates cases where those P's, the upward 00:38:34.970 --> 00:38:39.220 transition probabilities, depend on the 00:38:39.220 --> 00:38:42.400 state of the chain. 00:38:42.400 --> 00:38:44.040 So how do we study this chain? 00:38:44.040 --> 00:38:49.220 You can sit down and write the system of n linear equations 00:38:49.220 --> 00:38:50.550 in the pi's. 00:38:50.550 --> 00:38:52.920 And this way, find the steady-state probabilities of 00:38:52.920 --> 00:38:53.790 this chain. 00:38:53.790 --> 00:38:55.850 But this is a little harder. 00:38:55.850 --> 00:38:59.310 It's more work than one actually needs to do. 00:38:59.310 --> 00:39:03.200 There's a very clever shortcut that applies 00:39:03.200 --> 00:39:05.160 to birth/death processes. 00:39:05.160 --> 00:39:08.410 And it's based on the frequency interpretation that 00:39:08.410 --> 00:39:10.000 we discussed a little while ago. 00:39:14.070 --> 00:39:17.390 Let's put a line somewhere in the middle of this chain, and 00:39:17.390 --> 00:39:21.710 focus on the relation between this part and that part in 00:39:21.710 --> 00:39:22.750 more detail. 00:39:22.750 --> 00:39:25.460 So think of the chain continuing in this direction, 00:39:25.460 --> 00:39:26.390 that direction. 00:39:26.390 --> 00:39:30.020 But let's just focus on 2 adjacent states, and look at 00:39:30.020 --> 00:39:32.270 this particular cut. 00:39:32.270 --> 00:39:34.270 What is the chain going to do? 00:39:34.270 --> 00:39:35.550 Let's say it starts here. 00:39:35.550 --> 00:39:37.250 It's going to move around. 00:39:37.250 --> 00:39:40.280 At some point, it makes a transition to the other side. 00:39:40.280 --> 00:39:42.860 And that's a transition from i to i+1. 00:39:42.860 --> 00:39:45.470 It stays on the other side for some time. 00:39:45.470 --> 00:39:48.490 It gets here, and eventually, it's going to make a 00:39:48.490 --> 00:39:50.340 transition to this side. 00:39:50.340 --> 00:39:53.110 Then it keeps moving and so on. 00:39:53.110 --> 00:39:57.680 Now, there's a certain balance that must be obeyed here. 00:39:57.680 --> 00:40:01.300 The number of upward transitions through this line 00:40:01.300 --> 00:40:04.220 cannot be very different from the number of downward 00:40:04.220 --> 00:40:06.080 transitions. 00:40:06.080 --> 00:40:09.530 Because we cross this way, then next time, 00:40:09.530 --> 00:40:10.630 we'll cross that way. 00:40:10.630 --> 00:40:12.580 Then next time, we'll cross this way. 00:40:12.580 --> 00:40:13.970 We'll cross that way. 00:40:13.970 --> 00:40:18.940 So the frequency with which transitions of this kind occur 00:40:18.940 --> 00:40:21.500 has to be the same as the long-term frequency that 00:40:21.500 --> 00:40:24.430 transitions of that kind occur. 00:40:24.430 --> 00:40:28.510 You cannot go up 100 times and go down only 50 times. 00:40:28.510 --> 00:40:31.740 If you have gone up 100 times, it means that you have gone 00:40:31.740 --> 00:40:36.960 down 99, or 100, or 101, but nothing much more 00:40:36.960 --> 00:40:38.570 different than that. 00:40:38.570 --> 00:40:41.440 So the frequency with which transitions of 00:40:41.440 --> 00:40:43.670 this kind get observed. 00:40:43.670 --> 00:40:47.610 That is, out of a large number of transitions, what fraction 00:40:47.610 --> 00:40:49.890 of transitions are of these kind? 00:40:49.890 --> 00:40:52.160 That fraction has to be the same as the fraction of 00:40:52.160 --> 00:40:54.870 transitions that happened to be of that kind. 00:40:54.870 --> 00:40:56.620 What are these fractions? 00:40:56.620 --> 00:40:58.480 We discussed that before. 00:40:58.480 --> 00:41:04.840 The fraction of times at which transitions of this kind are 00:41:04.840 --> 00:41:08.340 observed is the fraction of time that we happen to be at 00:41:08.340 --> 00:41:09.350 that state. 00:41:09.350 --> 00:41:11.790 And out of the times that we are in that state, the 00:41:11.790 --> 00:41:15.010 fraction of transitions that happen to be upward 00:41:15.010 --> 00:41:16.040 transitions. 00:41:16.040 --> 00:41:20.820 So this is the frequency with which transitions of this kind 00:41:20.820 --> 00:41:22.430 are observed. 00:41:22.430 --> 00:41:25.350 And with the same argument, this is the frequency with 00:41:25.350 --> 00:41:28.670 which transitions of that kind are observed. 00:41:28.670 --> 00:41:31.120 Since these two frequencies are the same, these two 00:41:31.120 --> 00:41:34.390 numbers must be the same, and we get an equation that 00:41:34.390 --> 00:41:38.350 relates the Pi to P_(i+1). 00:41:38.350 --> 00:41:43.040 This has a nice form because it gives us a recursion. 00:41:43.040 --> 00:41:45.760 If we knew pi(i), we could then 00:41:45.760 --> 00:41:48.860 immediately calculate pi(i+1). 00:41:48.860 --> 00:41:51.860 So it's a system of equations that's very 00:41:51.860 --> 00:41:54.860 easy to solve almost. 00:41:54.860 --> 00:41:57.320 But how do we get started? 00:41:57.320 --> 00:42:01.850 If I knew pi(0), I could find by pi(1) and then use this 00:42:01.850 --> 00:42:05.330 recursion to find pi(2), pi(3), and so on. 00:42:05.330 --> 00:42:06.970 But we don't know pi(0). 00:42:06.970 --> 00:42:09.920 It's one more unknown. 00:42:09.920 --> 00:42:14.290 It's an unknown, and we need to actually use the extra 00:42:14.290 --> 00:42:20.000 normalization condition that the sum of the pi's is 1. 00:42:20.000 --> 00:42:23.810 And after we use that normalization condition, then 00:42:23.810 --> 00:42:26.580 we can find all of the pi's. 00:42:32.550 --> 00:42:38.450 So you basically fix pi(0) as a symbol, solve this equation 00:42:38.450 --> 00:42:41.580 symbolically, and everything gets 00:42:41.580 --> 00:42:43.830 expressed in terms of pi(0). 00:42:43.830 --> 00:42:46.510 And then use that normalization condition to 00:42:46.510 --> 00:42:48.700 find pi(0), and you're done. 00:42:48.700 --> 00:42:51.570 Let's illustrate the details of this procedure on a 00:42:51.570 --> 00:42:53.690 particular special case. 00:42:53.690 --> 00:42:57.360 So in our special case, we're going to simplify things now 00:42:57.360 --> 00:43:01.760 by assuming that all those upward P's are the same, and 00:43:01.760 --> 00:43:05.780 all of those downward Q's are the same. 00:43:05.780 --> 00:43:08.860 So at each point in time, if you're sitting somewhere in 00:43:08.860 --> 00:43:13.060 the middle, you have probability P of moving up and 00:43:13.060 --> 00:43:16.710 probability Q of moving down. 00:43:16.710 --> 00:43:23.460 This rho, the ratio of P/Q is frequency of going up versus 00:43:23.460 --> 00:43:25.670 frequency of going down. 00:43:25.670 --> 00:43:29.070 If it's a service system, you can think of it as a measure 00:43:29.070 --> 00:43:32.140 of how loaded the system is. 00:43:32.140 --> 00:43:39.100 If P is equal to Q, it's means that if you're at this state, 00:43:39.100 --> 00:43:42.700 you're equally likely to move left or right, so the system 00:43:42.700 --> 00:43:44.820 is kind of balanced. 00:43:44.820 --> 00:43:46.980 The state doesn't have a tendency to move in this 00:43:46.980 --> 00:43:49.450 direction or in that direction. 00:43:49.450 --> 00:43:53.860 If rho is bigger than 1 so that P is bigger than Q, it 00:43:53.860 --> 00:43:56.730 means that whenever I'm at some state in the middle, I'm 00:43:56.730 --> 00:44:00.830 more likely to move right rather than move left, which 00:44:00.830 --> 00:44:04.230 means that my state, of course it's random, but it has a 00:44:04.230 --> 00:44:07.170 tendency to move in that direction. 00:44:07.170 --> 00:44:10.280 And if you think of this as a number of customers in queue, 00:44:10.280 --> 00:44:13.750 it means your system has the tendency to become loaded and 00:44:13.750 --> 00:44:15.420 to build up a queue. 00:44:15.420 --> 00:44:19.790 So rho being bigger than 1 corresponds to a heavy load, 00:44:19.790 --> 00:44:21.570 where queues build up. 00:44:21.570 --> 00:44:25.470 Rho less than 1 corresponds to the system where queues have 00:44:25.470 --> 00:44:27.310 the tendency to drain down. 00:44:30.880 --> 00:44:32.680 Now, let's write down the equations. 00:44:32.680 --> 00:44:40.540 We have this recursion P_(i+1) is Pi times Pi over Qi. 00:44:40.540 --> 00:44:44.040 In our case here, the P's and the Q's do not depend on the 00:44:44.040 --> 00:44:47.190 particular index, so we get this relation. 00:44:47.190 --> 00:44:51.830 And this P over Q is just the load factor rho. 00:44:51.830 --> 00:44:54.790 Once you look at this equation, clearly you realize 00:44:54.790 --> 00:44:58.440 that by pi(1) is rho times pi(0). 00:44:58.440 --> 00:45:02.286 pi(2) is going to be -- 00:45:02.286 --> 00:45:04.480 So we'll do it in detail. 00:45:04.480 --> 00:45:08.130 So pi(1) is pi(0) times rho. 00:45:08.130 --> 00:45:15.580 pi(2) is pi(1) times rho, which is pi(0) times 00:45:15.580 --> 00:45:18.350 rho-squared. 00:45:18.350 --> 00:45:21.340 And then you continue doing this calculation. 00:45:21.340 --> 00:45:25.530 And you find that you can express every pi(i) in terms 00:45:25.530 --> 00:45:31.490 of pi(0) and you get this factor of rho^i. 00:45:31.490 --> 00:45:34.840 And then you use the last equation that we have -- that 00:45:34.840 --> 00:45:38.110 the sum of the probabilities has to be equal to 1. 00:45:38.110 --> 00:45:41.670 And that equation is going to tell us that the sum over all 00:45:41.670 --> 00:45:50.890 i's from 0 to m of pi(0) rho to the i is equal to 1. 00:45:50.890 --> 00:45:58.730 And therefore, pi(0) is 1 over (the sum over the rho to the i 00:45:58.730 --> 00:46:03.100 for i going from 0 to m). 00:46:03.100 --> 00:46:09.870 So now we found pi(0), and by plugging in this expression, 00:46:09.870 --> 00:46:12.680 we have the steady-state probabilities of all of the 00:46:12.680 --> 00:46:14.950 different states. 00:46:14.950 --> 00:46:18.990 Let's look at some special cases of this. 00:46:18.990 --> 00:46:24.390 Suppose that rho is equal to 1. 00:46:24.390 --> 00:46:30.410 If rho is equal to 1, then pi(i) is equal to pi(0). 00:46:30.410 --> 00:46:33.250 It means that all the steady-state 00:46:33.250 --> 00:46:35.840 probabilities are equal. 00:46:35.840 --> 00:46:39.380 It's means that every state is equally 00:46:39.380 --> 00:46:42.750 likely in the long run. 00:46:42.750 --> 00:46:44.790 So this is an example. 00:46:44.790 --> 00:46:48.730 It's called a symmetric random walk. 00:46:48.730 --> 00:46:53.260 It's a very popular model for modeling people who are drunk. 00:46:53.260 --> 00:46:56.730 So you start at a state at any point in time. 00:46:56.730 --> 00:47:00.140 Either you stay in place, or you have an equal probability 00:47:00.140 --> 00:47:02.910 of going left or going right. 00:47:02.910 --> 00:47:06.220 There's no bias in either direction. 00:47:06.220 --> 00:47:10.710 You might think that in such a process, you will tend to kind 00:47:10.710 --> 00:47:14.750 of get stuck near one end or the other end. 00:47:14.750 --> 00:47:17.320 Well, it's not really clear what to expect. 00:47:17.320 --> 00:47:21.260 It turns out that in such a model, in the long run, the 00:47:21.260 --> 00:47:24.610 drunk person is equally likely to be at any 00:47:24.610 --> 00:47:26.430 one of those states. 00:47:26.430 --> 00:47:31.300 The steady-state probability is the same for all i's if rho 00:47:31.300 --> 00:47:33.880 is equal to 1. 00:47:33.880 --> 00:47:39.570 And so if you show up at a random time, and you ask where 00:47:39.570 --> 00:47:43.670 is my state, you will be told it's equally likely to be at 00:47:43.670 --> 00:47:46.600 any one of those places. 00:47:46.600 --> 00:47:48.370 So let's make that note. 00:47:48.370 --> 00:47:51.980 If rho equal to 1, implies that all the 00:47:51.980 --> 00:47:53.660 pi(i)'s are 1/(M+1) -- 00:47:57.040 --> 00:48:01.320 M+1 because that's how many states we have in our model. 00:48:01.320 --> 00:48:04.210 Now, let's look at a different case. 00:48:04.210 --> 00:48:08.630 Suppose that M is a huge number. 00:48:08.630 --> 00:48:13.600 So essentially, our supermarket has a very large 00:48:13.600 --> 00:48:19.600 space, a lot of space to store their customers. 00:48:19.600 --> 00:48:24.130 But suppose that the system is on the stable side. 00:48:24.130 --> 00:48:27.900 P is less than Q, which means that there's a tendency for 00:48:27.900 --> 00:48:31.420 customers to be served faster than they arrive. 00:48:31.420 --> 00:48:35.500 The drift in this chain, it tends to be in that direction. 00:48:35.500 --> 00:48:41.920 So when rho is less than 1, which is this case, and when M 00:48:41.920 --> 00:48:45.810 is going to infinity, this infinite sum is the sum of a 00:48:45.810 --> 00:48:47.690 geometric series. 00:48:47.690 --> 00:48:50.480 And you recognize it (hopefully) -- 00:48:53.480 --> 00:48:56.520 this series is going to 1/(1-rho). 00:48:56.520 --> 00:49:00.280 And because it's in the denominator, pi(0) ends up 00:49:00.280 --> 00:49:02.970 being 1-rho. 00:49:02.970 --> 00:49:06.230 So by taking the limit as M goes to infinity, in this 00:49:06.230 --> 00:49:09.460 case, and when rho is less than 1 so that this series is 00:49:09.460 --> 00:49:12.220 convergent, we get this formula. 00:49:12.220 --> 00:49:15.780 So we get the closed-form formula for the pi(i)'s. 00:49:15.780 --> 00:49:19.840 In particular, pi(i) is (1- rho)(rho to the i). 00:49:19.840 --> 00:49:21.150 to 00:49:21.150 --> 00:49:24.520 So these pi(i)'s are essentially a probability 00:49:24.520 --> 00:49:26.200 distribution. 00:49:26.200 --> 00:49:32.100 They tell us if we show up at time 1 billion and we ask, 00:49:32.100 --> 00:49:33.900 where is my state? 00:49:33.900 --> 00:49:37.500 You will be told that the state is 0. 00:49:37.500 --> 00:49:41.040 Your system is empty with probability 1-rho, minus or 00:49:41.040 --> 00:49:44.410 there's one customer in the system, and that happens with 00:49:44.410 --> 00:49:46.460 probability (rho - 1) times rho. 00:49:46.460 --> 00:49:49.950 And it keeps going down this way. 00:49:49.950 --> 00:49:53.920 And it's pretty much a geometric distribution except 00:49:53.920 --> 00:49:58.130 that it has shifted so that it starts at 0 whereas the usual 00:49:58.130 --> 00:50:00.890 geometric distribution starts at 1. 00:50:00.890 --> 00:50:04.670 So this is a mini introduction into queuing theory. 00:50:04.670 --> 00:50:08.730 This is the first and simplest model that one encounters when 00:50:08.730 --> 00:50:10.850 you start studying queuing theory. 00:50:10.850 --> 00:50:13.360 This is clearly a model of a queueing phenomenon such as 00:50:13.360 --> 00:50:16.450 the supermarket counter with the P's corresponding to 00:50:16.450 --> 00:50:19.280 arrivals, the Q's corresponding to departures. 00:50:19.280 --> 00:50:22.690 And this particular queuing system when M is very, very 00:50:22.690 --> 00:50:26.670 large and rho is less than 1, has a very simple and nice 00:50:26.670 --> 00:50:28.880 solution in closed form. 00:50:28.880 --> 00:50:31.650 And that's why it's very much liked. 00:50:31.650 --> 00:50:33.610 And let me just take two seconds to 00:50:33.610 --> 00:50:38.030 draw one last picture. 00:50:38.030 --> 00:50:40.300 So this is the probability of the different i's. 00:50:40.300 --> 00:50:41.420 It gives you a PMF. 00:50:41.420 --> 00:50:43.890 This PMF has an expected value. 00:50:43.890 --> 00:50:47.090 And the expectation, the expected number of customers 00:50:47.090 --> 00:50:50.560 in the system, is given by this formula. 00:50:50.560 --> 00:50:54.230 And this formula, which is interesting to anyone who 00:50:54.230 --> 00:50:57.070 tries to analyze a system of this kind, 00:50:57.070 --> 00:50:58.430 tells you the following. 00:50:58.430 --> 00:51:03.970 That as long as a rho is less than 1, then the expected 00:51:03.970 --> 00:51:07.270 number of customers in the system is finite. 00:51:07.270 --> 00:51:09.970 But if rho becomes very close to 1 -- 00:51:09.970 --> 00:51:13.790 So if your load factor is something like .99, you expect 00:51:13.790 --> 00:51:17.630 to have a large number of customers in the system at any 00:51:17.630 --> 00:51:19.040 given time. 00:51:19.040 --> 00:51:20.440 OK. 00:51:20.440 --> 00:51:22.420 All right. 00:51:22.420 --> 00:51:23.270 Have a good weekend. 00:51:23.270 --> 00:51:25.390 We'll continue next time.