WEBVTT 00:00:00.000 --> 00:00:00.040 align:middle line:90% 00:00:00.040 --> 00:00:02.460 align:middle line:84% The following content is provided under a Creative 00:00:02.460 --> 00:00:03.870 align:middle line:90% Commons license. 00:00:03.870 --> 00:00:06.910 align:middle line:84% Your support will help MIT OpenCourseWare continue to 00:00:06.910 --> 00:00:08.700 align:middle line:90% offer high-quality, educational 00:00:08.700 --> 00:00:10.560 align:middle line:90% resources for free. 00:00:10.560 --> 00:00:13.460 align:middle line:84% To make a donation or view additional materials from 00:00:13.460 --> 00:00:19.290 align:middle line:84% hundreds of MIT courses, visit MIT OpenCourseWare at 00:00:19.290 --> 00:00:21.764 align:middle line:90% ocw.mit.edu. 00:00:21.764 --> 00:00:22.590 align:middle line:90% PROFESSOR: All right. 00:00:22.590 --> 00:00:26.070 align:middle line:84% So today, we're going to start by taking stock of what we 00:00:26.070 --> 00:00:27.670 align:middle line:90% discussed last time, review the 00:00:27.670 --> 00:00:29.810 align:middle line:90% definition of Markov chains. 00:00:29.810 --> 00:00:32.400 align:middle line:84% And then most of the lecture, we're going to concentrate on 00:00:32.400 --> 00:00:34.520 align:middle line:90% their steady-state behavior. 00:00:34.520 --> 00:00:38.860 align:middle line:84% Meaning, we're going to look at what does a Markov chain do 00:00:38.860 --> 00:00:40.820 align:middle line:90% if it has run for a long time. 00:00:40.820 --> 00:00:42.860 align:middle line:84% What can we say about the probabilities of 00:00:42.860 --> 00:00:44.610 align:middle line:90% the different states? 00:00:44.610 --> 00:00:47.390 align:middle line:84% So what I would like to repeat is a statement I made last 00:00:47.390 --> 00:00:49.730 align:middle line:84% time that Markov chains is a very, very 00:00:49.730 --> 00:00:51.910 align:middle line:90% useful class of models. 00:00:51.910 --> 00:00:56.820 align:middle line:84% Pretty much anything in the real world can be 00:00:56.820 --> 00:01:01.380 align:middle line:84% approximately modeled by a Markov chain provided that you 00:01:01.380 --> 00:01:05.019 align:middle line:84% set your states in the proper way. 00:01:05.019 --> 00:01:06.890 align:middle line:84% So we're going to see some examples. 00:01:06.890 --> 00:01:09.310 align:middle line:84% You're going to see more examples in the problems 00:01:09.310 --> 00:01:11.140 align:middle line:84% you're going to do in homework and recitation. 00:01:11.140 --> 00:01:13.880 align:middle line:90% 00:01:13.880 --> 00:01:15.810 align:middle line:84% On the other hand, we're not going to go 00:01:15.810 --> 00:01:17.830 align:middle line:90% too deep into examples. 00:01:17.830 --> 00:01:20.310 align:middle line:84% Rather, we're going to develop the general methodology. 00:01:20.310 --> 00:02:11.440 align:middle line:90% 00:02:11.440 --> 00:02:12.690 align:middle line:90% OK. 00:02:12.690 --> 00:02:19.310 align:middle line:90% 00:02:19.310 --> 00:02:19.730 align:middle line:90% All right. 00:02:19.730 --> 00:02:22.280 align:middle line:84% Markov models can be pretty general. 00:02:22.280 --> 00:02:24.670 align:middle line:84% They can run in continuous or discrete time. 00:02:24.670 --> 00:02:27.540 align:middle line:84% They can have continuous or discrete state spaces. 00:02:27.540 --> 00:02:30.280 align:middle line:84% In this class, we're going to stick just to the case where 00:02:30.280 --> 00:02:33.880 align:middle line:84% the state space is discrete and time is discrete because 00:02:33.880 --> 00:02:35.570 align:middle line:90% this is the simplest case. 00:02:35.570 --> 00:02:38.760 align:middle line:84% And also, it's the one where you build your intuition 00:02:38.760 --> 00:02:40.830 align:middle line:84% before going to more general cases 00:02:40.830 --> 00:02:42.840 align:middle line:90% perhaps in other classes. 00:02:42.840 --> 00:02:47.810 align:middle line:84% So the state is discrete and finite. 00:02:47.810 --> 00:02:50.130 align:middle line:84% There's a finite number of states. 00:02:50.130 --> 00:02:53.540 align:middle line:84% At any point in time, the process is sitting on one of 00:02:53.540 --> 00:02:54.580 align:middle line:90% those states. 00:02:54.580 --> 00:03:00.520 align:middle line:84% Time is discrete, so at each unit of time, somebody 00:03:00.520 --> 00:03:03.000 align:middle line:84% whistles and then the state jumps. 00:03:03.000 --> 00:03:06.110 align:middle line:84% And when it jumps, it can either land in the same place, 00:03:06.110 --> 00:03:08.180 align:middle line:90% or it can land somewhere else. 00:03:08.180 --> 00:03:10.740 align:middle line:84% And the evolution of the process is described by 00:03:10.740 --> 00:03:13.130 align:middle line:90% transition probabilities. 00:03:13.130 --> 00:03:16.740 align:middle line:84% Pij is the probability that the next state is j given that 00:03:16.740 --> 00:03:18.630 align:middle line:90% the current state is i. 00:03:18.630 --> 00:03:21.640 align:middle line:84% And the most important property that the Markov chain 00:03:21.640 --> 00:03:24.690 align:middle line:84% has, the definition of a Markov chain or Markov 00:03:24.690 --> 00:03:28.880 align:middle line:84% process, is that this probability, Pij, is the same 00:03:28.880 --> 00:03:31.490 align:middle line:84% every time that you land at state i -- 00:03:31.490 --> 00:03:36.320 align:middle line:84% no matter how you got there and also no matter 00:03:36.320 --> 00:03:38.000 align:middle line:90% what time it is. 00:03:38.000 --> 00:03:41.650 align:middle line:84% So the model we have is time homogeneous, which basically 00:03:41.650 --> 00:03:44.630 align:middle line:84% means that those transition probabilities are the same at 00:03:44.630 --> 00:03:45.720 align:middle line:90% every time. 00:03:45.720 --> 00:03:49.320 align:middle line:84% So the model is time invariant in that sense. 00:03:49.320 --> 00:03:52.670 align:middle line:84% So we're interested in what the chain or the process is 00:03:52.670 --> 00:03:54.780 align:middle line:90% going to do in the longer run. 00:03:54.780 --> 00:03:57.060 align:middle line:84% So we're interested, let's say, in the probability that 00:03:57.060 --> 00:04:00.760 align:middle line:84% starting at a certain state, n times steps later, we find 00:04:00.760 --> 00:04:03.760 align:middle line:84% ourselves at some particular state j. 00:04:03.760 --> 00:04:06.350 align:middle line:84% Fortunately, we can calculate those probabilities 00:04:06.350 --> 00:04:07.660 align:middle line:90% recursively. 00:04:07.660 --> 00:04:12.990 align:middle line:84% Of course, at the first time 1, the probability of being 1 00:04:12.990 --> 00:04:16.950 align:middle line:84% time later at state j given that we are right now at state 00:04:16.950 --> 00:04:20.910 align:middle line:84% i, by definition, this is just the transition probabilities. 00:04:20.910 --> 00:04:26.000 align:middle line:84% So by knowing these, we can start a recursion that tells 00:04:26.000 --> 00:04:27.730 align:middle line:90% us the transition probabilities 00:04:27.730 --> 00:04:31.430 align:middle line:90% for more than n steps. 00:04:31.430 --> 00:04:32.865 align:middle line:84% This recursion, it's a formula. 00:04:32.865 --> 00:04:34.000 align:middle line:90% It's always true. 00:04:34.000 --> 00:04:36.860 align:middle line:84% You can copy it or memorize it. 00:04:36.860 --> 00:04:40.460 align:middle line:84% But there is a big idea behind that formula that you should 00:04:40.460 --> 00:04:41.480 align:middle line:90% keep in mind. 00:04:41.480 --> 00:04:43.750 align:middle line:84% And basically, the divide and conquer idea. 00:04:43.750 --> 00:04:47.530 align:middle line:84% It's an application of the total probability law. 00:04:47.530 --> 00:04:49.080 align:middle line:90% So let's fix i. 00:04:49.080 --> 00:04:52.360 align:middle line:84% The probability that you find yourself at state j, you break 00:04:52.360 --> 00:04:55.770 align:middle line:84% it up into the probabilities of the different ways that you 00:04:55.770 --> 00:04:57.850 align:middle line:90% can get to state j. 00:04:57.850 --> 00:04:59.240 align:middle line:90% What are those different ways? 00:04:59.240 --> 00:05:02.460 align:middle line:84% The different ways are the different states k at which 00:05:02.460 --> 00:05:05.220 align:middle line:84% you might find yourself the previous time. 00:05:05.220 --> 00:05:08.400 align:middle line:84% So with some probability, with this probability, you find 00:05:08.400 --> 00:05:11.070 align:middle line:84% yourself at state k the previous time. 00:05:11.070 --> 00:05:13.350 align:middle line:84% And then with probability Pkj, you make a 00:05:13.350 --> 00:05:15.030 align:middle line:90% transition to state j. 00:05:15.030 --> 00:05:19.000 align:middle line:84% So this is a possible scenario that takes you to state j 00:05:19.000 --> 00:05:21.090 align:middle line:90% after n transitions. 00:05:21.090 --> 00:05:25.350 align:middle line:84% And by summing over all the k's, then we have considered 00:05:25.350 --> 00:05:28.430 align:middle line:90% all the possible scenarios. 00:05:28.430 --> 00:05:31.590 align:middle line:84% Now, before we move to the more serious stuff, let's do a 00:05:31.590 --> 00:05:38.420 align:middle line:84% little bit of warm up to get a handle on how we use 00:05:38.420 --> 00:05:42.170 align:middle line:84% transition probabilities to calculate more general 00:05:42.170 --> 00:05:45.250 align:middle line:84% probabilities, then talk about some structural properties of 00:05:45.250 --> 00:05:47.980 align:middle line:84% Markov chains, and then eventually get to the main 00:05:47.980 --> 00:05:51.310 align:middle line:84% business of today, which is a steady-state behavior. 00:05:51.310 --> 00:05:56.380 align:middle line:84% So somebody gives you this chain, and our convention is 00:05:56.380 --> 00:06:00.340 align:middle line:84% that those arcs that are not shown here corresponds to 0 00:06:00.340 --> 00:06:01.510 align:middle line:90% probabilities. 00:06:01.510 --> 00:06:05.760 align:middle line:84% And each one of the arcs that's shown has a non-zero 00:06:05.760 --> 00:06:09.060 align:middle line:84% probability, and somebody gives it to us. 00:06:09.060 --> 00:06:11.650 align:middle line:84% Suppose that the chain starts at state 1. 00:06:11.650 --> 00:06:15.720 align:middle line:84% We want to calculate the probability that it follows 00:06:15.720 --> 00:06:16.830 align:middle line:90% this particular path. 00:06:16.830 --> 00:06:20.620 align:middle line:84% That is, it goes to 2, then to 6, then to 7. 00:06:20.620 --> 00:06:22.920 align:middle line:84% How do we calculate the probability of a particular 00:06:22.920 --> 00:06:24.280 align:middle line:90% trajectory? 00:06:24.280 --> 00:06:26.700 align:middle line:84% Well, this is the probability-- 00:06:26.700 --> 00:06:30.980 align:middle line:84% so it's the probability of the trajectory from 1 that you go 00:06:30.980 --> 00:06:34.480 align:middle line:90% to 2, then to 6, then to 7. 00:06:34.480 --> 00:06:38.020 align:middle line:84% So the probability of this trajectory is we use the 00:06:38.020 --> 00:06:39.270 align:middle line:90% multiplication rule. 00:06:39.270 --> 00:06:42.140 align:middle line:84% The probability of several things happening is the 00:06:42.140 --> 00:06:44.700 align:middle line:84% probability that the first thing happens, which is a 00:06:44.700 --> 00:06:47.140 align:middle line:90% transition from 1 to 2. 00:06:47.140 --> 00:06:53.190 align:middle line:84% And then given that we are at state 2, we multiply with a 00:06:53.190 --> 00:06:57.120 align:middle line:84% conditional probability that the next event happens. 00:06:57.120 --> 00:07:02.370 align:middle line:84% That is, that X2 is equal to 6 given that right now, we are 00:07:02.370 --> 00:07:03.570 align:middle line:90% at state 1. 00:07:03.570 --> 00:07:07.760 align:middle line:84% And that conditional probability is just P26. 00:07:07.760 --> 00:07:09.840 align:middle line:84% And notice that this conditional probability 00:07:09.840 --> 00:07:13.380 align:middle line:84% applies no matter how we got to state 2. 00:07:13.380 --> 00:07:15.340 align:middle line:90% This is the Markov assumption. 00:07:15.340 --> 00:07:17.970 align:middle line:84% So we don't care about the fact that we came in in a 00:07:17.970 --> 00:07:19.120 align:middle line:90% particular way. 00:07:19.120 --> 00:07:22.370 align:middle line:84% Given that we came in here, this probability P26, that the 00:07:22.370 --> 00:07:24.640 align:middle line:90% next transition takes us to 6. 00:07:24.640 --> 00:07:28.620 align:middle line:84% And then given that all that stuff happened, so given that 00:07:28.620 --> 00:07:32.060 align:middle line:84% right now, we are at state 6, we need to multiply with a 00:07:32.060 --> 00:07:34.680 align:middle line:84% conditional probability that the next transition takes us 00:07:34.680 --> 00:07:36.020 align:middle line:90% to state 7. 00:07:36.020 --> 00:07:39.920 align:middle line:90% And this is just the P67. 00:07:39.920 --> 00:07:44.140 align:middle line:84% So to find the probability of following a specific 00:07:44.140 --> 00:07:49.280 align:middle line:84% trajectory, you just multiply the transition probabilities 00:07:49.280 --> 00:07:50.885 align:middle line:84% along the particular trajectory. 00:07:50.885 --> 00:07:54.490 align:middle line:90% 00:07:54.490 --> 00:07:58.150 align:middle line:84% Now, if you want to calculate something else, such as for 00:07:58.150 --> 00:08:02.100 align:middle line:84% example, the probability that 4 time steps later, I find 00:08:02.100 --> 00:08:06.250 align:middle line:84% myself at state 7 given that they started, let's say, at 00:08:06.250 --> 00:08:07.680 align:middle line:90% this state. 00:08:07.680 --> 00:08:09.920 align:middle line:84% How do you calculate this probability? 00:08:09.920 --> 00:08:15.350 align:middle line:84% One way is to use the recursion for the Rijs that we 00:08:15.350 --> 00:08:17.990 align:middle line:90% know that it is always valid. 00:08:17.990 --> 00:08:20.970 align:middle line:84% But for short and simple examples, and with a small 00:08:20.970 --> 00:08:24.890 align:middle line:84% time horizon, perhaps you can do this in a brute force way. 00:08:24.890 --> 00:08:27.100 align:middle line:84% What would be the brute force way? 00:08:27.100 --> 00:08:30.410 align:middle line:84% This is the event that 4 time steps later, I find 00:08:30.410 --> 00:08:32.210 align:middle line:90% myself at state 7. 00:08:32.210 --> 00:08:36.350 align:middle line:84% This event can happen in various ways. 00:08:36.350 --> 00:08:41.070 align:middle line:84% So we can take stock of all the different ways, and write 00:08:41.070 --> 00:08:42.760 align:middle line:90% down their probabilities. 00:08:42.760 --> 00:08:44.800 align:middle line:90% So starting from 2. 00:08:44.800 --> 00:08:49.870 align:middle line:84% One possibility is to follow this trajectory, 1 transition, 00:08:49.870 --> 00:08:53.790 align:middle line:84% 2 transitions, 3 transitions, 4 transitions. 00:08:53.790 --> 00:08:55.810 align:middle line:90% And that takes me to state 7. 00:08:55.810 --> 00:08:57.760 align:middle line:84% What's the probability of this trajectory? 00:08:57.760 --> 00:09:05.160 align:middle line:84% It's P26 times P67 times P76 and then times P67. 00:09:05.160 --> 00:09:08.000 align:middle line:84% So this is a probability of a particular trajectory that 00:09:08.000 --> 00:09:11.040 align:middle line:84% takes you to state 7 after 4 time steps. 00:09:11.040 --> 00:09:14.790 align:middle line:84% But there's other trajectories as well. 00:09:14.790 --> 00:09:16.300 align:middle line:90% What could be it? 00:09:16.300 --> 00:09:23.470 align:middle line:84% I might start from state 2, go to state 6, stay at state 6, 00:09:23.470 --> 00:09:26.450 align:middle line:90% stay at state 6 once more. 00:09:26.450 --> 00:09:31.670 align:middle line:84% And then from state 6, go to state 7. 00:09:31.670 --> 00:09:36.800 align:middle line:90% And so there must be one more. 00:09:36.800 --> 00:09:38.470 align:middle line:90% What's the other one? 00:09:38.470 --> 00:09:45.620 align:middle line:90% I guess I could go 1, 2, 6, 7. 00:09:45.620 --> 00:09:45.690 align:middle line:90% OK. 00:09:45.690 --> 00:09:48.012 align:middle line:90% That's the other trajectory. 00:09:48.012 --> 00:10:01.850 align:middle line:84% Plus P21 times P12 times P26 and times P67. 00:10:01.850 --> 00:10:05.580 align:middle line:84% So the transition probability, the overall probability of 00:10:05.580 --> 00:10:09.490 align:middle line:84% finding ourselves at state 7, is broken down as the sum of 00:10:09.490 --> 00:10:12.580 align:middle line:84% the probabilities of all the different ways that I can get 00:10:12.580 --> 00:10:16.000 align:middle line:90% to state 7 in exactly 4 steps. 00:10:16.000 --> 00:10:19.460 align:middle line:84% So we could always do that without knowing much about 00:10:19.460 --> 00:10:21.750 align:middle line:84% Markov chains or the general formula for the 00:10:21.750 --> 00:10:24.490 align:middle line:90% Rij's that we had. 00:10:24.490 --> 00:10:26.370 align:middle line:84% What's the trouble with this procedure? 00:10:26.370 --> 00:10:29.190 align:middle line:84% The trouble with this procedure is that the number 00:10:29.190 --> 00:10:34.450 align:middle line:84% of possible trajectories becomes quite large if this 00:10:34.450 --> 00:10:37.550 align:middle line:90% index is a little bigger. 00:10:37.550 --> 00:10:41.510 align:middle line:84% If this 4 was 100, and you ask how many different 00:10:41.510 --> 00:10:45.240 align:middle line:84% trajectories of length 100 are there to take me from here to 00:10:45.240 --> 00:10:48.170 align:middle line:84% there, that number of trajectories would be huge. 00:10:48.170 --> 00:10:51.120 align:middle line:84% It grows exponentially with the time horizon. 00:10:51.120 --> 00:10:55.110 align:middle line:84% And this kind of calculation would be impossible. 00:10:55.110 --> 00:10:58.310 align:middle line:84% The basic equation, the recursion that have for the 00:10:58.310 --> 00:11:01.650 align:middle line:84% Rij's is basically a clever way of organizing this 00:11:01.650 --> 00:11:04.870 align:middle line:84% computation so that the amount of computation that you do is 00:11:04.870 --> 00:11:06.960 align:middle line:84% not exponential in the time horizon. 00:11:06.960 --> 00:11:11.100 align:middle line:84% Rather, it's sort of linear with the time horizon. 00:11:11.100 --> 00:11:14.590 align:middle line:84% For each time step you need in the time horizon, you just 00:11:14.590 --> 00:11:17.260 align:middle line:84% keep repeating the same iteration over and over. 00:11:17.260 --> 00:11:20.460 align:middle line:90% 00:11:20.460 --> 00:11:20.565 align:middle line:90% OK. 00:11:20.565 --> 00:11:24.940 align:middle line:84% Now, the other thing that we discussed last time, briefly, 00:11:24.940 --> 00:11:28.530 align:middle line:84% was a classification of the different states of the Markov 00:11:28.530 --> 00:11:31.510 align:middle line:90% chain in two different types. 00:11:31.510 --> 00:11:37.020 align:middle line:84% A Markov chain, in general, has states that are recurrent, 00:11:37.020 --> 00:11:39.670 align:middle line:84% which means that from a recurrent state, I can go 00:11:39.670 --> 00:11:41.000 align:middle line:90% somewhere else. 00:11:41.000 --> 00:11:44.910 align:middle line:84% But from that somewhere else, there's always some way of 00:11:44.910 --> 00:11:46.040 align:middle line:90% coming back. 00:11:46.040 --> 00:11:49.910 align:middle line:84% So if you have a chain of this form, no matter where you go, 00:11:49.910 --> 00:11:52.500 align:middle line:84% no matter where you start, you can always come 00:11:52.500 --> 00:11:54.540 align:middle line:90% back where you started. 00:11:54.540 --> 00:11:56.800 align:middle line:84% States of this kind are called recurrent. 00:11:56.800 --> 00:12:00.320 align:middle line:84% On the other hand, if you have a few states all this kind, a 00:12:00.320 --> 00:12:04.560 align:middle line:84% transition of this type, then these states are transient in 00:12:04.560 --> 00:12:07.560 align:middle line:84% the sense that from those states, it's possible to go 00:12:07.560 --> 00:12:11.520 align:middle line:84% somewhere else from which place there's no way to come 00:12:11.520 --> 00:12:13.580 align:middle line:90% back where you started. 00:12:13.580 --> 00:12:18.370 align:middle line:84% The general structure of a Markov chain is basically a 00:12:18.370 --> 00:12:20.660 align:middle line:84% collection of transient states. 00:12:20.660 --> 00:12:23.960 align:middle line:84% You're certain that you are going to leave the transient 00:12:23.960 --> 00:12:27.370 align:middle line:90% states eventually. 00:12:27.370 --> 00:12:30.300 align:middle line:84% And after you leave the transient states, you enter 00:12:30.300 --> 00:12:33.570 align:middle line:84% into a class of states in which you are trapped. 00:12:33.570 --> 00:12:35.890 align:middle line:84% You are trapped if you get inside here. 00:12:35.890 --> 00:12:39.040 align:middle line:84% You are trapped if you get inside there. 00:12:39.040 --> 00:12:41.270 align:middle line:84% This is a recurrent class of states. 00:12:41.270 --> 00:12:43.210 align:middle line:84% From any state, you can get to any other 00:12:43.210 --> 00:12:44.380 align:middle line:90% state within this class. 00:12:44.380 --> 00:12:46.100 align:middle line:84% That's another recurrent class. 00:12:46.100 --> 00:12:49.380 align:middle line:84% From any state inside here, you can get anywhere else 00:12:49.380 --> 00:12:50.680 align:middle line:90% inside that class. 00:12:50.680 --> 00:12:52.930 align:middle line:84% But these 2 classes, you do not communicate. 00:12:52.930 --> 00:12:56.310 align:middle line:84% If you start here, there's no way to get there. 00:12:56.310 --> 00:12:59.820 align:middle line:84% If you have 2 recurrent classes, then it's clear that 00:12:59.820 --> 00:13:02.310 align:middle line:84% the initial conditions of your Markov chain 00:13:02.310 --> 00:13:04.130 align:middle line:90% matter in the long run. 00:13:04.130 --> 00:13:07.450 align:middle line:84% If you start here, you will be stuck inside here for the long 00:13:07.450 --> 00:13:09.140 align:middle line:90% run and similarly about here. 00:13:09.140 --> 00:13:11.680 align:middle line:84% So the initial conditions do make a difference. 00:13:11.680 --> 00:13:14.720 align:middle line:84% On the other hand, if this class was not here and you 00:13:14.720 --> 00:13:17.300 align:middle line:84% only had that class, what would happen to the chain? 00:13:17.300 --> 00:13:18.480 align:middle line:90% Let's say you start here. 00:13:18.480 --> 00:13:19.490 align:middle line:90% You move around. 00:13:19.490 --> 00:13:21.730 align:middle line:84% At some point, you make that transition. 00:13:21.730 --> 00:13:23.260 align:middle line:90% You get stuck in here. 00:13:23.260 --> 00:13:26.410 align:middle line:84% And inside here, you keep circulating, because of the 00:13:26.410 --> 00:13:30.210 align:middle line:84% randomness, you keep visiting all states over and over. 00:13:30.210 --> 00:13:35.800 align:middle line:84% And hopefully or possibly, in the long run, it doesn't 00:13:35.800 --> 00:13:39.610 align:middle line:84% matter exactly what time it is or where you started, but the 00:13:39.610 --> 00:13:43.860 align:middle line:84% probability of being at that particular state is the same 00:13:43.860 --> 00:13:46.310 align:middle line:84% no matter what the initial condition was. 00:13:46.310 --> 00:13:48.820 align:middle line:84% So with a single recurrent class, we hope that the 00:13:48.820 --> 00:13:50.750 align:middle line:84% initial conditions do not matter. 00:13:50.750 --> 00:13:55.780 align:middle line:84% With 2 or more recurrent classes, initial conditions 00:13:55.780 --> 00:13:58.660 align:middle line:90% will definitely matter. 00:13:58.660 --> 00:14:03.620 align:middle line:84% So how many recurrent classes we have is something that has 00:14:03.620 --> 00:14:06.790 align:middle line:84% to do with the long-term behavior of the chain and the 00:14:06.790 --> 00:14:09.790 align:middle line:84% extent to which initial conditions matter. 00:14:09.790 --> 00:14:16.000 align:middle line:84% Another way that initial conditions may matter is if a 00:14:16.000 --> 00:14:19.360 align:middle line:84% chain has a periodic structure. 00:14:19.360 --> 00:14:21.990 align:middle line:84% There are many ways of defining periodicity. 00:14:21.990 --> 00:14:25.240 align:middle line:84% The one that I find sort of the most intuitive and with 00:14:25.240 --> 00:14:27.410 align:middle line:84% the least amount of mathematical 00:14:27.410 --> 00:14:29.670 align:middle line:90% symbols is the following. 00:14:29.670 --> 00:14:34.630 align:middle line:84% The state space of a chain is said to be periodic if you can 00:14:34.630 --> 00:14:39.510 align:middle line:84% lump the states into a number of clusters called 00:14:39.510 --> 00:14:42.550 align:middle line:90% d clusters or groups. 00:14:42.550 --> 00:14:45.980 align:middle line:84% And the transition diagram has the property that from a 00:14:45.980 --> 00:14:48.860 align:middle line:84% cluster, you always make a transition 00:14:48.860 --> 00:14:50.870 align:middle line:90% into the next cluster. 00:14:50.870 --> 00:14:52.860 align:middle line:90% So here d is equal to 2. 00:14:52.860 --> 00:14:55.570 align:middle line:84% We have two subsets of the state space. 00:14:55.570 --> 00:14:58.130 align:middle line:84% Whenever we're here, next time we'll be there. 00:14:58.130 --> 00:15:01.080 align:middle line:84% Whenever we're here, next time we will be there. 00:15:01.080 --> 00:15:03.830 align:middle line:84% So this chain has a periodic structure. 00:15:03.830 --> 00:15:05.880 align:middle line:84% There may be still some randomness. 00:15:05.880 --> 00:15:10.270 align:middle line:84% When I jump from here to here, the state to which I jump may 00:15:10.270 --> 00:15:14.490 align:middle line:84% be random, but I'm sure that I'm going to be inside here. 00:15:14.490 --> 00:15:17.610 align:middle line:84% And then next time, I will be sure that I'm inside here. 00:15:17.610 --> 00:15:20.310 align:middle line:84% This would be a structure of a diagram in which we have a 00:15:20.310 --> 00:15:21.410 align:middle line:90% period of 3. 00:15:21.410 --> 00:15:25.540 align:middle line:84% If you start in this lump, you know that the next time, you 00:15:25.540 --> 00:15:27.480 align:middle line:84% would be in a state inside here. 00:15:27.480 --> 00:15:30.830 align:middle line:84% Next time, you'll be in a state inside here, and so on. 00:15:30.830 --> 00:15:35.220 align:middle line:84% So these chains certainly have a periodic structure. 00:15:35.220 --> 00:15:37.860 align:middle line:84% And that periodicity gets maintained. 00:15:37.860 --> 00:15:42.500 align:middle line:84% If I start, let's say, at this lump, at even times, 00:15:42.500 --> 00:15:44.500 align:middle line:90% I'm sure I'm here. 00:15:44.500 --> 00:15:47.660 align:middle line:84% At odd times, I'm sure I am here. 00:15:47.660 --> 00:15:51.600 align:middle line:84% So the exact time does matter in determining the 00:15:51.600 --> 00:15:54.660 align:middle line:84% probabilities of the different states. 00:15:54.660 --> 00:15:57.140 align:middle line:84% And in particular, the probability of being at the 00:15:57.140 --> 00:16:00.500 align:middle line:84% particular state cannot convert to a state value. 00:16:00.500 --> 00:16:03.410 align:middle line:84% The probability of being at the state inside here is going 00:16:03.410 --> 00:16:06.300 align:middle line:90% to be 0 for all times. 00:16:06.300 --> 00:16:08.290 align:middle line:84% In general, it's going to be some positive 00:16:08.290 --> 00:16:10.160 align:middle line:90% number for even times. 00:16:10.160 --> 00:16:13.810 align:middle line:84% So it goes 0 positive, zero, positive, 0 positive. 00:16:13.810 --> 00:16:15.370 align:middle line:90% Doesn't settle to anything. 00:16:15.370 --> 00:16:19.860 align:middle line:84% So when we have periodicity, we do not expect the states 00:16:19.860 --> 00:16:22.600 align:middle line:84% probabilities to converge to something, but rather, we 00:16:22.600 --> 00:16:24.580 align:middle line:90% expect them to oscillate. 00:16:24.580 --> 00:16:26.830 align:middle line:84% Now, how can we tell whether a Markov chain 00:16:26.830 --> 00:16:29.920 align:middle line:90% is periodic or not? 00:16:29.920 --> 00:16:33.900 align:middle line:84% There are systematic ways of doing it, but usually with the 00:16:33.900 --> 00:16:36.560 align:middle line:84% types of examples we see in this class, we just eyeball 00:16:36.560 --> 00:16:39.890 align:middle line:84% the chain, and we tell whether it's periodic or not. 00:16:39.890 --> 00:16:45.240 align:middle line:84% So is this chain down here, is it the periodic one or not? 00:16:45.240 --> 00:16:48.870 align:middle line:84% How many people think it's periodic? 00:16:48.870 --> 00:16:50.680 align:middle line:90% No one. 00:16:50.680 --> 00:16:51.200 align:middle line:90% One. 00:16:51.200 --> 00:16:54.070 align:middle line:84% How many people think it's not periodic? 00:16:54.070 --> 00:16:54.560 align:middle line:90% OK. 00:16:54.560 --> 00:16:56.270 align:middle line:90% Not periodic? 00:16:56.270 --> 00:16:57.570 align:middle line:90% Let's see. 00:16:57.570 --> 00:16:59.490 align:middle line:90% Let me do some drawing here. 00:16:59.490 --> 00:17:03.230 align:middle line:90% 00:17:03.230 --> 00:17:04.119 align:middle line:90% OK. 00:17:04.119 --> 00:17:05.369 align:middle line:90% Is it periodic? 00:17:05.369 --> 00:17:07.856 align:middle line:90% 00:17:07.856 --> 00:17:09.140 align:middle line:90% It is. 00:17:09.140 --> 00:17:14.180 align:middle line:84% From a red state, you can only get to a white state. 00:17:14.180 --> 00:17:17.849 align:middle line:84% And from a white state, you can only get to a red state. 00:17:17.849 --> 00:17:20.660 align:middle line:84% So this chain, even though it's not apparent from the 00:17:20.660 --> 00:17:24.589 align:middle line:84% picture, actually has this structure. 00:17:24.589 --> 00:17:28.600 align:middle line:84% We can group the states into red states and white states. 00:17:28.600 --> 00:17:32.810 align:middle line:84% And from reds, we always go to a white, and from a white, we 00:17:32.810 --> 00:17:34.500 align:middle line:90% always go to a red. 00:17:34.500 --> 00:17:36.540 align:middle line:84% So this tells you that sometimes 00:17:36.540 --> 00:17:38.680 align:middle line:90% eyeballing is not as easy. 00:17:38.680 --> 00:17:40.810 align:middle line:84% If you have lots and lots of states, you might have some 00:17:40.810 --> 00:17:43.280 align:middle line:90% trouble doing this exercise. 00:17:43.280 --> 00:17:47.230 align:middle line:84% On the other hand, something very useful to know. 00:17:47.230 --> 00:17:50.270 align:middle line:84% Sometimes it's extremely easy to tell that the 00:17:50.270 --> 00:17:52.360 align:middle line:90% chain is not periodic. 00:17:52.360 --> 00:17:53.770 align:middle line:90% What's that case? 00:17:53.770 --> 00:17:58.600 align:middle line:84% Suppose that your chain has a self-transition somewhere. 00:17:58.600 --> 00:18:01.670 align:middle line:84% Then automatically, you know that your 00:18:01.670 --> 00:18:04.660 align:middle line:90% chain is not periodic. 00:18:04.660 --> 00:18:08.560 align:middle line:84% So remember, the definition of periodicity requires that if 00:18:08.560 --> 00:18:12.230 align:middle line:84% you are in a certain group of states, next time, you will be 00:18:12.230 --> 00:18:14.380 align:middle line:90% in a different group. 00:18:14.380 --> 00:18:16.450 align:middle line:84% But if you have self-transitions, that 00:18:16.450 --> 00:18:17.600 align:middle line:90% property is not true. 00:18:17.600 --> 00:18:20.790 align:middle line:84% If you have a possible self-transition, it's possible 00:18:20.790 --> 00:18:24.530 align:middle line:84% that you stay inside your own group for the next time step. 00:18:24.530 --> 00:18:29.560 align:middle line:84% So whenever you have a self-transition, this implies 00:18:29.560 --> 00:18:31.440 align:middle line:84% that the chain is not periodic. 00:18:31.440 --> 00:18:34.280 align:middle line:90% 00:18:34.280 --> 00:18:39.240 align:middle line:84% And usually that's the simplest and easy way that we 00:18:39.240 --> 00:18:44.210 align:middle line:84% can tell most of the time that the chain is not periodic. 00:18:44.210 --> 00:18:49.110 align:middle line:84% So now, we come to the big topic of today, the central 00:18:49.110 --> 00:18:53.080 align:middle line:84% topic, which is the question about what does the chain do 00:18:53.080 --> 00:18:55.550 align:middle line:90% in the long run. 00:18:55.550 --> 00:19:00.060 align:middle line:84% The question we are asking and which we motivated last time 00:19:00.060 --> 00:19:02.510 align:middle line:90% by looking at an example. 00:19:02.510 --> 00:19:05.790 align:middle line:84% It's something that did happen in our example of last time. 00:19:05.790 --> 00:19:08.000 align:middle line:84% So we're asking whether this happens for 00:19:08.000 --> 00:19:09.440 align:middle line:90% every Markov chain. 00:19:09.440 --> 00:19:12.400 align:middle line:84% We're asking the question whether the probability of 00:19:12.400 --> 00:19:18.250 align:middle line:84% being at state j at some time n settles to a 00:19:18.250 --> 00:19:20.350 align:middle line:90% steady-state value. 00:19:20.350 --> 00:19:22.900 align:middle line:90% Let's call it pi sub j. 00:19:22.900 --> 00:19:26.960 align:middle line:84% That these were asking whether this quantity has a limit as n 00:19:26.960 --> 00:19:29.400 align:middle line:84% goes to infinity, so that we can talk about the 00:19:29.400 --> 00:19:32.300 align:middle line:84% steady-state probability of state j. 00:19:32.300 --> 00:19:36.060 align:middle line:84% And furthermore, we asked whether the steady-state 00:19:36.060 --> 00:19:38.900 align:middle line:84% probability of that state does not depend 00:19:38.900 --> 00:19:40.800 align:middle line:90% on the initial state. 00:19:40.800 --> 00:19:44.110 align:middle line:84% In other words, after the chain runs for a long, long 00:19:44.110 --> 00:19:48.620 align:middle line:84% time, it doesn't matter exactly what time it is, and 00:19:48.620 --> 00:19:51.990 align:middle line:84% it doesn't matter where the chain started from. 00:19:51.990 --> 00:19:54.860 align:middle line:84% You can tell me the probability that the state is 00:19:54.860 --> 00:19:58.700 align:middle line:84% a particular j is approximately the steady-state 00:19:58.700 --> 00:20:00.450 align:middle line:90% probability pi sub j. 00:20:00.450 --> 00:20:03.470 align:middle line:84% It doesn't matter exactly what time it is as long as you tell 00:20:03.470 --> 00:20:06.900 align:middle line:84% me that a lot of time has elapsed so 00:20:06.900 --> 00:20:09.520 align:middle line:90% that n is a big number. 00:20:09.520 --> 00:20:11.150 align:middle line:90% So this is the question. 00:20:11.150 --> 00:20:14.210 align:middle line:84% We have seen examples, and we understand that this is not 00:20:14.210 --> 00:20:16.600 align:middle line:90% going to be the case always. 00:20:16.600 --> 00:20:19.880 align:middle line:84% For example, as I just discussed, if we have 2 00:20:19.880 --> 00:20:23.850 align:middle line:84% recurrent classes, where we start does matter. 00:20:23.850 --> 00:20:28.060 align:middle line:84% The probability pi(j) of being in that state j is going to be 00:20:28.060 --> 00:20:32.650 align:middle line:84% 0 if we start here, but it would be something positive if 00:20:32.650 --> 00:20:34.710 align:middle line:90% we were to start in that lump. 00:20:34.710 --> 00:20:37.650 align:middle line:84% So the initial state does matter if we have multiple 00:20:37.650 --> 00:20:39.690 align:middle line:90% recurrent classes. 00:20:39.690 --> 00:20:45.590 align:middle line:84% But if we have only a single class of recurrent states from 00:20:45.590 --> 00:20:48.780 align:middle line:84% each one of which you can get to any other one, then we 00:20:48.780 --> 00:20:49.980 align:middle line:90% don't have that problem. 00:20:49.980 --> 00:20:53.010 align:middle line:84% Then we expect initial conditions to be forgotten. 00:20:53.010 --> 00:20:55.100 align:middle line:84% So that's one condition that we need. 00:20:55.100 --> 00:20:58.960 align:middle line:90% 00:20:58.960 --> 00:21:01.670 align:middle line:84% And then the other condition that we need is that the chain 00:21:01.670 --> 00:21:02.930 align:middle line:90% is not periodic. 00:21:02.930 --> 00:21:07.580 align:middle line:84% If the chain is periodic, then these Rij's do not converge. 00:21:07.580 --> 00:21:09.510 align:middle line:90% They keep oscillating. 00:21:09.510 --> 00:21:13.190 align:middle line:84% If we do not have periodicity, then there is hope that we 00:21:13.190 --> 00:21:16.070 align:middle line:84% will get the convergence that we need. 00:21:16.070 --> 00:21:19.210 align:middle line:84% It turns out this is the big theory of Markov chains-- the 00:21:19.210 --> 00:21:20.900 align:middle line:84% steady-state convergence theorem. 00:21:20.900 --> 00:21:26.470 align:middle line:84% It turns out that yes, the rijs do converge to a 00:21:26.470 --> 00:21:29.510 align:middle line:84% steady-state limit, which we call a steady-state 00:21:29.510 --> 00:21:35.180 align:middle line:84% probability as long as these two conditions are satisfied. 00:21:35.180 --> 00:21:37.500 align:middle line:84% We're not going to prove this theorem. 00:21:37.500 --> 00:21:41.790 align:middle line:84% If you're really interested, the end of chapter exercises 00:21:41.790 --> 00:21:45.670 align:middle line:84% basically walk you through a proof of this result, but it's 00:21:45.670 --> 00:21:49.470 align:middle line:84% probably a little too much for doing it in this class. 00:21:49.470 --> 00:21:52.140 align:middle line:84% What is the intuitive idea behind this theorem? 00:21:52.140 --> 00:21:52.860 align:middle line:90% Let's see. 00:21:52.860 --> 00:21:56.830 align:middle line:84% Let's think intuitively as to why the initial 00:21:56.830 --> 00:21:59.010 align:middle line:90% state doesn't matter. 00:21:59.010 --> 00:22:02.870 align:middle line:84% Think of two copies of the chain that starts at different 00:22:02.870 --> 00:22:06.430 align:middle line:84% initial states, and the state moves randomly. 00:22:06.430 --> 00:22:09.390 align:middle line:84% As the state moves randomly starting from the two initial 00:22:09.390 --> 00:22:11.900 align:middle line:90% states a random trajectory. 00:22:11.900 --> 00:22:15.610 align:middle line:84% as long as you have a single recurrent class at some point, 00:22:15.610 --> 00:22:19.170 align:middle line:84% and you don't have periodicity at some point, those states, 00:22:19.170 --> 00:22:22.610 align:middle line:84% those two trajectories, are going to collide. 00:22:22.610 --> 00:22:25.650 align:middle line:84% Just because there's enough randomness there. 00:22:25.650 --> 00:22:28.830 align:middle line:84% Even though we started from different places, the state is 00:22:28.830 --> 00:22:30.490 align:middle line:90% going to be the same. 00:22:30.490 --> 00:22:33.540 align:middle line:84% After the state becomes the same, then the future of these 00:22:33.540 --> 00:22:37.100 align:middle line:84% trajectories, probabilistically, is the same 00:22:37.100 --> 00:22:39.800 align:middle line:84% because they both started at the same state. 00:22:39.800 --> 00:22:42.540 align:middle line:84% So this means that the initial conditions 00:22:42.540 --> 00:22:45.330 align:middle line:90% stopped having any influence. 00:22:45.330 --> 00:22:50.040 align:middle line:84% That's sort of the high-level idea of why the initial state 00:22:50.040 --> 00:22:50.820 align:middle line:90% gets forgotten. 00:22:50.820 --> 00:22:53.780 align:middle line:84% Even if you started at different initial states, at 00:22:53.780 --> 00:22:57.030 align:middle line:84% some time, you may find yourself to be in the same 00:22:57.030 --> 00:22:59.120 align:middle line:90% state as the other trajectory. 00:22:59.120 --> 00:23:05.370 align:middle line:84% And once that happens, your initial conditions cannot have 00:23:05.370 --> 00:23:08.210 align:middle line:90% any effect into the future. 00:23:08.210 --> 00:23:09.230 align:middle line:90% All right. 00:23:09.230 --> 00:23:15.650 align:middle line:84% So let's see how we might calculate those steady-state 00:23:15.650 --> 00:23:16.870 align:middle line:90% probabilities. 00:23:16.870 --> 00:23:19.850 align:middle line:84% The way we calculate the steady-state probabilities is 00:23:19.850 --> 00:23:24.230 align:middle line:84% by taking this recursion, which is always true for the 00:23:24.230 --> 00:23:27.150 align:middle line:84% end-step transition probabilities, and take the 00:23:27.150 --> 00:23:29.400 align:middle line:90% limit of both sides. 00:23:29.400 --> 00:23:32.970 align:middle line:84% The limit of this side is the steady-state probability of 00:23:32.970 --> 00:23:36.010 align:middle line:90% state j, which is pi sub j. 00:23:36.010 --> 00:23:38.140 align:middle line:84% The limit of this side, we put the 00:23:38.140 --> 00:23:40.120 align:middle line:90% limit inside the summation. 00:23:40.120 --> 00:23:44.330 align:middle line:84% Now, as n goes to infinity, n - also goes to infinity. 00:23:44.330 --> 00:23:48.530 align:middle line:84% So this Rik is going to be the steady-state probability of 00:23:48.530 --> 00:23:51.150 align:middle line:90% state k starting from state i. 00:23:51.150 --> 00:23:53.170 align:middle line:84% Now where we started doesn't matter. 00:23:53.170 --> 00:23:54.620 align:middle line:84% So this is just the steady-state 00:23:54.620 --> 00:23:56.290 align:middle line:90% probability of state k. 00:23:56.290 --> 00:24:00.000 align:middle line:84% So this term converges to that one, and this gives us an 00:24:00.000 --> 00:24:03.010 align:middle line:84% equation that's satisfied by the steady-state 00:24:03.010 --> 00:24:04.030 align:middle line:90% probabilities. 00:24:04.030 --> 00:24:06.150 align:middle line:84% Actually, it's not one equation. 00:24:06.150 --> 00:24:10.580 align:middle line:84% We get one equation for each one of the j's. 00:24:10.580 --> 00:24:13.220 align:middle line:84% So if we have 10 possible states, we're going to get the 00:24:13.220 --> 00:24:15.580 align:middle line:90% system of 10 linear equations. 00:24:15.580 --> 00:24:18.840 align:middle line:84% In the unknowns, pi(1) up to pi(10). 00:24:18.840 --> 00:24:19.210 align:middle line:90% OK. 00:24:19.210 --> 00:24:20.800 align:middle line:90% 10 unknowns, 10 equations. 00:24:20.800 --> 00:24:23.150 align:middle line:84% You might think that we are in business. 00:24:23.150 --> 00:24:27.646 align:middle line:84% But actually, this system of equations is singular. 00:24:27.646 --> 00:24:30.180 align:middle line:84% 0 is a possible solution of this system. 00:24:30.180 --> 00:24:32.900 align:middle line:84% If you plug pi equal to zero everywhere, the 00:24:32.900 --> 00:24:33.900 align:middle line:90% equations are satisfied. 00:24:33.900 --> 00:24:37.550 align:middle line:84% It does not have a unique solution, so maybe we need one 00:24:37.550 --> 00:24:40.580 align:middle line:84% more condition to get the uniquely solvable system of 00:24:40.580 --> 00:24:41.900 align:middle line:90% linear equations. 00:24:41.900 --> 00:24:44.350 align:middle line:84% It turns out that this system of equations 00:24:44.350 --> 00:24:46.150 align:middle line:90% has a unique solution. 00:24:46.150 --> 00:24:48.980 align:middle line:84% If you impose an additional condition, which is pretty 00:24:48.980 --> 00:24:52.160 align:middle line:84% natural, the pi(j)'s are the probabilities of the different 00:24:52.160 --> 00:24:54.960 align:middle line:84% states, so they should add to 1. 00:24:54.960 --> 00:24:58.410 align:middle line:84% So you want this one equation to the mix. 00:24:58.410 --> 00:25:05.490 align:middle line:84% And once you do that, then this system of equations is 00:25:05.490 --> 00:25:07.340 align:middle line:84% going to have a unique solution. 00:25:07.340 --> 00:25:09.550 align:middle line:84% And so we can find the steady-state probabilities of 00:25:09.550 --> 00:25:12.980 align:middle line:84% the Markov chain by just solving these linear 00:25:12.980 --> 00:25:16.130 align:middle line:84% equations, which is numerically straightforward. 00:25:16.130 --> 00:25:18.790 align:middle line:84% Now, these equations are quite important. 00:25:18.790 --> 00:25:23.240 align:middle line:84% I mean, they're the central point in the Markov chain. 00:25:23.240 --> 00:25:24.220 align:middle line:90% They have a name. 00:25:24.220 --> 00:25:27.030 align:middle line:84% They're called the balance equations. 00:25:27.030 --> 00:25:31.260 align:middle line:84% And it's worth interpreting them in a 00:25:31.260 --> 00:25:33.450 align:middle line:90% somewhat different way. 00:25:33.450 --> 00:25:37.030 align:middle line:84% So intuitively, one can sometimes think of 00:25:37.030 --> 00:25:39.290 align:middle line:90% probabilities as frequencies. 00:25:39.290 --> 00:25:45.780 align:middle line:84% For example, if I toss an unbiased coin, probability 1/2 00:25:45.780 --> 00:25:49.600 align:middle line:84% of heads, you could also say that if I keep flipping that 00:25:49.600 --> 00:25:52.510 align:middle line:84% coin, in the long run, 1/2 of the time, I'm 00:25:52.510 --> 00:25:54.500 align:middle line:90% going to see heads. 00:25:54.500 --> 00:25:58.910 align:middle line:84% Similarly, let's try an interpretation of this pi(j), 00:25:58.910 --> 00:26:02.500 align:middle line:84% the steady-state probability, the long-term probability of 00:26:02.500 --> 00:26:04.980 align:middle line:90% finding myself at state j. 00:26:04.980 --> 00:26:08.440 align:middle line:84% Let's try to interpret it as the frequency with which I 00:26:08.440 --> 00:26:12.620 align:middle line:84% find myself at state j if I run a very, very long 00:26:12.620 --> 00:26:14.940 align:middle line:84% trajectory over that Markov chain. 00:26:14.940 --> 00:26:18.400 align:middle line:84% So the trajectory moves around, visits states. 00:26:18.400 --> 00:26:22.470 align:middle line:84% It visits the different states with different frequencies. 00:26:22.470 --> 00:26:27.380 align:middle line:84% And let's think of the probability that you are at a 00:26:27.380 --> 00:26:32.270 align:middle line:84% certain state as being sort of the same as the frequency of 00:26:32.270 --> 00:26:34.420 align:middle line:90% visiting that state. 00:26:34.420 --> 00:26:37.290 align:middle line:84% This turns out to be a correct statement. 00:26:37.290 --> 00:26:41.040 align:middle line:84% If you were more rigorous, you would have to prove it. 00:26:41.040 --> 00:26:44.390 align:middle line:84% But it's an interpretation which is valid and which gives 00:26:44.390 --> 00:26:48.560 align:middle line:84% us a lot of intuition about what these equation is saying. 00:26:48.560 --> 00:26:50.170 align:middle line:90% So let's think as follows. 00:26:50.170 --> 00:26:54.240 align:middle line:84% Let's focus on a particular state j, and think of 00:26:54.240 --> 00:27:00.660 align:middle line:84% transitions into the state j versus transitions out of the 00:27:00.660 --> 00:27:05.410 align:middle line:84% state j, or transitions into j versus transitions 00:27:05.410 --> 00:27:07.080 align:middle line:90% starting from j. 00:27:07.080 --> 00:27:10.010 align:middle line:84% So transition starting from that includes a 00:27:10.010 --> 00:27:11.260 align:middle line:90% self-transition. 00:27:11.260 --> 00:27:14.980 align:middle line:90% 00:27:14.980 --> 00:27:15.110 align:middle line:90% Ok. 00:27:15.110 --> 00:27:18.300 align:middle line:84% So how often do we get a transition, if we interpret 00:27:18.300 --> 00:27:21.230 align:middle line:84% the pi(j)'s as frequencies, how often do we get a 00:27:21.230 --> 00:27:23.110 align:middle line:90% transition into j? 00:27:23.110 --> 00:27:25.870 align:middle line:90% Here's how we think about it. 00:27:25.870 --> 00:27:31.110 align:middle line:84% A fraction pi(1) of the time, we're going to be at state 1. 00:27:31.110 --> 00:27:35.070 align:middle line:84% Whenever we are at state 1, there's going to be a 00:27:35.070 --> 00:27:40.550 align:middle line:84% probability, P1j, that we make a transition of this kind. 00:27:40.550 --> 00:27:44.730 align:middle line:84% So out of the times that we're at state 1, there's a 00:27:44.730 --> 00:27:50.141 align:middle line:84% frequency, P1j with which the next transition is into j. 00:27:50.141 --> 00:27:53.230 align:middle line:90% 00:27:53.230 --> 00:27:57.870 align:middle line:84% So out of the overall number of transitions that happen at 00:27:57.870 --> 00:28:01.820 align:middle line:84% the trajectory, what fraction of those transitions is 00:28:01.820 --> 00:28:03.700 align:middle line:90% exactly of that kind? 00:28:03.700 --> 00:28:06.710 align:middle line:84% That fraction of transitions is the fraction of time that 00:28:06.710 --> 00:28:11.940 align:middle line:84% you find yourself at 1 times the fraction with which out of 00:28:11.940 --> 00:28:15.700 align:middle line:84% one you happen to visit next state j. 00:28:15.700 --> 00:28:19.300 align:middle line:84% So we interpreted this number as the frequency of 00:28:19.300 --> 00:28:21.610 align:middle line:90% transitions of this kind. 00:28:21.610 --> 00:28:24.780 align:middle line:84% At any given time, our chain can do transitions of 00:28:24.780 --> 00:28:28.470 align:middle line:84% different kinds, transitions of the general form from some 00:28:28.470 --> 00:28:30.670 align:middle line:90% k, I go to some l. 00:28:30.670 --> 00:28:33.600 align:middle line:84% So we try to do some accounting. 00:28:33.600 --> 00:28:37.740 align:middle line:84% How often does a transition of each particular kind happen? 00:28:37.740 --> 00:28:40.950 align:middle line:84% And this is the frequency with which transitions of that 00:28:40.950 --> 00:28:42.970 align:middle line:90% particular kind happens. 00:28:42.970 --> 00:28:44.610 align:middle line:84% Now, what's the total frequency of 00:28:44.610 --> 00:28:46.870 align:middle line:90% transitions into state j? 00:28:46.870 --> 00:28:49.880 align:middle line:84% Transitions into state j can happen by having a transition 00:28:49.880 --> 00:28:54.510 align:middle line:84% from 1 to j, from 2 to j, or from state m to j. 00:28:54.510 --> 00:28:58.590 align:middle line:84% So to find the total frequency with which we would observe 00:28:58.590 --> 00:29:03.960 align:middle line:84% transitions into j is going to be this particular sum. 00:29:03.960 --> 00:29:09.090 align:middle line:84% Now, you are at state j if and only if the last transition 00:29:09.090 --> 00:29:11.440 align:middle line:90% was into state j. 00:29:11.440 --> 00:29:16.230 align:middle line:84% So the frequency with which you are at j is the frequency 00:29:16.230 --> 00:29:20.100 align:middle line:84% with which transitions into j happen. 00:29:20.100 --> 00:29:24.020 align:middle line:84% So this equation expresses exactly that statement. 00:29:24.020 --> 00:29:27.740 align:middle line:84% The probability of being at state j is the sum of the 00:29:27.740 --> 00:29:32.440 align:middle line:84% probabilities that the last transition was into state j. 00:29:32.440 --> 00:29:35.120 align:middle line:84% Or in terms of frequencies, the frequency with which you 00:29:35.120 --> 00:29:39.170 align:middle line:84% find yourself at state j is the sum of the frequencies of 00:29:39.170 --> 00:29:42.580 align:middle line:84% all the possible transition types that take you 00:29:42.580 --> 00:29:45.360 align:middle line:90% inside state j. 00:29:45.360 --> 00:29:47.770 align:middle line:84% So that's a useful intuition to have, and we're going to 00:29:47.770 --> 00:29:52.360 align:middle line:84% see an example a little later that it gives us short cuts 00:29:52.360 --> 00:29:55.240 align:middle line:90% into analyzing Markov chains. 00:29:55.240 --> 00:29:58.270 align:middle line:84% But before we move, let's revisit the 00:29:58.270 --> 00:30:01.450 align:middle line:90% example from last time. 00:30:01.450 --> 00:30:03.560 align:middle line:84% And let us write down the balance 00:30:03.560 --> 00:30:06.090 align:middle line:90% equations for this example. 00:30:06.090 --> 00:30:09.380 align:middle line:84% So the steady-state probability that I find myself 00:30:09.380 --> 00:30:16.370 align:middle line:84% at state 1 is the probability that the previous time I was 00:30:16.370 --> 00:30:21.550 align:middle line:84% at state 1 and I made a self-transition-- 00:30:21.550 --> 00:30:25.540 align:middle line:84% So the probability that I was here last time and I made a 00:30:25.540 --> 00:30:28.290 align:middle line:84% transition of this kind, plus the probability that the last 00:30:28.290 --> 00:30:32.220 align:middle line:84% time I was here and I made a transition of that kind. 00:30:32.220 --> 00:30:36.100 align:middle line:90% So plus pi(2) times 0.2. 00:30:36.100 --> 00:30:43.060 align:middle line:84% And similarly, for the other states, the steady-state 00:30:43.060 --> 00:30:46.250 align:middle line:84% probably that I find myself at state 2 is the probability 00:30:46.250 --> 00:30:50.650 align:middle line:84% that last time I was at state 1 and I made a transition into 00:30:50.650 --> 00:30:53.780 align:middle line:84% state 2, plus the probability that the last time I was at 00:30:53.780 --> 00:30:57.750 align:middle line:84% state 2 and I made the transition into state 1. 00:30:57.750 --> 00:30:59.910 align:middle line:84% Now, these are two equations and two 00:30:59.910 --> 00:31:02.040 align:middle line:90% unknowns, pi(1) and pi(2). 00:31:02.040 --> 00:31:06.150 align:middle line:84% But you notice that both of these equations tell you the 00:31:06.150 --> 00:31:07.170 align:middle line:90% same thing. 00:31:07.170 --> 00:31:12.226 align:middle line:84% They tell you that 0.5pi(1) equals 0.2pi(2). 00:31:12.226 --> 00:31:17.770 align:middle line:90% 00:31:17.770 --> 00:31:21.820 align:middle line:84% Either of these equations tell you exactly this if you move 00:31:21.820 --> 00:31:22.910 align:middle line:90% terms around. 00:31:22.910 --> 00:31:25.450 align:middle line:84% So these two equations are not really two equations. 00:31:25.450 --> 00:31:27.010 align:middle line:90% It's just one equation. 00:31:27.010 --> 00:31:30.520 align:middle line:84% They are linearly dependent equations, and in order to 00:31:30.520 --> 00:31:33.320 align:middle line:84% solve the problem, we need the additional condition that 00:31:33.320 --> 00:31:36.300 align:middle line:90% pi(1) + pi(2) is equal to 1. 00:31:36.300 --> 00:31:38.420 align:middle line:84% Now, we have our system of two equations, 00:31:38.420 --> 00:31:39.880 align:middle line:90% which you can solve. 00:31:39.880 --> 00:31:45.030 align:middle line:84% And once you solve it, you find that pi(1) is 2/7 and 00:31:45.030 --> 00:31:48.610 align:middle line:90% pi(2) is 5/7. 00:31:48.610 --> 00:31:52.880 align:middle line:84% So these are the steady state probabilities of the two 00:31:52.880 --> 00:31:54.130 align:middle line:90% different states. 00:31:54.130 --> 00:31:56.600 align:middle line:90% 00:31:56.600 --> 00:32:01.770 align:middle line:84% If we start this chain, at some state, let's say state 1, 00:32:01.770 --> 00:32:07.050 align:middle line:84% and we let it run for a long, long time, the chain settles 00:32:07.050 --> 00:32:08.580 align:middle line:90% into steady state. 00:32:08.580 --> 00:32:09.440 align:middle line:90% What does that mean? 00:32:09.440 --> 00:32:12.470 align:middle line:84% It does not mean that the state itself 00:32:12.470 --> 00:32:13.960 align:middle line:90% enters steady state. 00:32:13.960 --> 00:32:17.390 align:middle line:84% The state will keep jumping around forever and ever. 00:32:17.390 --> 00:32:21.040 align:middle line:84% It will keep visiting both states once in a while. 00:32:21.040 --> 00:32:23.250 align:middle line:90% So the jumping never ceases. 00:32:23.250 --> 00:32:25.790 align:middle line:84% The thing that gets into steady state is the 00:32:25.790 --> 00:32:30.180 align:middle line:84% probability of finding yourself at state 1. 00:32:30.180 --> 00:32:34.020 align:middle line:84% So the probability that you find yourself at state 1 at 00:32:34.020 --> 00:32:37.640 align:middle line:84% time one trillion is approximately 2/7. 00:32:37.640 --> 00:32:40.990 align:middle line:84% The probability you find yourself at state 1 at time 00:32:40.990 --> 00:32:45.590 align:middle line:84% two trillions is again, approximately 2/7. 00:32:45.590 --> 00:32:48.640 align:middle line:84% So the probability of being in that state settles into a 00:32:48.640 --> 00:32:52.270 align:middle line:90% steady value. 00:32:52.270 --> 00:32:55.630 align:middle line:84% That's what the steady-state convergence means. 00:32:55.630 --> 00:32:58.370 align:middle line:84% It's convergence of probabilities, not convergence 00:32:58.370 --> 00:33:00.680 align:middle line:90% of the process itself. 00:33:00.680 --> 00:33:04.750 align:middle line:84% And again, the two main things that are happening in this 00:33:04.750 --> 00:33:07.940 align:middle line:84% example, and more generally, when we have a single class 00:33:07.940 --> 00:33:10.650 align:middle line:84% and no periodicity, is that the initial 00:33:10.650 --> 00:33:12.600 align:middle line:90% state does not matter. 00:33:12.600 --> 00:33:15.980 align:middle line:84% There's enough randomness here so that no matter where you 00:33:15.980 --> 00:33:19.720 align:middle line:84% start, the randomness kind of washes out any memory of where 00:33:19.720 --> 00:33:20.780 align:middle line:90% you started. 00:33:20.780 --> 00:33:23.270 align:middle line:84% And also in this example, clearly, we do not have 00:33:23.270 --> 00:33:27.500 align:middle line:84% periodicity because we have self arcs. 00:33:27.500 --> 00:33:30.960 align:middle line:84% And this, in particular, implies that the exact time 00:33:30.960 --> 00:33:32.210 align:middle line:90% does not matter. 00:33:32.210 --> 00:33:35.740 align:middle line:90% 00:33:35.740 --> 00:33:41.510 align:middle line:84% So now, we're going to spend the rest of our time by 00:33:41.510 --> 00:33:45.330 align:middle line:84% looking into a special class of chains that's a little 00:33:45.330 --> 00:33:47.450 align:middle line:84% easier to deal with, but still, it's 00:33:47.450 --> 00:33:49.436 align:middle line:90% an important class. 00:33:49.436 --> 00:33:52.010 align:middle line:90% So what's the moral from here? 00:33:52.010 --> 00:33:55.680 align:middle line:84% This was a simple example with two states, and we could find 00:33:55.680 --> 00:33:59.440 align:middle line:84% the steady-state probabilities by solving a simple system of 00:33:59.440 --> 00:34:01.320 align:middle line:90% two-by-two equations. 00:34:01.320 --> 00:34:05.690 align:middle line:84% If you have a chain with 100 states, it's no problem for a 00:34:05.690 --> 00:34:09.120 align:middle line:84% computer to solve a system of 100-by-100 equations. 00:34:09.120 --> 00:34:12.800 align:middle line:84% But you can certainly not do it by hand, and usually, you 00:34:12.800 --> 00:34:15.560 align:middle line:84% cannot get any closed-form formulas, so you do not 00:34:15.560 --> 00:34:17.969 align:middle line:84% necessarily get a lot of insight. 00:34:17.969 --> 00:34:21.330 align:middle line:84% So one looks for special structures or models that 00:34:21.330 --> 00:34:25.610 align:middle line:84% maybe give you a little more insight or maybe lead you to 00:34:25.610 --> 00:34:27.690 align:middle line:90% closed-form formulas. 00:34:27.690 --> 00:34:32.130 align:middle line:84% And an interesting subclass of Markov chains in which all of 00:34:32.130 --> 00:34:35.080 align:middle line:84% these nice things do happen, is the class 00:34:35.080 --> 00:34:39.080 align:middle line:90% of birth/death processes. 00:34:39.080 --> 00:34:41.500 align:middle line:84% So what's a birth/death process? 00:34:41.500 --> 00:34:44.510 align:middle line:84% It's a Markov chain who's diagram looks 00:34:44.510 --> 00:34:46.810 align:middle line:90% basically like this. 00:34:46.810 --> 00:34:52.020 align:middle line:84% So the states of the Markov chain start from 0 and go up 00:34:52.020 --> 00:34:54.460 align:middle line:90% to some finite integer m. 00:34:54.460 --> 00:34:57.400 align:middle line:84% What's special about this chain is that if you are at a 00:34:57.400 --> 00:35:02.710 align:middle line:84% certain state, next time you can either go up by 1, you can 00:35:02.710 --> 00:35:06.640 align:middle line:84% go down by 1, or you can stay in place. 00:35:06.640 --> 00:35:09.820 align:middle line:84% So it's like keeping track of some population 00:35:09.820 --> 00:35:11.310 align:middle line:90% at any given time. 00:35:11.310 --> 00:35:13.820 align:middle line:84% One person gets born, or one person 00:35:13.820 --> 00:35:15.680 align:middle line:90% dies, or nothing happens. 00:35:15.680 --> 00:35:19.150 align:middle line:84% Again, we're not accounting for twins here. 00:35:19.150 --> 00:35:24.430 align:middle line:84% So we're given this structure, and we are given the 00:35:24.430 --> 00:35:27.660 align:middle line:84% transition probabilities, the probabilities associated with 00:35:27.660 --> 00:35:29.630 align:middle line:84% transitions of the different types. 00:35:29.630 --> 00:35:32.880 align:middle line:84% So we use P's for the upward transitions, Q's for the 00:35:32.880 --> 00:35:34.550 align:middle line:90% downward transitions. 00:35:34.550 --> 00:35:37.830 align:middle line:84% An example of a chain of this kind was the supermarket 00:35:37.830 --> 00:35:40.410 align:middle line:84% counter model that we discussed last time. 00:35:40.410 --> 00:35:45.080 align:middle line:84% That is, a customer arrives, so this increments 00:35:45.080 --> 00:35:46.310 align:middle line:90% the state by 1. 00:35:46.310 --> 00:35:49.670 align:middle line:84% Or a customer finishes service, in which case, the 00:35:49.670 --> 00:35:53.020 align:middle line:84% state gets decremented by 1, or nothing happens in which 00:35:53.020 --> 00:35:55.500 align:middle line:90% you stay in place, and so on. 00:35:55.500 --> 00:35:59.880 align:middle line:84% In the supermarket model, these P's inside here were all 00:35:59.880 --> 00:36:03.780 align:middle line:84% taken to be equal because we assume that the arrival rate 00:36:03.780 --> 00:36:07.400 align:middle line:84% was sort of constant at each time slot. 00:36:07.400 --> 00:36:10.960 align:middle line:84% But you can generalize a little bit by assuming that 00:36:10.960 --> 00:36:15.330 align:middle line:84% these transition probabilities P1 here, P2 there, and so on 00:36:15.330 --> 00:36:18.190 align:middle line:84% may be different from state to state. 00:36:18.190 --> 00:36:21.750 align:middle line:84% So in general, from state i, there's going to be a 00:36:21.750 --> 00:36:24.400 align:middle line:84% transition probability Pi that the next 00:36:24.400 --> 00:36:26.210 align:middle line:90% transition is upwards. 00:36:26.210 --> 00:36:29.820 align:middle line:84% And there's going to be a probability Qi that the next 00:36:29.820 --> 00:36:31.910 align:middle line:90% transition is downwards. 00:36:31.910 --> 00:36:35.310 align:middle line:84% And so from that state, the probability that the next 00:36:35.310 --> 00:36:37.640 align:middle line:84% transition is downwards is going to be Q_(i+1). 00:36:37.640 --> 00:36:40.930 align:middle line:90% 00:36:40.930 --> 00:36:43.650 align:middle line:84% So this is the structure of our chain. 00:36:43.650 --> 00:36:47.580 align:middle line:84% As I said, it's a crude model of what happens at the 00:36:47.580 --> 00:36:53.820 align:middle line:84% supermarket counter but it's also a good model for lots of 00:36:53.820 --> 00:36:55.370 align:middle line:90% types of service systems. 00:36:55.370 --> 00:36:59.400 align:middle line:84% Again, you have a server somewhere that has a buffer. 00:36:59.400 --> 00:37:00.970 align:middle line:90% Jobs come into the buffer. 00:37:00.970 --> 00:37:02.460 align:middle line:90% So the buffer builds up. 00:37:02.460 --> 00:37:06.880 align:middle line:84% The server processes jobs, so the buffer keeps going down. 00:37:06.880 --> 00:37:10.470 align:middle line:84% And the state of the chain would be the number of jobs 00:37:10.470 --> 00:37:12.570 align:middle line:84% that you have inside your buffer. 00:37:12.570 --> 00:37:18.690 align:middle line:84% Or you could be thinking about active phone calls out of a 00:37:18.690 --> 00:37:19.680 align:middle line:90% certain city. 00:37:19.680 --> 00:37:22.520 align:middle line:84% Each time that the phone call is placed, the number of 00:37:22.520 --> 00:37:24.090 align:middle line:84% active phone calls goes up by 1. 00:37:24.090 --> 00:37:28.360 align:middle line:84% Each time that the phone call stops happening, is 00:37:28.360 --> 00:37:31.790 align:middle line:84% terminated, then the count goes down by 1. 00:37:31.790 --> 00:37:34.710 align:middle line:84% So it's for processes of this kind that a model with this 00:37:34.710 --> 00:37:36.890 align:middle line:90% structure is going to show up. 00:37:36.890 --> 00:37:39.240 align:middle line:84% And they do show up in many, many models. 00:37:39.240 --> 00:37:43.730 align:middle line:84% Or you can think about the number of people in a certain 00:37:43.730 --> 00:37:45.690 align:middle line:84% population that have a disease. 00:37:45.690 --> 00:37:51.010 align:middle line:84% So 1 more person gets the flu, the count goes up. 00:37:51.010 --> 00:37:55.320 align:middle line:84% 1 more person gets healed, the count goes down. 00:37:55.320 --> 00:37:58.350 align:middle line:84% And these probabilities in such an epidemic model would 00:37:58.350 --> 00:38:02.270 align:middle line:84% certainly depend on the current state. 00:38:02.270 --> 00:38:06.280 align:middle line:84% If lots of people already have the flu, the probability that 00:38:06.280 --> 00:38:10.010 align:middle line:84% another person catches it would be pretty high. 00:38:10.010 --> 00:38:13.800 align:middle line:84% Whereas, if no one has the flu, then the probability that 00:38:13.800 --> 00:38:16.820 align:middle line:84% you get a transition where someone catches the flu, that 00:38:16.820 --> 00:38:18.960 align:middle line:84% probability would be pretty small. 00:38:18.960 --> 00:38:26.990 align:middle line:84% So the transition rates, the incidence of new people who 00:38:26.990 --> 00:38:30.010 align:middle line:84% have the disease definitely depends on how many people 00:38:30.010 --> 00:38:31.560 align:middle line:90% already have the disease. 00:38:31.560 --> 00:38:34.970 align:middle line:84% And that motivates cases where those P's, the upward 00:38:34.970 --> 00:38:39.220 align:middle line:84% transition probabilities, depend on the 00:38:39.220 --> 00:38:42.400 align:middle line:90% state of the chain. 00:38:42.400 --> 00:38:44.040 align:middle line:90% So how do we study this chain? 00:38:44.040 --> 00:38:49.220 align:middle line:84% You can sit down and write the system of n linear equations 00:38:49.220 --> 00:38:50.550 align:middle line:90% in the pi's. 00:38:50.550 --> 00:38:52.920 align:middle line:84% And this way, find the steady-state probabilities of 00:38:52.920 --> 00:38:53.790 align:middle line:90% this chain. 00:38:53.790 --> 00:38:55.850 align:middle line:90% But this is a little harder. 00:38:55.850 --> 00:38:59.310 align:middle line:84% It's more work than one actually needs to do. 00:38:59.310 --> 00:39:03.200 align:middle line:84% There's a very clever shortcut that applies 00:39:03.200 --> 00:39:05.160 align:middle line:90% to birth/death processes. 00:39:05.160 --> 00:39:08.410 align:middle line:84% And it's based on the frequency interpretation that 00:39:08.410 --> 00:39:10.000 align:middle line:84% we discussed a little while ago. 00:39:10.000 --> 00:39:14.070 align:middle line:90% 00:39:14.070 --> 00:39:17.390 align:middle line:84% Let's put a line somewhere in the middle of this chain, and 00:39:17.390 --> 00:39:21.710 align:middle line:84% focus on the relation between this part and that part in 00:39:21.710 --> 00:39:22.750 align:middle line:90% more detail. 00:39:22.750 --> 00:39:25.460 align:middle line:84% So think of the chain continuing in this direction, 00:39:25.460 --> 00:39:26.390 align:middle line:90% that direction. 00:39:26.390 --> 00:39:30.020 align:middle line:84% But let's just focus on 2 adjacent states, and look at 00:39:30.020 --> 00:39:32.270 align:middle line:90% this particular cut. 00:39:32.270 --> 00:39:34.270 align:middle line:90% What is the chain going to do? 00:39:34.270 --> 00:39:35.550 align:middle line:90% Let's say it starts here. 00:39:35.550 --> 00:39:37.250 align:middle line:90% It's going to move around. 00:39:37.250 --> 00:39:40.280 align:middle line:84% At some point, it makes a transition to the other side. 00:39:40.280 --> 00:39:42.860 align:middle line:84% And that's a transition from i to i+1. 00:39:42.860 --> 00:39:45.470 align:middle line:84% It stays on the other side for some time. 00:39:45.470 --> 00:39:48.490 align:middle line:84% It gets here, and eventually, it's going to make a 00:39:48.490 --> 00:39:50.340 align:middle line:90% transition to this side. 00:39:50.340 --> 00:39:53.110 align:middle line:84% Then it keeps moving and so on. 00:39:53.110 --> 00:39:57.680 align:middle line:84% Now, there's a certain balance that must be obeyed here. 00:39:57.680 --> 00:40:01.300 align:middle line:84% The number of upward transitions through this line 00:40:01.300 --> 00:40:04.220 align:middle line:84% cannot be very different from the number of downward 00:40:04.220 --> 00:40:06.080 align:middle line:90% transitions. 00:40:06.080 --> 00:40:09.530 align:middle line:84% Because we cross this way, then next time, 00:40:09.530 --> 00:40:10.630 align:middle line:90% we'll cross that way. 00:40:10.630 --> 00:40:12.580 align:middle line:84% Then next time, we'll cross this way. 00:40:12.580 --> 00:40:13.970 align:middle line:90% We'll cross that way. 00:40:13.970 --> 00:40:18.940 align:middle line:84% So the frequency with which transitions of this kind occur 00:40:18.940 --> 00:40:21.500 align:middle line:84% has to be the same as the long-term frequency that 00:40:21.500 --> 00:40:24.430 align:middle line:84% transitions of that kind occur. 00:40:24.430 --> 00:40:28.510 align:middle line:84% You cannot go up 100 times and go down only 50 times. 00:40:28.510 --> 00:40:31.740 align:middle line:84% If you have gone up 100 times, it means that you have gone 00:40:31.740 --> 00:40:36.960 align:middle line:84% down 99, or 100, or 101, but nothing much more 00:40:36.960 --> 00:40:38.570 align:middle line:90% different than that. 00:40:38.570 --> 00:40:41.440 align:middle line:84% So the frequency with which transitions of 00:40:41.440 --> 00:40:43.670 align:middle line:90% this kind get observed. 00:40:43.670 --> 00:40:47.610 align:middle line:84% That is, out of a large number of transitions, what fraction 00:40:47.610 --> 00:40:49.890 align:middle line:84% of transitions are of these kind? 00:40:49.890 --> 00:40:52.160 align:middle line:84% That fraction has to be the same as the fraction of 00:40:52.160 --> 00:40:54.870 align:middle line:84% transitions that happened to be of that kind. 00:40:54.870 --> 00:40:56.620 align:middle line:90% What are these fractions? 00:40:56.620 --> 00:40:58.480 align:middle line:90% We discussed that before. 00:40:58.480 --> 00:41:04.840 align:middle line:84% The fraction of times at which transitions of this kind are 00:41:04.840 --> 00:41:08.340 align:middle line:84% observed is the fraction of time that we happen to be at 00:41:08.340 --> 00:41:09.350 align:middle line:90% that state. 00:41:09.350 --> 00:41:11.790 align:middle line:84% And out of the times that we are in that state, the 00:41:11.790 --> 00:41:15.010 align:middle line:84% fraction of transitions that happen to be upward 00:41:15.010 --> 00:41:16.040 align:middle line:90% transitions. 00:41:16.040 --> 00:41:20.820 align:middle line:84% So this is the frequency with which transitions of this kind 00:41:20.820 --> 00:41:22.430 align:middle line:90% are observed. 00:41:22.430 --> 00:41:25.350 align:middle line:84% And with the same argument, this is the frequency with 00:41:25.350 --> 00:41:28.670 align:middle line:84% which transitions of that kind are observed. 00:41:28.670 --> 00:41:31.120 align:middle line:84% Since these two frequencies are the same, these two 00:41:31.120 --> 00:41:34.390 align:middle line:84% numbers must be the same, and we get an equation that 00:41:34.390 --> 00:41:38.350 align:middle line:90% relates the Pi to P_(i+1). 00:41:38.350 --> 00:41:43.040 align:middle line:84% This has a nice form because it gives us a recursion. 00:41:43.040 --> 00:41:45.760 align:middle line:90% If we knew pi(i), we could then 00:41:45.760 --> 00:41:48.860 align:middle line:90% immediately calculate pi(i+1). 00:41:48.860 --> 00:41:51.860 align:middle line:84% So it's a system of equations that's very 00:41:51.860 --> 00:41:54.860 align:middle line:90% easy to solve almost. 00:41:54.860 --> 00:41:57.320 align:middle line:90% But how do we get started? 00:41:57.320 --> 00:42:01.850 align:middle line:84% If I knew pi(0), I could find by pi(1) and then use this 00:42:01.850 --> 00:42:05.330 align:middle line:84% recursion to find pi(2), pi(3), and so on. 00:42:05.330 --> 00:42:06.970 align:middle line:90% But we don't know pi(0). 00:42:06.970 --> 00:42:09.920 align:middle line:90% It's one more unknown. 00:42:09.920 --> 00:42:14.290 align:middle line:84% It's an unknown, and we need to actually use the extra 00:42:14.290 --> 00:42:20.000 align:middle line:84% normalization condition that the sum of the pi's is 1. 00:42:20.000 --> 00:42:23.810 align:middle line:84% And after we use that normalization condition, then 00:42:23.810 --> 00:42:26.580 align:middle line:90% we can find all of the pi's. 00:42:26.580 --> 00:42:32.550 align:middle line:90% 00:42:32.550 --> 00:42:38.450 align:middle line:84% So you basically fix pi(0) as a symbol, solve this equation 00:42:38.450 --> 00:42:41.580 align:middle line:84% symbolically, and everything gets 00:42:41.580 --> 00:42:43.830 align:middle line:90% expressed in terms of pi(0). 00:42:43.830 --> 00:42:46.510 align:middle line:84% And then use that normalization condition to 00:42:46.510 --> 00:42:48.700 align:middle line:90% find pi(0), and you're done. 00:42:48.700 --> 00:42:51.570 align:middle line:84% Let's illustrate the details of this procedure on a 00:42:51.570 --> 00:42:53.690 align:middle line:90% particular special case. 00:42:53.690 --> 00:42:57.360 align:middle line:84% So in our special case, we're going to simplify things now 00:42:57.360 --> 00:43:01.760 align:middle line:84% by assuming that all those upward P's are the same, and 00:43:01.760 --> 00:43:05.780 align:middle line:84% all of those downward Q's are the same. 00:43:05.780 --> 00:43:08.860 align:middle line:84% So at each point in time, if you're sitting somewhere in 00:43:08.860 --> 00:43:13.060 align:middle line:84% the middle, you have probability P of moving up and 00:43:13.060 --> 00:43:16.710 align:middle line:90% probability Q of moving down. 00:43:16.710 --> 00:43:23.460 align:middle line:84% This rho, the ratio of P/Q is frequency of going up versus 00:43:23.460 --> 00:43:25.670 align:middle line:90% frequency of going down. 00:43:25.670 --> 00:43:29.070 align:middle line:84% If it's a service system, you can think of it as a measure 00:43:29.070 --> 00:43:32.140 align:middle line:90% of how loaded the system is. 00:43:32.140 --> 00:43:39.100 align:middle line:84% If P is equal to Q, it's means that if you're at this state, 00:43:39.100 --> 00:43:42.700 align:middle line:84% you're equally likely to move left or right, so the system 00:43:42.700 --> 00:43:44.820 align:middle line:90% is kind of balanced. 00:43:44.820 --> 00:43:46.980 align:middle line:84% The state doesn't have a tendency to move in this 00:43:46.980 --> 00:43:49.450 align:middle line:84% direction or in that direction. 00:43:49.450 --> 00:43:53.860 align:middle line:84% If rho is bigger than 1 so that P is bigger than Q, it 00:43:53.860 --> 00:43:56.730 align:middle line:84% means that whenever I'm at some state in the middle, I'm 00:43:56.730 --> 00:44:00.830 align:middle line:84% more likely to move right rather than move left, which 00:44:00.830 --> 00:44:04.230 align:middle line:84% means that my state, of course it's random, but it has a 00:44:04.230 --> 00:44:07.170 align:middle line:84% tendency to move in that direction. 00:44:07.170 --> 00:44:10.280 align:middle line:84% And if you think of this as a number of customers in queue, 00:44:10.280 --> 00:44:13.750 align:middle line:84% it means your system has the tendency to become loaded and 00:44:13.750 --> 00:44:15.420 align:middle line:90% to build up a queue. 00:44:15.420 --> 00:44:19.790 align:middle line:84% So rho being bigger than 1 corresponds to a heavy load, 00:44:19.790 --> 00:44:21.570 align:middle line:90% where queues build up. 00:44:21.570 --> 00:44:25.470 align:middle line:84% Rho less than 1 corresponds to the system where queues have 00:44:25.470 --> 00:44:27.310 align:middle line:90% the tendency to drain down. 00:44:27.310 --> 00:44:30.880 align:middle line:90% 00:44:30.880 --> 00:44:32.680 align:middle line:84% Now, let's write down the equations. 00:44:32.680 --> 00:44:40.540 align:middle line:84% We have this recursion P_(i+1) is Pi times Pi over Qi. 00:44:40.540 --> 00:44:44.040 align:middle line:84% In our case here, the P's and the Q's do not depend on the 00:44:44.040 --> 00:44:47.190 align:middle line:84% particular index, so we get this relation. 00:44:47.190 --> 00:44:51.830 align:middle line:84% And this P over Q is just the load factor rho. 00:44:51.830 --> 00:44:54.790 align:middle line:84% Once you look at this equation, clearly you realize 00:44:54.790 --> 00:44:58.440 align:middle line:84% that by pi(1) is rho times pi(0). 00:44:58.440 --> 00:45:02.286 align:middle line:90% pi(2) is going to be -- 00:45:02.286 --> 00:45:04.480 align:middle line:90% So we'll do it in detail. 00:45:04.480 --> 00:45:08.130 align:middle line:90% So pi(1) is pi(0) times rho. 00:45:08.130 --> 00:45:15.580 align:middle line:84% pi(2) is pi(1) times rho, which is pi(0) times 00:45:15.580 --> 00:45:18.350 align:middle line:90% rho-squared. 00:45:18.350 --> 00:45:21.340 align:middle line:84% And then you continue doing this calculation. 00:45:21.340 --> 00:45:25.530 align:middle line:84% And you find that you can express every pi(i) in terms 00:45:25.530 --> 00:45:31.490 align:middle line:84% of pi(0) and you get this factor of rho^i. 00:45:31.490 --> 00:45:34.840 align:middle line:84% And then you use the last equation that we have -- that 00:45:34.840 --> 00:45:38.110 align:middle line:84% the sum of the probabilities has to be equal to 1. 00:45:38.110 --> 00:45:41.670 align:middle line:84% And that equation is going to tell us that the sum over all 00:45:41.670 --> 00:45:50.890 align:middle line:84% i's from 0 to m of pi(0) rho to the i is equal to 1. 00:45:50.890 --> 00:45:58.730 align:middle line:84% And therefore, pi(0) is 1 over (the sum over the rho to the i 00:45:58.730 --> 00:46:03.100 align:middle line:90% for i going from 0 to m). 00:46:03.100 --> 00:46:09.870 align:middle line:84% So now we found pi(0), and by plugging in this expression, 00:46:09.870 --> 00:46:12.680 align:middle line:84% we have the steady-state probabilities of all of the 00:46:12.680 --> 00:46:14.950 align:middle line:90% different states. 00:46:14.950 --> 00:46:18.990 align:middle line:84% Let's look at some special cases of this. 00:46:18.990 --> 00:46:24.390 align:middle line:84% Suppose that rho is equal to 1. 00:46:24.390 --> 00:46:30.410 align:middle line:84% If rho is equal to 1, then pi(i) is equal to pi(0). 00:46:30.410 --> 00:46:33.250 align:middle line:84% It means that all the steady-state 00:46:33.250 --> 00:46:35.840 align:middle line:90% probabilities are equal. 00:46:35.840 --> 00:46:39.380 align:middle line:84% It's means that every state is equally 00:46:39.380 --> 00:46:42.750 align:middle line:90% likely in the long run. 00:46:42.750 --> 00:46:44.790 align:middle line:90% So this is an example. 00:46:44.790 --> 00:46:48.730 align:middle line:84% It's called a symmetric random walk. 00:46:48.730 --> 00:46:53.260 align:middle line:84% It's a very popular model for modeling people who are drunk. 00:46:53.260 --> 00:46:56.730 align:middle line:84% So you start at a state at any point in time. 00:46:56.730 --> 00:47:00.140 align:middle line:84% Either you stay in place, or you have an equal probability 00:47:00.140 --> 00:47:02.910 align:middle line:90% of going left or going right. 00:47:02.910 --> 00:47:06.220 align:middle line:84% There's no bias in either direction. 00:47:06.220 --> 00:47:10.710 align:middle line:84% You might think that in such a process, you will tend to kind 00:47:10.710 --> 00:47:14.750 align:middle line:84% of get stuck near one end or the other end. 00:47:14.750 --> 00:47:17.320 align:middle line:84% Well, it's not really clear what to expect. 00:47:17.320 --> 00:47:21.260 align:middle line:84% It turns out that in such a model, in the long run, the 00:47:21.260 --> 00:47:24.610 align:middle line:84% drunk person is equally likely to be at any 00:47:24.610 --> 00:47:26.430 align:middle line:90% one of those states. 00:47:26.430 --> 00:47:31.300 align:middle line:84% The steady-state probability is the same for all i's if rho 00:47:31.300 --> 00:47:33.880 align:middle line:90% is equal to 1. 00:47:33.880 --> 00:47:39.570 align:middle line:84% And so if you show up at a random time, and you ask where 00:47:39.570 --> 00:47:43.670 align:middle line:84% is my state, you will be told it's equally likely to be at 00:47:43.670 --> 00:47:46.600 align:middle line:90% any one of those places. 00:47:46.600 --> 00:47:48.370 align:middle line:90% So let's make that note. 00:47:48.370 --> 00:47:51.980 align:middle line:84% If rho equal to 1, implies that all the 00:47:51.980 --> 00:47:53.660 align:middle line:90% pi(i)'s are 1/(M+1) -- 00:47:53.660 --> 00:47:57.040 align:middle line:90% 00:47:57.040 --> 00:48:01.320 align:middle line:84% M+1 because that's how many states we have in our model. 00:48:01.320 --> 00:48:04.210 align:middle line:84% Now, let's look at a different case. 00:48:04.210 --> 00:48:08.630 align:middle line:84% Suppose that M is a huge number. 00:48:08.630 --> 00:48:13.600 align:middle line:84% So essentially, our supermarket has a very large 00:48:13.600 --> 00:48:19.600 align:middle line:84% space, a lot of space to store their customers. 00:48:19.600 --> 00:48:24.130 align:middle line:84% But suppose that the system is on the stable side. 00:48:24.130 --> 00:48:27.900 align:middle line:84% P is less than Q, which means that there's a tendency for 00:48:27.900 --> 00:48:31.420 align:middle line:84% customers to be served faster than they arrive. 00:48:31.420 --> 00:48:35.500 align:middle line:84% The drift in this chain, it tends to be in that direction. 00:48:35.500 --> 00:48:41.920 align:middle line:84% So when rho is less than 1, which is this case, and when M 00:48:41.920 --> 00:48:45.810 align:middle line:84% is going to infinity, this infinite sum is the sum of a 00:48:45.810 --> 00:48:47.690 align:middle line:90% geometric series. 00:48:47.690 --> 00:48:50.480 align:middle line:84% And you recognize it (hopefully) -- 00:48:50.480 --> 00:48:53.480 align:middle line:90% 00:48:53.480 --> 00:48:56.520 align:middle line:84% this series is going to 1/(1-rho). 00:48:56.520 --> 00:49:00.280 align:middle line:84% And because it's in the denominator, pi(0) ends up 00:49:00.280 --> 00:49:02.970 align:middle line:90% being 1-rho. 00:49:02.970 --> 00:49:06.230 align:middle line:84% So by taking the limit as M goes to infinity, in this 00:49:06.230 --> 00:49:09.460 align:middle line:84% case, and when rho is less than 1 so that this series is 00:49:09.460 --> 00:49:12.220 align:middle line:84% convergent, we get this formula. 00:49:12.220 --> 00:49:15.780 align:middle line:84% So we get the closed-form formula for the pi(i)'s. 00:49:15.780 --> 00:49:19.840 align:middle line:84% In particular, pi(i) is (1- rho)(rho to the i). 00:49:19.840 --> 00:49:21.150 align:middle line:90% to 00:49:21.150 --> 00:49:24.520 align:middle line:84% So these pi(i)'s are essentially a probability 00:49:24.520 --> 00:49:26.200 align:middle line:90% distribution. 00:49:26.200 --> 00:49:32.100 align:middle line:84% They tell us if we show up at time 1 billion and we ask, 00:49:32.100 --> 00:49:33.900 align:middle line:90% where is my state? 00:49:33.900 --> 00:49:37.500 align:middle line:84% You will be told that the state is 0. 00:49:37.500 --> 00:49:41.040 align:middle line:84% Your system is empty with probability 1-rho, minus or 00:49:41.040 --> 00:49:44.410 align:middle line:84% there's one customer in the system, and that happens with 00:49:44.410 --> 00:49:46.460 align:middle line:84% probability (rho - 1) times rho. 00:49:46.460 --> 00:49:49.950 align:middle line:84% And it keeps going down this way. 00:49:49.950 --> 00:49:53.920 align:middle line:84% And it's pretty much a geometric distribution except 00:49:53.920 --> 00:49:58.130 align:middle line:84% that it has shifted so that it starts at 0 whereas the usual 00:49:58.130 --> 00:50:00.890 align:middle line:84% geometric distribution starts at 1. 00:50:00.890 --> 00:50:04.670 align:middle line:84% So this is a mini introduction into queuing theory. 00:50:04.670 --> 00:50:08.730 align:middle line:84% This is the first and simplest model that one encounters when 00:50:08.730 --> 00:50:10.850 align:middle line:84% you start studying queuing theory. 00:50:10.850 --> 00:50:13.360 align:middle line:84% This is clearly a model of a queueing phenomenon such as 00:50:13.360 --> 00:50:16.450 align:middle line:84% the supermarket counter with the P's corresponding to 00:50:16.450 --> 00:50:19.280 align:middle line:84% arrivals, the Q's corresponding to departures. 00:50:19.280 --> 00:50:22.690 align:middle line:84% And this particular queuing system when M is very, very 00:50:22.690 --> 00:50:26.670 align:middle line:84% large and rho is less than 1, has a very simple and nice 00:50:26.670 --> 00:50:28.880 align:middle line:90% solution in closed form. 00:50:28.880 --> 00:50:31.650 align:middle line:84% And that's why it's very much liked. 00:50:31.650 --> 00:50:33.610 align:middle line:84% And let me just take two seconds to 00:50:33.610 --> 00:50:38.030 align:middle line:90% draw one last picture. 00:50:38.030 --> 00:50:40.300 align:middle line:84% So this is the probability of the different i's. 00:50:40.300 --> 00:50:41.420 align:middle line:90% It gives you a PMF. 00:50:41.420 --> 00:50:43.890 align:middle line:84% This PMF has an expected value. 00:50:43.890 --> 00:50:47.090 align:middle line:84% And the expectation, the expected number of customers 00:50:47.090 --> 00:50:50.560 align:middle line:84% in the system, is given by this formula. 00:50:50.560 --> 00:50:54.230 align:middle line:84% And this formula, which is interesting to anyone who 00:50:54.230 --> 00:50:57.070 align:middle line:84% tries to analyze a system of this kind, 00:50:57.070 --> 00:50:58.430 align:middle line:90% tells you the following. 00:50:58.430 --> 00:51:03.970 align:middle line:84% That as long as a rho is less than 1, then the expected 00:51:03.970 --> 00:51:07.270 align:middle line:84% number of customers in the system is finite. 00:51:07.270 --> 00:51:09.970 align:middle line:84% But if rho becomes very close to 1 -- 00:51:09.970 --> 00:51:13.790 align:middle line:84% So if your load factor is something like .99, you expect 00:51:13.790 --> 00:51:17.630 align:middle line:84% to have a large number of customers in the system at any 00:51:17.630 --> 00:51:19.040 align:middle line:90% given time. 00:51:19.040 --> 00:51:20.440 align:middle line:90% OK. 00:51:20.440 --> 00:51:22.420 align:middle line:90% All right. 00:51:22.420 --> 00:51:23.270 align:middle line:90% Have a good weekend. 00:51:23.270 --> 00:51:25.390 align:middle line:90% We'll continue next time.