WEBVTT 00:00:00.000 --> 00:00:00.040 align:middle line:90% 00:00:00.040 --> 00:00:02.460 align:middle line:84% The following content is provided under a Creative 00:00:02.460 --> 00:00:03.870 align:middle line:90% Commons license. 00:00:03.870 --> 00:00:06.910 align:middle line:84% Your support will help MIT OpenCourseWare continue to 00:00:06.910 --> 00:00:10.560 align:middle line:84% offer high quality educational resources for free. 00:00:10.560 --> 00:00:13.460 align:middle line:84% To make a donation or view additional materials from 00:00:13.460 --> 00:00:17.390 align:middle line:84% hundreds of MIT courses, visit MIT OpenCourseWare at 00:00:17.390 --> 00:00:18.640 align:middle line:90% ocw.mit.edu. 00:00:18.640 --> 00:00:23.440 align:middle line:90% 00:00:23.440 --> 00:00:25.930 align:middle line:84% JOHN TSITSIKLIS: So what we're going to do is to review what 00:00:25.930 --> 00:00:29.390 align:middle line:90% we have discussed last time. 00:00:29.390 --> 00:00:32.380 align:middle line:84% Then we're going to talk about the classic application of 00:00:32.380 --> 00:00:35.950 align:middle line:84% Markov chains to analyze how do you 00:00:35.950 --> 00:00:39.130 align:middle line:90% dimension a phone system. 00:00:39.130 --> 00:00:42.400 align:middle line:84% And finally, there will be two new things today. 00:00:42.400 --> 00:00:44.940 align:middle line:84% We will see how we can calculate certain interesting 00:00:44.940 --> 00:00:48.950 align:middle line:84% quantities that have to do with Markov chains. 00:00:48.950 --> 00:00:50.200 align:middle line:90% So let us start. 00:00:50.200 --> 00:00:52.860 align:middle line:90% 00:00:52.860 --> 00:00:56.680 align:middle line:84% We've got our Markov chain and let's make the assumption that 00:00:56.680 --> 00:00:58.800 align:middle line:90% our chain is kind of nice. 00:00:58.800 --> 00:01:01.550 align:middle line:84% And by nice we mean that we've got maybe 00:01:01.550 --> 00:01:02.800 align:middle line:90% some transient states. 00:01:02.800 --> 00:01:06.380 align:middle line:90% 00:01:06.380 --> 00:01:11.580 align:middle line:84% And then we've got a single recurrent class 00:01:11.580 --> 00:01:15.450 align:middle line:90% of recurrent states. 00:01:15.450 --> 00:01:18.600 align:middle line:84% So this is a single recurrent class in the sense that from 00:01:18.600 --> 00:01:22.080 align:middle line:84% any state in that class you can get to any other state. 00:01:22.080 --> 00:01:24.790 align:middle line:84% So once you're in here you're going to circulate and keep 00:01:24.790 --> 00:01:26.650 align:middle line:90% visiting all of those states. 00:01:26.650 --> 00:01:28.340 align:middle line:90% Those states appear transient. 00:01:28.340 --> 00:01:32.020 align:middle line:84% The trajectory may move around here, but eventually one of 00:01:32.020 --> 00:01:34.320 align:middle line:84% these transitions will happen and you're going to 00:01:34.320 --> 00:01:36.040 align:middle line:90% end up in this lump. 00:01:36.040 --> 00:01:38.640 align:middle line:90% 00:01:38.640 --> 00:01:42.680 align:middle line:84% Let's make the assumption that the single recurrent class is 00:01:42.680 --> 00:01:43.930 align:middle line:90% not periodic. 00:01:43.930 --> 00:01:48.170 align:middle line:90% 00:01:48.170 --> 00:01:51.740 align:middle line:84% These are the nicest kind of Markov chains. 00:01:51.740 --> 00:01:54.500 align:middle line:84% And they're nicest because they have the following 00:01:54.500 --> 00:01:58.900 align:middle line:84% property, the probability that you find yourself at some 00:01:58.900 --> 00:02:03.100 align:middle line:84% particular state j at the time n when that 00:02:03.100 --> 00:02:04.800 align:middle line:90% time is very large. 00:02:04.800 --> 00:02:08.130 align:middle line:84% That probability settles to a steady state value that we 00:02:08.130 --> 00:02:10.139 align:middle line:90% denote by pi sub j. 00:02:10.139 --> 00:02:12.630 align:middle line:84% And there are two parts in the statement. 00:02:12.630 --> 00:02:15.470 align:middle line:84% One part is that this limit exists. 00:02:15.470 --> 00:02:19.010 align:middle line:84% So the probability of state j settles to something, and 00:02:19.010 --> 00:02:23.080 align:middle line:84% furthermore that probability is not affected by i. 00:02:23.080 --> 00:02:25.550 align:middle line:84% It doesn't matter where you started, no matter where you 00:02:25.550 --> 00:02:28.310 align:middle line:84% started, the probability of state j is going to be the 00:02:28.310 --> 00:02:30.720 align:middle line:90% same in the long run. 00:02:30.720 --> 00:02:34.990 align:middle line:84% Maybe a clearer notation could be of this form. 00:02:34.990 --> 00:02:37.650 align:middle line:84% The probability of being at state j given the initial 00:02:37.650 --> 00:02:42.570 align:middle line:84% state being i is equal to pi(j) in the limit. 00:02:42.570 --> 00:02:47.380 align:middle line:84% Now, if I don't tell you where you started and you look at 00:02:47.380 --> 00:02:52.520 align:middle line:84% the unconditional probability of being at state i, you can 00:02:52.520 --> 00:02:56.260 align:middle line:84% average over the initial states, use the total 00:02:56.260 --> 00:03:01.560 align:middle line:84% expectation theorem and you're going to get the same answer 00:03:01.560 --> 00:03:05.330 align:middle line:90% pi(j) in the limit. 00:03:05.330 --> 00:03:09.400 align:middle line:84% So this tells you that to the conditional probability given 00:03:09.400 --> 00:03:11.820 align:middle line:84% the initial state in the limit is the same as the 00:03:11.820 --> 00:03:13.780 align:middle line:90% unconditional probability. 00:03:13.780 --> 00:03:17.200 align:middle line:84% And that's a situation that we recognize as being one where 00:03:17.200 --> 00:03:18.890 align:middle line:90% we have independence. 00:03:18.890 --> 00:03:24.970 align:middle line:84% So what this result tells us is that Xn and Xi are 00:03:24.970 --> 00:03:27.410 align:middle line:90% approximately independent. 00:03:27.410 --> 00:03:32.690 align:middle line:84% They become independent in the limit as n goes to infinity. 00:03:32.690 --> 00:03:35.130 align:middle line:84% So that's what the steady state theorem tells us. 00:03:35.130 --> 00:03:38.860 align:middle line:84% The initial conditions don't matter, so your state at some 00:03:38.860 --> 00:03:43.440 align:middle line:84% large time n has nothing to do, is not affected by what 00:03:43.440 --> 00:03:45.350 align:middle line:90% your initial state was. 00:03:45.350 --> 00:03:47.790 align:middle line:84% Knowing the initial state doesn't tell you anything 00:03:47.790 --> 00:03:50.830 align:middle line:84% about your state at time n, therefore the 00:03:50.830 --> 00:03:52.890 align:middle line:90% states at the times-- 00:03:52.890 --> 00:03:56.580 align:middle line:84% sorry that should be a 1, or it should be a 0 -- 00:03:56.580 --> 00:04:02.350 align:middle line:84% so the state is not affected by where the process started. 00:04:02.350 --> 00:04:05.450 align:middle line:84% So if the Markov chain is to operate for a long time and 00:04:05.450 --> 00:04:08.520 align:middle line:84% we're interested in the question where is the state, 00:04:08.520 --> 00:04:11.470 align:middle line:84% then your answer would be, I don't know, it's random. 00:04:11.470 --> 00:04:14.020 align:middle line:84% But it's going to be a particular j with this 00:04:14.020 --> 00:04:15.500 align:middle line:90% particular probability. 00:04:15.500 --> 00:04:18.550 align:middle line:84% So the steady state probabilities are interesting 00:04:18.550 --> 00:04:21.060 align:middle line:84% to us and that raises the question of how 00:04:21.060 --> 00:04:22.470 align:middle line:90% do we compute them. 00:04:22.470 --> 00:04:25.460 align:middle line:84% The way we compute them is by solving a linear system of 00:04:25.460 --> 00:04:28.580 align:middle line:84% equations, which are called the balance equations, 00:04:28.580 --> 00:04:31.510 align:middle line:84% together with an extra equation, the normalization 00:04:31.510 --> 00:04:33.690 align:middle line:84% equation that has to be satisfied by probability, 00:04:33.690 --> 00:04:37.940 align:middle line:84% because probabilities must always add up to 1. 00:04:37.940 --> 00:04:40.660 align:middle line:84% We talked about the interpretation of this 00:04:40.660 --> 00:04:42.280 align:middle line:90% equation last time. 00:04:42.280 --> 00:04:45.500 align:middle line:84% It's basically a conservation of probability 00:04:45.500 --> 00:04:47.400 align:middle line:90% flow in some sense. 00:04:47.400 --> 00:04:50.140 align:middle line:90% What comes in must get out. 00:04:50.140 --> 00:04:53.190 align:middle line:84% The probability of finding yourself at state j at a 00:04:53.190 --> 00:04:58.150 align:middle line:84% particular time is the total probability of the last 00:04:58.150 --> 00:05:01.390 align:middle line:84% transition taking me into state j. 00:05:01.390 --> 00:05:04.860 align:middle line:84% The last transition takes me into state j in various ways. 00:05:04.860 --> 00:05:07.890 align:middle line:84% It could be that the previous time I was at the particular 00:05:07.890 --> 00:05:12.550 align:middle line:84% state, j and i made a transition from k into j. 00:05:12.550 --> 00:05:15.610 align:middle line:84% So this number here, we interpret as the frequency 00:05:15.610 --> 00:05:18.190 align:middle line:84% with which transitions of these particular 00:05:18.190 --> 00:05:21.900 align:middle line:90% type k to j, occur. 00:05:21.900 --> 00:05:25.840 align:middle line:84% And then by adding over all k's we consider transitions of 00:05:25.840 --> 00:05:29.770 align:middle line:84% all types that lead us inside state j. 00:05:29.770 --> 00:05:34.240 align:middle line:84% So the probability of being at the j is the sum total of the 00:05:34.240 --> 00:05:38.690 align:middle line:84% probabilities of getting into j. 00:05:38.690 --> 00:05:42.800 align:middle line:84% What if we had multiple recurrent classes? 00:05:42.800 --> 00:05:48.645 align:middle line:84% So if we take this picture and change it to this. 00:05:48.645 --> 00:05:53.420 align:middle line:90% 00:05:53.420 --> 00:05:55.920 align:middle line:84% So here we got a secondary recurrent class. 00:05:55.920 --> 00:05:57.750 align:middle line:84% If you're here, you cannot get there. 00:05:57.750 --> 00:06:00.050 align:middle line:84% If you are here, you cannot get there. 00:06:00.050 --> 00:06:02.370 align:middle line:90% What happens in the long run? 00:06:02.370 --> 00:06:05.990 align:middle line:84% Well, in the long run, if you start from here you're going 00:06:05.990 --> 00:06:09.360 align:middle line:84% to make a transition eventually, either of this 00:06:09.360 --> 00:06:12.540 align:middle line:84% type and you would end up here, or you will make a 00:06:12.540 --> 00:06:16.210 align:middle line:84% transitional of that type and you will end up there. 00:06:16.210 --> 00:06:21.740 align:middle line:84% If you end up here, the long term statistics of your chain, 00:06:21.740 --> 00:06:24.860 align:middle line:84% that is, the probabilities of the different states, will be 00:06:24.860 --> 00:06:27.080 align:middle line:84% the steady state probabilities of this 00:06:27.080 --> 00:06:30.230 align:middle line:90% chain regarded in isolation. 00:06:30.230 --> 00:06:34.440 align:middle line:84% So you go ahead and you solve this system of equations just 00:06:34.440 --> 00:06:37.270 align:middle line:84% for this chain, and these will be your steady state 00:06:37.270 --> 00:06:38.560 align:middle line:90% probabilities. 00:06:38.560 --> 00:06:41.660 align:middle line:84% If you happened to get in here. 00:06:41.660 --> 00:06:45.650 align:middle line:84% If, on the other hand, it happens that you went there, 00:06:45.650 --> 00:06:49.710 align:middle line:84% given that event, then what happens in the long run has to 00:06:49.710 --> 00:06:53.390 align:middle line:84% do with just this chain running by itself. 00:06:53.390 --> 00:06:55.610 align:middle line:84% So you find the steady state probabilities 00:06:55.610 --> 00:06:57.340 align:middle line:90% inside that sub chain. 00:06:57.340 --> 00:06:59.860 align:middle line:84% So you solve the linear system, the steady state 00:06:59.860 --> 00:07:03.520 align:middle line:84% equations, for this chain separately and for that chain 00:07:03.520 --> 00:07:04.590 align:middle line:90% separately. 00:07:04.590 --> 00:07:08.160 align:middle line:84% If you happen to start inside here then the steady state 00:07:08.160 --> 00:07:12.650 align:middle line:84% probabilities for this sub chain are going to apply. 00:07:12.650 --> 00:07:16.420 align:middle line:84% Now of course this raises the question, if I start here, how 00:07:16.420 --> 00:07:19.500 align:middle line:84% do I know whether I'm going to get here or there? 00:07:19.500 --> 00:07:21.360 align:middle line:84% Well, you don't know, it's random. 00:07:21.360 --> 00:07:23.620 align:middle line:84% It may turn out that you get to here, it may turn out that 00:07:23.620 --> 00:07:24.760 align:middle line:90% you get there. 00:07:24.760 --> 00:07:27.790 align:middle line:84% So we will be interested in calculating the probabilities 00:07:27.790 --> 00:07:31.240 align:middle line:84% that eventually you end up here versus the probability 00:07:31.240 --> 00:07:33.570 align:middle line:84% that eventually you end up there. 00:07:33.570 --> 00:07:36.590 align:middle line:84% This is something that we're going to do towards the end of 00:07:36.590 --> 00:07:37.840 align:middle line:90% today's lecture. 00:07:37.840 --> 00:07:43.730 align:middle line:90% 00:07:43.730 --> 00:07:48.420 align:middle line:84% So, as a warm up, just to see how we interpret those steady 00:07:48.420 --> 00:07:52.710 align:middle line:84% state probabilities, let us look at our familiar example. 00:07:52.710 --> 00:07:54.680 align:middle line:84% This is a 2-state Markov chain. 00:07:54.680 --> 00:07:57.860 align:middle line:84% Last time we did write down the balance equations for this 00:07:57.860 --> 00:08:01.760 align:middle line:84% chain and we found the steady state probabilities to be 2/7 00:08:01.760 --> 00:08:04.250 align:middle line:90% and 5/7 respectively. 00:08:04.250 --> 00:08:07.040 align:middle line:84% So let us try to calculate some quantities. 00:08:07.040 --> 00:08:10.520 align:middle line:84% Suppose that you start at state 1, and you want to 00:08:10.520 --> 00:08:12.940 align:middle line:84% calculate to this particular probability. 00:08:12.940 --> 00:08:15.620 align:middle line:84% So since we're assuming that we're starting at state 1, 00:08:15.620 --> 00:08:19.010 align:middle line:84% essentially here we are conditioning on the initial 00:08:19.010 --> 00:08:21.790 align:middle line:90% state being equal to 1. 00:08:21.790 --> 00:08:23.920 align:middle line:84% Now the conditional probability of two things 00:08:23.920 --> 00:08:30.370 align:middle line:84% happening is the probability that the first thing happens. 00:08:30.370 --> 00:08:34.090 align:middle line:84% But we're living in the world where we said that the initial 00:08:34.090 --> 00:08:35.720 align:middle line:90% state was 1. 00:08:35.720 --> 00:08:40.000 align:middle line:84% And then given that this thing happened, the probability that 00:08:40.000 --> 00:08:43.530 align:middle line:90% the second thing happens. 00:08:43.530 --> 00:08:46.200 align:middle line:84% But again, we're talking about conditional probabilities 00:08:46.200 --> 00:08:50.150 align:middle line:84% given that the initial state was 1. 00:08:50.150 --> 00:08:53.080 align:middle line:90% So what is this quantity? 00:08:53.080 --> 00:08:57.040 align:middle line:84% This one is the transition probability from state 1 to 00:08:57.040 --> 00:09:00.800 align:middle line:90% state 1, so it's P11. 00:09:00.800 --> 00:09:04.730 align:middle line:84% How about the second probability? 00:09:04.730 --> 00:09:08.130 align:middle line:84% So given that you started at 1 and the next time you were at 00:09:08.130 --> 00:09:11.980 align:middle line:84% 1, what's the probability that at the time 100 you are at 1? 00:09:11.980 --> 00:09:15.160 align:middle line:84% Now because of the Markov property, if I tell you that 00:09:15.160 --> 00:09:17.520 align:middle line:84% at this time you are at 1, it doesn't 00:09:17.520 --> 00:09:19.190 align:middle line:90% matter how you get there. 00:09:19.190 --> 00:09:22.560 align:middle line:84% So this part of the conditioning doesn't matter. 00:09:22.560 --> 00:09:29.650 align:middle line:84% And what we have is the 99 step transition probability 00:09:29.650 --> 00:09:34.820 align:middle line:90% from state 1 to state 1. 00:09:34.820 --> 00:09:38.910 align:middle line:84% So the probability that you get to 1 and then 99 steps 00:09:38.910 --> 00:09:42.340 align:middle line:84% later you find yourself again at one is the probability that 00:09:42.340 --> 00:09:45.350 align:middle line:84% the first transition takes you to 1 times the probability 00:09:45.350 --> 00:09:51.810 align:middle line:84% that over the next 99 transitions starting from 1, 00:09:51.810 --> 00:09:55.930 align:middle line:84% after 99 steps you end up again at state 1. 00:09:55.930 --> 00:10:00.720 align:middle line:84% Now, 99 is possibly a big number, and so we approximate 00:10:00.720 --> 00:10:01.730 align:middle line:90% this quantity. 00:10:01.730 --> 00:10:05.810 align:middle line:84% We're using the steady state probability of state 1. 00:10:05.810 --> 00:10:08.110 align:middle line:84% And that gives us an approximation for this 00:10:08.110 --> 00:10:10.510 align:middle line:90% particular expression. 00:10:10.510 --> 00:10:14.790 align:middle line:84% We can do the same thing to calculate 00:10:14.790 --> 00:10:16.490 align:middle line:90% something of the same kind. 00:10:16.490 --> 00:10:19.260 align:middle line:90% So you start at state 1. 00:10:19.260 --> 00:10:23.230 align:middle line:84% What's the probability that 100 steps later you are again 00:10:23.230 --> 00:10:24.560 align:middle line:90% at state 1? 00:10:24.560 --> 00:10:26.000 align:middle line:90% So that's going to be P11-- 00:10:26.000 --> 00:10:28.970 align:middle line:90% 00:10:28.970 --> 00:10:29.890 align:middle line:90% not P -- 00:10:29.890 --> 00:10:31.140 align:middle line:90% R11. 00:10:31.140 --> 00:10:34.830 align:middle line:90% 00:10:34.830 --> 00:10:38.560 align:middle line:84% The 100 step transition probability that starting from 00:10:38.560 --> 00:10:43.980 align:middle line:84% 1 you get to 1, and then after you get to 1 at time 100 00:10:43.980 --> 00:10:46.120 align:middle line:84% what's the probability that the next time you find 00:10:46.120 --> 00:10:47.870 align:middle line:90% yourself at state 2? 00:10:47.870 --> 00:10:50.580 align:middle line:84% This is going to be the probability P12. 00:10:50.580 --> 00:10:55.320 align:middle line:84% And approximately, since 100 is a large number, this is 00:10:55.320 --> 00:10:57.890 align:middle line:90% approximately pi(1) times P12. 00:10:57.890 --> 00:11:06.230 align:middle line:90% 00:11:06.230 --> 00:11:07.070 align:middle line:90% OK. 00:11:07.070 --> 00:11:12.180 align:middle line:84% So that's how we can use steady state probabilities to 00:11:12.180 --> 00:11:14.190 align:middle line:90% make approximations. 00:11:14.190 --> 00:11:19.090 align:middle line:84% Or you could, for example, if you continue doing examples of 00:11:19.090 --> 00:11:23.250 align:middle line:84% this kind, you could ask for what's the probability that X 00:11:23.250 --> 00:11:32.170 align:middle line:84% at time 100 is 1, and also X at time 200 is equal to 1. 00:11:32.170 --> 00:11:36.200 align:middle line:84% Then this is going to be the transition probability from 1 00:11:36.200 --> 00:11:43.140 align:middle line:84% to 1 in 100 steps, and then over the next 100 steps from 1 00:11:43.140 --> 00:11:46.510 align:middle line:90% you get again to 1. 00:11:46.510 --> 00:11:48.750 align:middle line:90% And this is going to be 00:11:48.750 --> 00:11:51.400 align:middle line:90% approximately pi(1) times pi(1). 00:11:51.400 --> 00:11:57.550 align:middle line:90% 00:11:57.550 --> 00:12:01.090 align:middle line:84% So we approximate multi-step transition probabilities by 00:12:01.090 --> 00:12:05.120 align:middle line:84% the steady state probabilities when the number n that's 00:12:05.120 --> 00:12:07.510 align:middle line:90% involved in here is big. 00:12:07.510 --> 00:12:11.270 align:middle line:84% Now I said that's 99 or 100 is big. 00:12:11.270 --> 00:12:15.180 align:middle line:84% How do we know that it's big enough so that the limit has 00:12:15.180 --> 00:12:19.560 align:middle line:84% taken effect, and that our approximation is good? 00:12:19.560 --> 00:12:23.640 align:middle line:84% This has something to do with the time scale of our Markov 00:12:23.640 --> 00:12:27.460 align:middle line:84% chain, and by time scale, I mean how long does it take for 00:12:27.460 --> 00:12:29.740 align:middle line:84% the initial states to be forgotten. 00:12:29.740 --> 00:12:35.080 align:middle line:84% How long does it take for there to be enough randomness 00:12:35.080 --> 00:12:37.700 align:middle line:84% so that things sort of mix and it doesn't 00:12:37.700 --> 00:12:39.330 align:middle line:90% matter where you started? 00:12:39.330 --> 00:12:43.720 align:middle line:84% So if you look at this chain, it takes on the average, let's 00:12:43.720 --> 00:12:47.660 align:middle line:84% say 5 tries to make a transition of this kind. 00:12:47.660 --> 00:12:51.970 align:middle line:84% It takes on the average 2 tries for a transition of that 00:12:51.970 --> 00:12:53.990 align:middle line:90% kind to take place. 00:12:53.990 --> 00:12:58.840 align:middle line:84% So every 10 time steps or so there's a little bit of 00:12:58.840 --> 00:12:59.660 align:middle line:90% randomness. 00:12:59.660 --> 00:13:03.000 align:middle line:84% Over 100 times steps there's a lot of randomness, so you 00:13:03.000 --> 00:13:06.420 align:middle line:84% expect that the initial state will have been forgotten. 00:13:06.420 --> 00:13:07.820 align:middle line:90% It doesn't matter. 00:13:07.820 --> 00:13:10.540 align:middle line:84% There's enough mixing and randomness that happens over 00:13:10.540 --> 00:13:11.870 align:middle line:90% 100 time steps. 00:13:11.870 --> 00:13:16.100 align:middle line:84% And so this approximation is good. 00:13:16.100 --> 00:13:19.080 align:middle line:84% On the other hand, if the numbers were different, the 00:13:19.080 --> 00:13:20.880 align:middle line:84% story would have been different. 00:13:20.880 --> 00:13:26.620 align:middle line:84% Suppose that this number is 0.999 and that number is 00:13:26.620 --> 00:13:33.570 align:middle line:84% something like 0.998, so that this number becomes 0.002, and 00:13:33.570 --> 00:13:37.910 align:middle line:90% that number becomes 0.001. 00:13:37.910 --> 00:13:40.620 align:middle line:84% Suppose that the numbers were of this kind. 00:13:40.620 --> 00:13:44.760 align:middle line:84% How long does it take to forget the initial state? 00:13:44.760 --> 00:13:48.840 align:middle line:84% If I start here, there's a probability of 1 in 1,000 that 00:13:48.840 --> 00:13:50.770 align:middle line:84% next time I'm going to be there. 00:13:50.770 --> 00:13:53.840 align:middle line:84% So on the average it's going to take me about a thousand 00:13:53.840 --> 00:13:58.330 align:middle line:84% tries just to leave that state. 00:13:58.330 --> 00:14:03.390 align:middle line:84% So, over roughly a thousand time steps my initial state 00:14:03.390 --> 00:14:05.210 align:middle line:90% really does matter. 00:14:05.210 --> 00:14:08.550 align:middle line:84% If I tell you that you started here, you're pretty certain 00:14:08.550 --> 00:14:11.320 align:middle line:84% that, let's say over the next 100 time steps, you 00:14:11.320 --> 00:14:12.710 align:middle line:90% will still be here. 00:14:12.710 --> 00:14:15.580 align:middle line:84% So the initial state has a big effect. 00:14:15.580 --> 00:14:19.600 align:middle line:84% In this case we say that this Markov chain has a much slower 00:14:19.600 --> 00:14:21.030 align:middle line:90% time scale. 00:14:21.030 --> 00:14:25.350 align:middle line:84% It takes a much longer time to mix, it takes a much longer 00:14:25.350 --> 00:14:29.150 align:middle line:84% time for the initial state to be forgotten, and this means 00:14:29.150 --> 00:14:32.670 align:middle line:84% that we cannot do this kind of approximation if the number of 00:14:32.670 --> 00:14:34.870 align:middle line:90% steps is just 99. 00:14:34.870 --> 00:14:39.620 align:middle line:84% Here we might need n to be as large as, let's say, 10,000 or 00:14:39.620 --> 00:14:43.320 align:middle line:84% so before we can start using the approximation. 00:14:43.320 --> 00:14:46.210 align:middle line:84% So when one uses that approximation, one needs to 00:14:46.210 --> 00:14:50.830 align:middle line:84% have some sense of how quickly does the state move around and 00:14:50.830 --> 00:14:52.210 align:middle line:90% take that into account. 00:14:52.210 --> 00:14:56.100 align:middle line:84% So there's a whole sub-field that deals with estimating or 00:14:56.100 --> 00:15:00.030 align:middle line:84% figuring out how quickly different Markov chains mix, 00:15:00.030 --> 00:15:02.460 align:middle line:84% and that's the question of when can you apply those 00:15:02.460 --> 00:15:03.710 align:middle line:90% steady state approximations. 00:15:03.710 --> 00:15:06.860 align:middle line:90% 00:15:06.860 --> 00:15:12.180 align:middle line:84% So now let's get a little closer to the real world. 00:15:12.180 --> 00:15:15.590 align:middle line:84% We're going to talk about a famous problem that was posed, 00:15:15.590 --> 00:15:19.220 align:middle line:84% started, and solved by a Danish engineer 00:15:19.220 --> 00:15:21.110 align:middle line:90% by the name of Erlang. 00:15:21.110 --> 00:15:25.180 align:middle line:84% This is the same person whose name is given to the Erlang 00:15:25.180 --> 00:15:27.080 align:middle line:84% distribution that we saw in the context 00:15:27.080 --> 00:15:28.940 align:middle line:90% of the Poisson processes. 00:15:28.940 --> 00:15:33.950 align:middle line:84% So this was more than 100 years ago, when phones had 00:15:33.950 --> 00:15:36.060 align:middle line:90% just started existing. 00:15:36.060 --> 00:15:42.900 align:middle line:84% And he was trying to figure out what it would take to set 00:15:42.900 --> 00:15:47.260 align:middle line:84% up a phone system that how many lines should you set up 00:15:47.260 --> 00:15:49.920 align:middle line:84% for a community to be able to communicate 00:15:49.920 --> 00:15:53.020 align:middle line:90% to the outside world. 00:15:53.020 --> 00:15:54.350 align:middle line:90% So here's the story. 00:15:54.350 --> 00:15:57.730 align:middle line:84% You've got a village, and that village has a certain 00:15:57.730 --> 00:16:03.705 align:middle line:84% population, and you want to set up phone lines. 00:16:03.705 --> 00:16:06.290 align:middle line:90% 00:16:06.290 --> 00:16:09.720 align:middle line:84% So you want to set up a number of phone lines, let's say that 00:16:09.720 --> 00:16:13.520 align:middle line:84% number is B, to the outside world. 00:16:13.520 --> 00:16:17.710 align:middle line:90% 00:16:17.710 --> 00:16:19.690 align:middle line:84% And how do you want to do that? 00:16:19.690 --> 00:16:22.000 align:middle line:84% Well, you want B to be kind of small. 00:16:22.000 --> 00:16:24.130 align:middle line:84% You don't want to set up too many wires 00:16:24.130 --> 00:16:25.680 align:middle line:90% because that's expensive. 00:16:25.680 --> 00:16:30.300 align:middle line:84% On the other hand, you want to have enough wires so that if a 00:16:30.300 --> 00:16:33.000 align:middle line:84% reasonable number of people place phone calls 00:16:33.000 --> 00:16:38.000 align:middle line:84% simultaneously, they will all get a line and they will be 00:16:38.000 --> 00:16:39.600 align:middle line:90% able to talk. 00:16:39.600 --> 00:16:45.220 align:middle line:84% So if B is 10 and 12 people want to talk at the same time, 00:16:45.220 --> 00:16:48.490 align:middle line:84% then 2 of these people would get a busy signal, and that's 00:16:48.490 --> 00:16:50.220 align:middle line:90% not something that we like. 00:16:50.220 --> 00:16:53.010 align:middle line:84% We would like B to be large enough so that there's a 00:16:53.010 --> 00:16:57.240 align:middle line:84% substantial probability, that there's almost certainty that, 00:16:57.240 --> 00:17:00.120 align:middle line:84% under reasonable conditions, no one is going 00:17:00.120 --> 00:17:02.230 align:middle line:90% to get a busy signal. 00:17:02.230 --> 00:17:06.720 align:middle line:84% So how do we go about modeling a situation like this? 00:17:06.720 --> 00:17:09.890 align:middle line:84% Well, to set up a model you need two pieces, one is to 00:17:09.890 --> 00:17:17.000 align:middle line:84% describe how do phone calls get initiated, and once a 00:17:17.000 --> 00:17:21.819 align:middle line:84% phone call gets started, how long does it take until the 00:17:21.819 --> 00:17:24.530 align:middle line:90% phone call is terminated? 00:17:24.530 --> 00:17:27.530 align:middle line:84% So we're going to make the simplest assumptions possible. 00:17:27.530 --> 00:17:29.890 align:middle line:84% Let's assume that phone calls originate 00:17:29.890 --> 00:17:31.920 align:middle line:90% as a Poisson process. 00:17:31.920 --> 00:17:34.210 align:middle line:84% That is, out of that population people do not 00:17:34.210 --> 00:17:35.290 align:middle line:90% really coordinate. 00:17:35.290 --> 00:17:38.160 align:middle line:84% At completely random times, different people with decide 00:17:38.160 --> 00:17:39.740 align:middle line:90% to pick up the phone. 00:17:39.740 --> 00:17:42.320 align:middle line:84% There's no dependencies between different people, 00:17:42.320 --> 00:17:44.790 align:middle line:84% there's nothing special about different times, different 00:17:44.790 --> 00:17:46.340 align:middle line:90% times are independent. 00:17:46.340 --> 00:17:50.550 align:middle line:84% So a Poisson model is a reasonable way of modeling 00:17:50.550 --> 00:17:51.410 align:middle line:90% this situation. 00:17:51.410 --> 00:17:55.870 align:middle line:84% And it's going to be a Poisson process with some rate lambda. 00:17:55.870 --> 00:17:59.870 align:middle line:84% Now, the rate lambda would be easy to estimate in practice. 00:17:59.870 --> 00:18:02.760 align:middle line:84% You observe what happens in that village just over a 00:18:02.760 --> 00:18:06.050 align:middle line:84% couple of days, and you figure out what's the rate at which 00:18:06.050 --> 00:18:09.550 align:middle line:84% people attempt to place phone calls. 00:18:09.550 --> 00:18:11.930 align:middle line:84% Now, about phone calls themselves, we're going to 00:18:11.930 --> 00:18:15.120 align:middle line:84% make the assumption that the duration of a phone call is a 00:18:15.120 --> 00:18:18.580 align:middle line:84% random variable that has an exponential distribution with 00:18:18.580 --> 00:18:20.630 align:middle line:90% a certain parameter mu. 00:18:20.630 --> 00:18:24.400 align:middle line:84% So 1/mu is the mean duration of a phone call. 00:18:24.400 --> 00:18:27.570 align:middle line:84% So the mean duration, , again, is easy to estimate. 00:18:27.570 --> 00:18:31.240 align:middle line:84% You just observe what's happening, see on the average 00:18:31.240 --> 00:18:33.680 align:middle line:84% how long these phone calls are. 00:18:33.680 --> 00:18:36.590 align:middle line:84% Is the exponential assumption a good assumption? 00:18:36.590 --> 00:18:39.990 align:middle line:84% Well, it's means that most phone calls will be kind of 00:18:39.990 --> 00:18:43.180 align:middle line:84% short, but there's going to be a fraction of phone calls that 00:18:43.180 --> 00:18:45.640 align:middle line:84% are going to be larger, and then a very small fraction 00:18:45.640 --> 00:18:47.900 align:middle line:84% that are going to be even larger. 00:18:47.900 --> 00:18:49.880 align:middle line:90% So it sounds plausible. 00:18:49.880 --> 00:18:57.960 align:middle line:84% It's not exactly realistic, that is, phone calls that last 00:18:57.960 --> 00:19:02.250 align:middle line:84% short of 15 seconds are not that common. 00:19:02.250 --> 00:19:04.890 align:middle line:84% So either nothing happens or you have to say a few 00:19:04.890 --> 00:19:06.630 align:middle line:90% sentences and so on. 00:19:06.630 --> 00:19:10.200 align:middle line:84% Also, back into the days when people used to connect to the 00:19:10.200 --> 00:19:15.430 align:middle line:84% internet using dial up modems, that assumption was completely 00:19:15.430 --> 00:19:20.390 align:middle line:84% destroyed, because people would dial up and then keep 00:19:20.390 --> 00:19:25.730 align:middle line:84% their phone line busy for a few hours, if the phone call 00:19:25.730 --> 00:19:26.930 align:middle line:90% was a free one. 00:19:26.930 --> 00:19:30.390 align:middle line:84% So at those times the exponential assumption for the 00:19:30.390 --> 00:19:33.260 align:middle line:84% phone call duration was completely destroyed. 00:19:33.260 --> 00:19:36.430 align:middle line:84% But leaving that detail aside, it's sort of a reasonable 00:19:36.430 --> 00:19:41.000 align:middle line:84% assumption to just get started with this problem. 00:19:41.000 --> 00:19:43.790 align:middle line:84% All right, so now that we have those assumptions, let's try 00:19:43.790 --> 00:19:46.510 align:middle line:90% to come up with the model. 00:19:46.510 --> 00:19:49.470 align:middle line:84% And we're going to set up a Markov process model. 00:19:49.470 --> 00:19:52.990 align:middle line:84% Now the Poisson process runs in continuous time, and call 00:19:52.990 --> 00:19:56.280 align:middle line:84% durations being exponential random variables also are 00:19:56.280 --> 00:19:59.330 align:middle line:84% continuous random variables, so it seems that we are in a 00:19:59.330 --> 00:20:01.020 align:middle line:90% continuous time universe. 00:20:01.020 --> 00:20:03.610 align:middle line:84% But we have only started Markov chains for the 00:20:03.610 --> 00:20:05.310 align:middle line:90% discrete time case. 00:20:05.310 --> 00:20:07.370 align:middle line:90% What are we going to do? 00:20:07.370 --> 00:20:10.470 align:middle line:84% We can either develop the theory of continuous time 00:20:10.470 --> 00:20:13.600 align:middle line:84% Markov chains, which is possible. 00:20:13.600 --> 00:20:16.300 align:middle line:84% But we are not going to do that in this class. 00:20:16.300 --> 00:20:20.460 align:middle line:84% Or we can discretize time and work with a 00:20:20.460 --> 00:20:22.070 align:middle line:90% discrete time model. 00:20:22.070 --> 00:20:24.850 align:middle line:84% So we're going to discretize time in the familiar way, the 00:20:24.850 --> 00:20:27.350 align:middle line:84% way we did it when we started the Poisson process. 00:20:27.350 --> 00:20:31.270 align:middle line:84% We're going to take the time axis and split it into little 00:20:31.270 --> 00:20:35.130 align:middle line:84% discrete mini slots, where every mini slot 00:20:35.130 --> 00:20:36.880 align:middle line:90% has a duration delta. 00:20:36.880 --> 00:20:41.520 align:middle line:84% So this delta is supposed to be a very small number. 00:20:41.520 --> 00:20:44.760 align:middle line:84% So what is the state of the system? 00:20:44.760 --> 00:20:47.290 align:middle line:84% So, you look at the situation in the system at some 00:20:47.290 --> 00:20:51.340 align:middle line:84% particular time and I ask you what is going on right now, 00:20:51.340 --> 00:20:53.680 align:middle line:84% what's the information you would tell me? 00:20:53.680 --> 00:20:56.990 align:middle line:84% Well, you would tell me that right now out of these capital 00:20:56.990 --> 00:21:02.160 align:middle line:84% B lines, 10 of them are busy, or 12 of them are busy. 00:21:02.160 --> 00:21:04.740 align:middle line:84% That describes the state of the system, that tells me 00:21:04.740 --> 00:21:06.560 align:middle line:84% what's happening at this point. 00:21:06.560 --> 00:21:10.800 align:middle line:84% So we set up our states base by being the numbers from 0 to 00:21:10.800 --> 00:21:15.510 align:middle line:84% B. 0 corresponds to a state in which all the phone lines are 00:21:15.510 --> 00:21:17.260 align:middle line:90% free, no one is talking. 00:21:17.260 --> 00:21:19.590 align:middle line:84% Capital B corresponds to a case where all the 00:21:19.590 --> 00:21:21.580 align:middle line:90% phone lines are busy. 00:21:21.580 --> 00:21:24.020 align:middle line:84% And then you've got states in between. 00:21:24.020 --> 00:21:28.050 align:middle line:84% And now let's look at the transition probabilities. 00:21:28.050 --> 00:21:32.580 align:middle line:84% Suppose that right so now we have i-1 lines that are busy. 00:21:32.580 --> 00:21:36.280 align:middle line:90% 00:21:36.280 --> 00:21:38.380 align:middle line:90% Or maybe, let me look here. 00:21:38.380 --> 00:21:41.460 align:middle line:84% Suppose that there's i lines that are busy. 00:21:41.460 --> 00:21:44.770 align:middle line:90% What can happen the next time? 00:21:44.770 --> 00:21:48.180 align:middle line:84% What can happen is that the new phone call gets placed, in 00:21:48.180 --> 00:21:53.670 align:middle line:84% which case my state moves up by 1, or an existing call 00:21:53.670 --> 00:21:59.060 align:middle line:84% terminates, in which case my state goes down by 1, or none 00:21:59.060 --> 00:22:03.530 align:middle line:84% of the two happens, in which case I stay at the same state. 00:22:03.530 --> 00:22:06.930 align:middle line:84% Well, it's also possible that the phone call gets terminated 00:22:06.930 --> 00:22:10.510 align:middle line:84% and a new phone call gets placed sort of simultaneously. 00:22:10.510 --> 00:22:14.080 align:middle line:84% But when you take your time slots to be very, very small, 00:22:14.080 --> 00:22:16.970 align:middle line:84% this is going to have a negligible probability order 00:22:16.970 --> 00:22:19.850 align:middle line:84% of delta squared, so we ignore this. 00:22:19.850 --> 00:22:22.190 align:middle line:84% So what's the probability of an upwards transition? 00:22:22.190 --> 00:22:25.160 align:middle line:84% That's the probability that the Poisson process records an 00:22:25.160 --> 00:22:29.250 align:middle line:84% arrival during a mini slot of duration delta. 00:22:29.250 --> 00:22:31.270 align:middle line:84% By the definition of the Poisson process, the 00:22:31.270 --> 00:22:35.550 align:middle line:84% probability of this happening is just lambda delta. 00:22:35.550 --> 00:22:39.200 align:middle line:84% So each one of these upwards transitions has the same 00:22:39.200 --> 00:22:41.000 align:middle line:90% probability of lambda delta. 00:22:41.000 --> 00:22:45.220 align:middle line:84% So you've got lambda deltas everywhere in this diagram. 00:22:45.220 --> 00:22:49.180 align:middle line:84% How about, now, phone call terminations? 00:22:49.180 --> 00:22:53.630 align:middle line:84% If you had the single call that was active, so if you 00:22:53.630 --> 00:22:56.510 align:middle line:84% were here, what's the probability that the phone 00:22:56.510 --> 00:22:57.820 align:middle line:90% call terminates? 00:22:57.820 --> 00:23:00.840 align:middle line:84% So the phone call has an exponential duration with 00:23:00.840 --> 00:23:02.820 align:middle line:90% parameter mu. 00:23:02.820 --> 00:23:05.860 align:middle line:84% And we discussed before that an exponential random variable 00:23:05.860 --> 00:23:08.940 align:middle line:84% can be thought of as the first arrival time 00:23:08.940 --> 00:23:10.780 align:middle line:90% in a Poisson process. 00:23:10.780 --> 00:23:14.750 align:middle line:84% So the probability that you get this event to happen over 00:23:14.750 --> 00:23:18.820 align:middle line:84% a delta time interval is just mu times delta. 00:23:18.820 --> 00:23:22.280 align:middle line:84% So if you have a single phone call that's happening right 00:23:22.280 --> 00:23:24.780 align:middle line:84% now, with probability mu times delta, that 00:23:24.780 --> 00:23:27.070 align:middle line:90% call is going to terminate. 00:23:27.070 --> 00:23:30.020 align:middle line:84% But suppose that we have i phone calls that 00:23:30.020 --> 00:23:31.750 align:middle line:90% are currently active. 00:23:31.750 --> 00:23:35.010 align:middle line:84% Each one of them has a probability of mu delta, of 00:23:35.010 --> 00:23:39.200 align:middle line:84% terminating, but collectively the probability that one of 00:23:39.200 --> 00:23:45.850 align:middle line:84% them terminates becomes i times mu delta. 00:23:45.850 --> 00:23:48.570 align:middle line:84% So that's because you get the mu delta contribution -- 00:23:48.570 --> 00:23:51.540 align:middle line:84% the probability of termination from each one of the different 00:23:51.540 --> 00:23:54.410 align:middle line:90% phone calls. 00:23:54.410 --> 00:23:58.290 align:middle line:84% OK, now this is an approximate calculation, because it 00:23:58.290 --> 00:24:01.850 align:middle line:84% ignores the possibility that two phone calls terminate at 00:24:01.850 --> 00:24:03.110 align:middle line:90% the same time. 00:24:03.110 --> 00:24:09.130 align:middle line:84% Again, the way to think of why this is the correct rate, when 00:24:09.130 --> 00:24:13.610 align:middle line:84% you have i phone calls that are simultaneously running and 00:24:13.610 --> 00:24:17.250 align:middle line:84% waiting for one of them to terminate, this is like having 00:24:17.250 --> 00:24:21.170 align:middle line:84% i separate Poisson processes that are running in parallel, 00:24:21.170 --> 00:24:23.720 align:middle line:84% and you ask for the probability that one of those 00:24:23.720 --> 00:24:25.980 align:middle line:90% processes records an event. 00:24:25.980 --> 00:24:28.470 align:middle line:84% Now when you put all those process together, it's like 00:24:28.470 --> 00:24:32.970 align:middle line:84% having a Poisson process with total rate i times mu, and so 00:24:32.970 --> 00:24:36.210 align:middle line:84% i times mu delta is the overall probability that 00:24:36.210 --> 00:24:39.580 align:middle line:84% something happens in terms of phone call terminations at 00:24:39.580 --> 00:24:40.760 align:middle line:90% those times. 00:24:40.760 --> 00:24:43.470 align:middle line:84% So in any case, this is the transition probability for 00:24:43.470 --> 00:24:46.280 align:middle line:90% downwards transitions. 00:24:46.280 --> 00:24:49.190 align:middle line:84% Now that we've got this, we can analyze this chain. 00:24:49.190 --> 00:24:53.220 align:middle line:84% This chain has the birth death form that we discussed towards 00:24:53.220 --> 00:24:54.770 align:middle line:90% the end of last lecture. 00:24:54.770 --> 00:24:58.360 align:middle line:84% And for birth death chains, it's easy to write it out to 00:24:58.360 --> 00:25:00.710 align:middle line:84% find the steady state probabilities. 00:25:00.710 --> 00:25:03.360 align:middle line:84% Instead of writing down the balance equations in the 00:25:03.360 --> 00:25:07.950 align:middle line:84% general form, we think in terms of a conservation of 00:25:07.950 --> 00:25:10.630 align:middle line:84% probabilities or of transitions by looking at what 00:25:10.630 --> 00:25:14.640 align:middle line:84% happens across a particular cut in this diagram. 00:25:14.640 --> 00:25:18.590 align:middle line:84% Number of transitions in the chain that cross from here to 00:25:18.590 --> 00:25:21.310 align:middle line:84% here has to be approximately equal to the number of 00:25:21.310 --> 00:25:24.850 align:middle line:84% transitions from here to there because whatever comes up must 00:25:24.850 --> 00:25:27.520 align:middle line:84% come down and then come up and so on. 00:25:27.520 --> 00:25:31.410 align:middle line:84% So the frequency with which transitions of this kind are 00:25:31.410 --> 00:25:34.070 align:middle line:84% observed has to be the same as the frequency of transitions 00:25:34.070 --> 00:25:35.690 align:middle line:90% of this kind. 00:25:35.690 --> 00:25:38.400 align:middle line:84% What's the frequency of how often the transitions of this 00:25:38.400 --> 00:25:40.280 align:middle line:90% kind happen? 00:25:40.280 --> 00:25:45.350 align:middle line:84% And by frequency I mean quite percentage of the mini slots 00:25:45.350 --> 00:25:48.120 align:middle line:84% involve a transition of this kind? 00:25:48.120 --> 00:25:51.420 align:middle line:84% Well, for a transition of that kind to happen we need to be 00:25:51.420 --> 00:25:55.730 align:middle line:84% at states i-1, which happens this much of the time. 00:25:55.730 --> 00:25:59.360 align:middle line:84% And then the probability lambda delta that the 00:25:59.360 --> 00:26:01.360 align:middle line:90% transition is of this kind. 00:26:01.360 --> 00:26:06.040 align:middle line:84% So the frequency of transitions of with which this 00:26:06.040 --> 00:26:11.460 align:middle line:84% kind of transition is observed is lambda delta times pi(i-1). 00:26:11.460 --> 00:26:17.680 align:middle line:84% This is the fraction of time steps at which a transition 00:26:17.680 --> 00:26:20.180 align:middle line:84% from specifically this state to specifically 00:26:20.180 --> 00:26:22.120 align:middle line:90% that state are observed. 00:26:22.120 --> 00:26:24.880 align:middle line:84% This has to be the same as the frequency with which 00:26:24.880 --> 00:26:28.360 align:middle line:84% transitions of that kind are observed, and that frequency 00:26:28.360 --> 00:26:32.310 align:middle line:84% is going to be i mu delta times pi(i), and then we 00:26:32.310 --> 00:26:37.870 align:middle line:84% cancel the deltas, and we are left with this equation here. 00:26:37.870 --> 00:26:43.260 align:middle line:84% So this equation expresses pi(i) in terms of pi(i-1). 00:26:43.260 --> 00:26:47.020 align:middle line:84% So if we knew pi(0) we can use that equation 00:26:47.020 --> 00:26:48.690 align:middle line:90% to determine pi(1). 00:26:48.690 --> 00:26:52.320 align:middle line:84% Once we know pi(1), we can use that equation to determine 00:26:52.320 --> 00:26:55.490 align:middle line:84% pi(2), and so on, you keep going. 00:26:55.490 --> 00:26:59.890 align:middle line:84% And the general formula that comes out of this, I will not 00:26:59.890 --> 00:27:02.350 align:middle line:84% do the algebra, it's a straightforward substitution, 00:27:02.350 --> 00:27:04.830 align:middle line:84% you find that pi(i), the steady state probability of 00:27:04.830 --> 00:27:08.650 align:middle line:84% state i is given by this expression, which involves the 00:27:08.650 --> 00:27:12.020 align:middle line:90% pi(0) from which we started. 00:27:12.020 --> 00:27:13.500 align:middle line:90% Now what is pi(0)? 00:27:13.500 --> 00:27:17.420 align:middle line:84% Well, we don't know yet, but we can find it by using the 00:27:17.420 --> 00:27:19.520 align:middle line:90% normalization equation. 00:27:19.520 --> 00:27:22.680 align:middle line:84% The sum of pi(i) has to be equal to 1. 00:27:22.680 --> 00:27:26.430 align:middle line:84% So the sum of all of those numbers has to be equal to 1. 00:27:26.430 --> 00:27:30.370 align:middle line:84% And the only way that this can happen is by setting pi(0) to 00:27:30.370 --> 00:27:33.320 align:middle line:84% be equal to that particular number. 00:27:33.320 --> 00:27:38.970 align:middle line:84% So if I tell you the value of capital B, you can set up this 00:27:38.970 --> 00:27:43.690 align:middle line:84% Markov chain, you can calculate pi(0), and then you 00:27:43.690 --> 00:27:47.790 align:middle line:84% can calculate pi(i), and so you know what fraction, you 00:27:47.790 --> 00:27:51.150 align:middle line:84% know the steady state probabilities of this chain, 00:27:51.150 --> 00:27:53.920 align:middle line:84% so you can answer the question. 00:27:53.920 --> 00:27:57.240 align:middle line:84% If I drop in at a random time, how likely is it that I'm 00:27:57.240 --> 00:28:00.200 align:middle line:84% going to find the states to be here, or the 00:28:00.200 --> 00:28:01.930 align:middle line:90% states to be there? 00:28:01.930 --> 00:28:03.790 align:middle line:84% So the steady state probabilities are 00:28:03.790 --> 00:28:07.390 align:middle line:84% probabilities, but we also interpret them as frequencies. 00:28:07.390 --> 00:28:12.000 align:middle line:84% So once I find pi(i), it also tells me what fraction of the 00:28:12.000 --> 00:28:16.870 align:middle line:90% time is the state equal to i. 00:28:16.870 --> 00:28:20.030 align:middle line:84% And you can answer that question for every possible i. 00:28:20.030 --> 00:28:22.180 align:middle line:84% Now, why did we do this exercise? 00:28:22.180 --> 00:28:25.240 align:middle line:84% We're interested in the probability of 00:28:25.240 --> 00:28:27.660 align:middle line:90% the system is busy. 00:28:27.660 --> 00:28:31.960 align:middle line:84% So if a person, a new phone call gets placed, it just 00:28:31.960 --> 00:28:33.020 align:middle line:90% drops out of the sky. 00:28:33.020 --> 00:28:37.560 align:middle line:84% According to that Poisson process, that new phone call 00:28:37.560 --> 00:28:41.770 align:middle line:84% is going to find the system at a random state. 00:28:41.770 --> 00:28:45.290 align:middle line:84% That random state is described in steady state by the 00:28:45.290 --> 00:28:47.440 align:middle line:90% probabilities pi(i)'s. 00:28:47.440 --> 00:28:51.940 align:middle line:84% And the probability that you find the system to be busy is 00:28:51.940 --> 00:28:55.110 align:middle line:84% the probability that when you drop in the state happens to 00:28:55.110 --> 00:29:01.380 align:middle line:84% be that particular number B. So i sub b is the probability 00:29:01.380 --> 00:29:02.680 align:middle line:90% of being busy. 00:29:02.680 --> 00:29:06.496 align:middle line:90% 00:29:06.496 --> 00:29:09.470 align:middle line:84% And this is the probability that you would like to be 00:29:09.470 --> 00:29:11.970 align:middle line:84% small in a well engineered system. 00:29:11.970 --> 00:29:16.890 align:middle line:84% So you ask the question, how should, given my lambda and 00:29:16.890 --> 00:29:22.570 align:middle line:84% mu, my design question is to determine capital B the number 00:29:22.570 --> 00:29:25.720 align:middle line:84% of phone lines so that this number is small. 00:29:25.720 --> 00:29:31.290 align:middle line:90% 00:29:31.290 --> 00:29:36.040 align:middle line:84% Could we have done, could we figure out a good value for B 00:29:36.040 --> 00:29:39.670 align:middle line:84% by doing a back of the envelope calculation? 00:29:39.670 --> 00:29:44.410 align:middle line:84% Let's suppose that lambda is 30 and that mu is 1/3. 00:29:44.410 --> 00:29:49.190 align:middle line:90% 00:29:49.190 --> 00:29:53.640 align:middle line:84% So I guess that's, let us these rates to 00:29:53.640 --> 00:29:55.135 align:middle line:90% be calls per minute. 00:29:55.135 --> 00:29:58.400 align:middle line:90% 00:29:58.400 --> 00:30:03.100 align:middle line:84% And this mu, again, is a rate per minute. 00:30:03.100 --> 00:30:07.860 align:middle line:84% Again, the units of mu are going to be calls per minute. 00:30:07.860 --> 00:30:11.310 align:middle line:84% So since our time unit is minutes, the mean duration of 00:30:11.310 --> 00:30:13.660 align:middle line:90% calls is 1/mu minutes. 00:30:13.660 --> 00:30:16.930 align:middle line:84% So a typical call, or on the average a 00:30:16.930 --> 00:30:19.185 align:middle line:90% call lasts for 3 minutes. 00:30:19.185 --> 00:30:23.900 align:middle line:90% 00:30:23.900 --> 00:30:27.860 align:middle line:84% So you get 30 calls per minute. 00:30:27.860 --> 00:30:31.540 align:middle line:84% Each call lasts for 3 minutes on the average. 00:30:31.540 --> 00:30:36.290 align:middle line:84% So on the average, if B was infinite, 00:30:36.290 --> 00:30:38.100 align:middle line:90% every call goes through. 00:30:38.100 --> 00:30:42.750 align:middle line:84% How many calls would be active on the average? 00:30:42.750 --> 00:30:44.570 align:middle line:90% So you get 30 per minute. 00:30:44.570 --> 00:30:48.490 align:middle line:84% If a call lasted exactly 1 minute, then at any time you 00:30:48.490 --> 00:30:51.210 align:middle line:84% would have 30 calls being active. 00:30:51.210 --> 00:30:54.850 align:middle line:84% Now a call lasts on the average for 3 minutes. 00:30:54.850 --> 00:30:58.120 align:middle line:84% So during each minute you generate 90 00:30:58.120 --> 00:31:00.460 align:middle line:90% minutes of talking time. 00:31:00.460 --> 00:31:05.400 align:middle line:84% So by thinking in terms of averages you would expect that 00:31:05.400 --> 00:31:08.070 align:middle line:84% at any time there would be about 90 00:31:08.070 --> 00:31:10.540 align:middle line:90% calls that are active. 00:31:10.540 --> 00:31:14.210 align:middle line:84% And if 90 calls are active on the average, you could say OK, 00:31:14.210 --> 00:31:18.600 align:middle line:84% I'm going to set up my capital B to be 90. 00:31:18.600 --> 00:31:22.250 align:middle line:84% But that's not very good, because if the average number 00:31:22.250 --> 00:31:25.950 align:middle line:84% of phone calls that want to happen is if the average 00:31:25.950 --> 00:31:29.900 align:middle line:84% number is 90, sometimes you're going to have 85, sometimes 00:31:29.900 --> 00:31:31.460 align:middle line:90% you will have 95. 00:31:31.460 --> 00:31:34.300 align:middle line:84% And to be sure that the phone calls will go through you 00:31:34.300 --> 00:31:37.790 align:middle line:84% probably want to choose your capital B to be a number a 00:31:37.790 --> 00:31:40.460 align:middle line:90% little larger than 90. 00:31:40.460 --> 00:31:42.600 align:middle line:90% How much larger than 90? 00:31:42.600 --> 00:31:47.070 align:middle line:84% Well, this is a question that you can answer numerically. 00:31:47.070 --> 00:31:50.280 align:middle line:84% So you go through the following procedure. 00:31:50.280 --> 00:31:55.590 align:middle line:84% I tried different values of capital B. For any given value 00:31:55.590 --> 00:31:59.230 align:middle line:84% of capital B, I do this numerical calculation, I find 00:31:59.230 --> 00:32:03.530 align:middle line:84% the probability that the system is busy, and then I ask 00:32:03.530 --> 00:32:07.060 align:middle line:84% what's the value of B that makes my probability of being 00:32:07.060 --> 00:32:10.510 align:middle line:84% busy to be, let's say, roughly 1 %. 00:32:10.510 --> 00:32:13.110 align:middle line:84% And if you do that calculation with the parameters that they 00:32:13.110 --> 00:32:18.620 align:middle line:84% gave you, you find that B would be something like 106. 00:32:18.620 --> 00:32:21.490 align:middle line:84% So with the parameters they gave where you have, on the 00:32:21.490 --> 00:32:26.550 align:middle line:84% average, 90 phone calls being active, you actually need some 00:32:26.550 --> 00:32:30.410 align:middle line:84% margin to protect against the [?] fluctuation, if suddenly 00:32:30.410 --> 00:32:34.010 align:middle line:84% by chance more people want to talk, and if you want to have 00:32:34.010 --> 00:32:37.890 align:middle line:84% a good guarantee that an incoming person will have a 00:32:37.890 --> 00:32:40.860 align:middle line:84% very small probability of finding a busy system, then 00:32:40.860 --> 00:32:44.860 align:middle line:84% you will need about 106 phone lines. 00:32:44.860 --> 00:32:49.740 align:middle line:84% So that's the calculation and the argument that the Erlang 00:32:49.740 --> 00:32:52.250 align:middle line:90% went through a long time ago. 00:32:52.250 --> 00:32:55.610 align:middle line:84% It's actually interesting that Erlang did this calculation 00:32:55.610 --> 00:32:58.590 align:middle line:84% before Markov chains were invented. 00:32:58.590 --> 00:33:02.250 align:middle line:84% So Markov's work, and the beginning of work on Markov 00:33:02.250 --> 00:33:06.770 align:middle line:84% chains, happens about 10-15 years after Erlang. 00:33:06.770 --> 00:33:10.180 align:middle line:84% So obviously he didn't call that a Markov chain. 00:33:10.180 --> 00:33:13.350 align:middle line:84% But it was something that he could study from first 00:33:13.350 --> 00:33:15.900 align:middle line:90% principles. 00:33:15.900 --> 00:33:19.160 align:middle line:84% So this is a pretty useful thing. 00:33:19.160 --> 00:33:24.530 align:middle line:84% These probabilities that come out of that model, at least in 00:33:24.530 --> 00:33:28.230 align:middle line:84% the old days, they would all be very well tabulated in 00:33:28.230 --> 00:33:32.670 align:middle line:84% handbooks that every decent phone company engineer would 00:33:32.670 --> 00:33:34.530 align:middle line:90% sort of have with them. 00:33:34.530 --> 00:33:38.890 align:middle line:84% So this is about as practical as it gets. 00:33:38.890 --> 00:33:42.340 align:middle line:84% It's one of the sort of standard real world 00:33:42.340 --> 00:33:43.810 align:middle line:90% applications of Markov chains. 00:33:43.810 --> 00:33:47.040 align:middle line:90% 00:33:47.040 --> 00:33:53.950 align:middle line:84% So now to close our subjects, we're going to consider a 00:33:53.950 --> 00:33:57.310 align:middle line:84% couple of new skills and see how we can calculate the few 00:33:57.310 --> 00:33:59.660 align:middle line:84% additional interesting quantities that have to do 00:33:59.660 --> 00:34:01.340 align:middle line:90% with the Markov chain. 00:34:01.340 --> 00:34:04.800 align:middle line:84% So the problem we're going to deal with here is the one I 00:34:04.800 --> 00:34:07.630 align:middle line:84% hinted that when I was talking about this picture. 00:34:07.630 --> 00:34:09.870 align:middle line:84% You start at a transient state, you're going to 00:34:09.870 --> 00:34:12.239 align:middle line:84% eventually end up here or there. 00:34:12.239 --> 00:34:16.389 align:middle line:84% We want to find the probabilities of one option of 00:34:16.389 --> 00:34:19.210 align:middle line:84% the two happening or the other happening. 00:34:19.210 --> 00:34:23.880 align:middle line:84% So in this picture we have a class of 00:34:23.880 --> 00:34:25.680 align:middle line:90% states that's are transient. 00:34:25.680 --> 00:34:29.130 align:middle line:90% 00:34:29.130 --> 00:34:34.219 align:middle line:84% These are transient because you're going to move around 00:34:34.219 --> 00:34:37.170 align:middle line:84% those states, but there's a transition that you can make, 00:34:37.170 --> 00:34:40.260 align:middle line:84% and you go to a state from which you cannot escape 00:34:40.260 --> 00:34:41.560 align:middle line:90% afterwards. 00:34:41.560 --> 00:34:43.710 align:middle line:84% Are you going to end up here or are you 00:34:43.710 --> 00:34:45.370 align:middle line:90% going to end up there? 00:34:45.370 --> 00:34:46.130 align:middle line:90% You don't know. 00:34:46.130 --> 00:34:47.280 align:middle line:90% It's random. 00:34:47.280 --> 00:34:50.940 align:middle line:84% Let's try to calculate the probability that you 00:34:50.940 --> 00:34:54.800 align:middle line:90% end up at state 4. 00:34:54.800 --> 00:34:59.990 align:middle line:84% Now, the probability that you end up at state 4 will depend 00:34:59.990 --> 00:35:02.200 align:middle line:90% on where you start. 00:35:02.200 --> 00:35:06.170 align:middle line:84% Because if you start here, you probably have more chances of 00:35:06.170 --> 00:35:09.720 align:middle line:84% getting to 4 because you get that chance immediately, 00:35:09.720 --> 00:35:12.390 align:middle line:84% whereas if you start here there's more chances that 00:35:12.390 --> 00:35:16.260 align:middle line:84% you're going to escape that way because it kind of takes 00:35:16.260 --> 00:35:17.770 align:middle line:90% you time to get there. 00:35:17.770 --> 00:35:20.810 align:middle line:84% It's more likely that you exit right away. 00:35:20.810 --> 00:35:26.040 align:middle line:84% So the probability of exiting and ending up at state 4 will 00:35:26.040 --> 00:35:28.190 align:middle line:90% depend on the initial state. 00:35:28.190 --> 00:35:33.900 align:middle line:84% That's why when we talk about these absorption probability 00:35:33.900 --> 00:35:38.270 align:middle line:84% we include an index i that tells us what the 00:35:38.270 --> 00:35:40.310 align:middle line:90% initial state is. 00:35:40.310 --> 00:35:44.350 align:middle line:84% And we want to find this absorption probability, the 00:35:44.350 --> 00:35:46.820 align:middle line:84% probability that we end up here for the 00:35:46.820 --> 00:35:48.660 align:middle line:90% different initial states. 00:35:48.660 --> 00:35:52.100 align:middle line:84% Now for some initial states this is very easy to answer. 00:35:52.100 --> 00:35:55.280 align:middle line:84% If you start at state 4, what's the probability that 00:35:55.280 --> 00:35:59.160 align:middle line:84% eventually you end up in this part of the chain? 00:35:59.160 --> 00:35:59.820 align:middle line:90% It's 1. 00:35:59.820 --> 00:36:02.585 align:middle line:84% You're certain to be there, that's where you started. 00:36:02.585 --> 00:36:06.140 align:middle line:84% If you start at state 5, what's the probability that 00:36:06.140 --> 00:36:08.880 align:middle line:84% you end up eventually at state 4? 00:36:08.880 --> 00:36:12.580 align:middle line:84% It's probability 0, there's no way to get there. 00:36:12.580 --> 00:36:17.930 align:middle line:84% Now, how about if you start at a state like state 2? 00:36:17.930 --> 00:36:20.840 align:middle line:90% 00:36:20.840 --> 00:36:26.350 align:middle line:84% If you start at state 2 then there's a few different things 00:36:26.350 --> 00:36:27.840 align:middle line:90% that can happen. 00:36:27.840 --> 00:36:32.900 align:middle line:84% Either you end up at state 4 right away and this happens 00:36:32.900 --> 00:36:41.150 align:middle line:84% with probability 0.2, or you end up at state 1, and this 00:36:41.150 --> 00:36:46.730 align:middle line:90% happens with probability 0.6. 00:36:46.730 --> 00:36:50.200 align:middle line:84% So if you end up at state 4, you are done. 00:36:50.200 --> 00:36:51.540 align:middle line:90% We are there. 00:36:51.540 --> 00:36:56.850 align:middle line:84% If you end up at state 1, then what? 00:36:56.850 --> 00:37:00.980 align:middle line:84% Starting from state 1 there's two possibilities. 00:37:00.980 --> 00:37:05.360 align:middle line:84% Either eventually you're going to end up at state 4, or 00:37:05.360 --> 00:37:10.010 align:middle line:84% eventually you're going to end up at state 5. 00:37:10.010 --> 00:37:14.310 align:middle line:84% What's the probability of this happening? 00:37:14.310 --> 00:37:19.680 align:middle line:84% We don't know what it is, but it's what we defined to be a1. 00:37:19.680 --> 00:37:21.130 align:middle line:90% This is the probability -- 00:37:21.130 --> 00:37:22.650 align:middle line:90% a1 is the probability -- 00:37:22.650 --> 00:37:26.650 align:middle line:84% that eventually you settle in state 4 given that the initial 00:37:26.650 --> 00:37:28.090 align:middle line:90% state was 1. 00:37:28.090 --> 00:37:30.580 align:middle line:90% So this probability is a1. 00:37:30.580 --> 00:37:34.510 align:middle line:84% So our event of interest can happen in two ways. 00:37:34.510 --> 00:37:38.350 align:middle line:84% Either I go there directly, or I go here 00:37:38.350 --> 00:37:39.710 align:middle line:90% with probability 0.6. 00:37:39.710 --> 00:37:43.730 align:middle line:84% And given that I go there, eventually I end up at state 00:37:43.730 --> 00:37:46.540 align:middle line:84% 4, which happens with probability a1. 00:37:46.540 --> 00:37:52.140 align:middle line:84% So the total probability of ending up at state 4 is going 00:37:52.140 --> 00:37:54.660 align:middle line:84% to be the sum of the probabilities of the different 00:37:54.660 --> 00:37:57.230 align:middle line:84% ways that this event can happen. 00:37:57.230 --> 00:38:04.250 align:middle line:84% So our equation, in this case, is going to be, that's a2, is 00:38:04.250 --> 00:38:07.420 align:middle line:84% going to be 0.2 (that's the probability of going there 00:38:07.420 --> 00:38:11.210 align:middle line:84% directly) plus with probability 0.8 I end up at 00:38:11.210 --> 00:38:17.160 align:middle line:84% state 1, and then from state 1 I will end up at state 4 with 00:38:17.160 --> 00:38:19.320 align:middle line:90% probability a1. 00:38:19.320 --> 00:38:25.330 align:middle line:84% So this is one particular equation that we've got for 00:38:25.330 --> 00:38:28.560 align:middle line:84% what happens if we start from this state. 00:38:28.560 --> 00:38:32.450 align:middle line:84% We can do a similar argument starting from any other state. 00:38:32.450 --> 00:38:35.790 align:middle line:84% Starting from state i the probability that eventually I 00:38:35.790 --> 00:38:39.960 align:middle line:84% end up at state 4 is, we consider the different 00:38:39.960 --> 00:38:43.350 align:middle line:84% possible scenarios of where do I go next, which is my state 00:38:43.350 --> 00:38:46.440 align:middle line:90% j, with probability Pij. 00:38:46.440 --> 00:38:50.980 align:middle line:84% Next time I go to j, and given that I started at j, this is 00:38:50.980 --> 00:38:53.630 align:middle line:84% the probability that I end up at state 4. 00:38:53.630 --> 00:38:56.920 align:middle line:84% So this equation that we have here is just an abstract 00:38:56.920 --> 00:39:02.540 align:middle line:84% version in symbols of what we wrote down for the particular 00:39:02.540 --> 00:39:04.540 align:middle line:84% case where the initial state was 2. 00:39:04.540 --> 00:39:08.880 align:middle line:84% So you write down an equation of this type for every state 00:39:08.880 --> 00:39:09.580 align:middle line:90% inside here. 00:39:09.580 --> 00:39:15.520 align:middle line:84% You'll have a separate equation for a1, a2, and a3. 00:39:15.520 --> 00:39:19.200 align:middle line:84% And that's going to be a system of 3 equations with 3 00:39:19.200 --> 00:39:25.250 align:middle line:84% unknowns, the a's inside the transient states. 00:39:25.250 --> 00:39:29.190 align:middle line:84% So you can solve that 3 by 3 system of equations. 00:39:29.190 --> 00:39:33.590 align:middle line:84% Fortunately, it turns out to have a unique solution, and so 00:39:33.590 --> 00:39:36.070 align:middle line:84% once you solve it you have found the probabilities of 00:39:36.070 --> 00:39:40.880 align:middle line:84% absorption and the probability that eventually you get 00:39:40.880 --> 00:39:42.610 align:middle line:90% absorbed at state 4. 00:39:42.610 --> 00:39:51.540 align:middle line:90% 00:39:51.540 --> 00:39:56.790 align:middle line:84% Now, in the picture that we had here, this was a single 00:39:56.790 --> 00:39:59.570 align:middle line:84% state, and that one was a single state. 00:39:59.570 --> 00:40:06.500 align:middle line:84% How do things change if our recurrent, or trapping sets 00:40:06.500 --> 00:40:09.600 align:middle line:90% consist of multiple states? 00:40:09.600 --> 00:40:16.820 align:middle line:84% Well, it doesn't really matter that we have multiple states. 00:40:16.820 --> 00:40:21.470 align:middle line:84% All that matters is that this is one lump and once we get 00:40:21.470 --> 00:40:24.150 align:middle line:90% there we are stuck in there. 00:40:24.150 --> 00:40:33.170 align:middle line:84% So if the picture was, let's say, like this, 0.1 and 0.2, 00:40:33.170 --> 00:40:36.960 align:middle line:84% that basically means that whenever you are in that state 00:40:36.960 --> 00:40:42.620 align:middle line:84% there's a total probability of 0.3 of ending in that lump and 00:40:42.620 --> 00:40:45.100 align:middle line:84% getting stuck inside that lump. 00:40:45.100 --> 00:40:50.660 align:middle line:84% So you would take that picture and change it and make it 00:40:50.660 --> 00:40:57.330 align:middle line:84% instead a total probability of 0.3, of ending somewhere 00:40:57.330 --> 00:41:00.270 align:middle line:90% inside that lump. 00:41:00.270 --> 00:41:03.670 align:middle line:84% And similarly, you take this lump and you view it as just 00:41:03.670 --> 00:41:07.570 align:middle line:84% one entity, and from any state you record the total 00:41:07.570 --> 00:41:10.090 align:middle line:84% probability that given that I'm here I 00:41:10.090 --> 00:41:12.140 align:middle line:90% end up in that entity. 00:41:12.140 --> 00:41:15.540 align:middle line:84% So basically, if the only thing you care is the 00:41:15.540 --> 00:41:18.760 align:middle line:84% probability that you're going to end up in this lump, you 00:41:18.760 --> 00:41:22.830 align:middle line:84% can replace that lump with a single state, view it as a 00:41:22.830 --> 00:41:24.600 align:middle line:90% single state, and calculate 00:41:24.600 --> 00:41:26.180 align:middle line:90% probabilities using this formula. 00:41:26.180 --> 00:41:33.830 align:middle line:90% 00:41:33.830 --> 00:41:36.780 align:middle line:84% All right, so now we know where the chain is 00:41:36.780 --> 00:41:37.860 align:middle line:90% going to get to. 00:41:37.860 --> 00:41:39.850 align:middle line:84% At least we know probabilistically. 00:41:39.850 --> 00:41:42.690 align:middle line:84% We know with what probability it is going to go here, and 00:41:42.690 --> 00:41:45.160 align:middle line:84% that also tells us the probability that eventually 00:41:45.160 --> 00:41:47.160 align:middle line:90% it's going to get there. 00:41:47.160 --> 00:41:55.780 align:middle line:84% Other question, how long is it going to take until we get to 00:41:55.780 --> 00:41:58.300 align:middle line:84% either this state or that state? 00:41:58.300 --> 00:42:02.160 align:middle line:84% We can call that event absorption, meaning that the 00:42:02.160 --> 00:42:05.130 align:middle line:84% state got somewhere into a recurrent class from which it 00:42:05.130 --> 00:42:06.380 align:middle line:90% could not get out. 00:42:06.380 --> 00:42:12.340 align:middle line:90% 00:42:12.340 --> 00:42:12.510 align:middle line:90% Okay. 00:42:12.510 --> 00:42:15.570 align:middle line:84% Let's deal with that question for the case where we have 00:42:15.570 --> 00:42:19.320 align:middle line:90% only 1 absorbing state. 00:42:19.320 --> 00:42:22.410 align:middle line:84% So here our Markov chain is a little simpler than the one in 00:42:22.410 --> 00:42:23.580 align:middle line:90% the previous slide. 00:42:23.580 --> 00:42:25.870 align:middle line:84% We've got our transient states, we've got our 00:42:25.870 --> 00:42:28.930 align:middle line:84% recurrent state, and once you get into the recurrent state 00:42:28.930 --> 00:42:31.770 align:middle line:90% you just stay there. 00:42:31.770 --> 00:42:35.330 align:middle line:84% So here we're certain that no matter where we start we're 00:42:35.330 --> 00:42:37.270 align:middle line:90% going to end up here. 00:42:37.270 --> 00:42:39.170 align:middle line:90% How long is it going to take? 00:42:39.170 --> 00:42:40.830 align:middle line:90% Well, we don't know. 00:42:40.830 --> 00:42:42.540 align:middle line:90% It's a random variable. 00:42:42.540 --> 00:42:44.660 align:middle line:84% The expected value of that random variable, 00:42:44.660 --> 00:42:46.890 align:middle line:90% let's call it mu. 00:42:46.890 --> 00:42:50.510 align:middle line:84% But how long it takes to get there certainly depends on 00:42:50.510 --> 00:42:52.470 align:middle line:90% where we start. 00:42:52.470 --> 00:42:56.390 align:middle line:84% So let's put in our notation again this index i that 00:42:56.390 --> 00:42:59.220 align:middle line:84% indicates where we started from. 00:42:59.220 --> 00:43:03.300 align:middle line:84% And now the argument is going to be of the same type as the 00:43:03.300 --> 00:43:05.480 align:middle line:90% one we used before. 00:43:05.480 --> 00:43:11.620 align:middle line:84% We can think in terms of a tree once more, that considers 00:43:11.620 --> 00:43:13.930 align:middle line:90% all the possible options. 00:43:13.930 --> 00:43:16.300 align:middle line:84% So suppose that you start at state 1. 00:43:16.300 --> 00:43:19.060 align:middle line:90% 00:43:19.060 --> 00:43:24.110 align:middle line:84% Starting from state 1, the expected time until you end up 00:43:24.110 --> 00:43:27.100 align:middle line:90% dropping states is mu1. 00:43:27.100 --> 00:43:30.700 align:middle line:84% Now, starting from state 1, what are the possibilities? 00:43:30.700 --> 00:43:33.090 align:middle line:84% You make your first transition, and that first 00:43:33.090 --> 00:43:36.550 align:middle line:84% transition is going to take you either to state 00:43:36.550 --> 00:43:38.610 align:middle line:90% 2 or to state 3. 00:43:38.610 --> 00:43:42.010 align:middle line:84% It takes you to state 2 with probability 0.6, it takes you 00:43:42.010 --> 00:43:48.320 align:middle line:84% to state 3 with probability 0.4. 00:43:48.320 --> 00:43:52.490 align:middle line:84% Starting from state 2, eventually you're going to get 00:43:52.490 --> 00:43:54.410 align:middle line:90% to state 4. 00:43:54.410 --> 00:43:55.990 align:middle line:90% How long does it take? 00:43:55.990 --> 00:43:58.480 align:middle line:84% We don't know, it's a random variable. 00:43:58.480 --> 00:44:02.325 align:middle line:84% But the expected time until this happens is mu2. 00:44:02.325 --> 00:44:05.100 align:middle line:90% 00:44:05.100 --> 00:44:08.250 align:middle line:84% Starting from state 2, how long does it take you to get 00:44:08.250 --> 00:44:10.300 align:middle line:90% to state 4. 00:44:10.300 --> 00:44:13.870 align:middle line:84% And similarly starting from state 3, it's going to take 00:44:13.870 --> 00:44:17.850 align:middle line:84% you on the average mu3 time steps until you 00:44:17.850 --> 00:44:20.070 align:middle line:90% get to state 4. 00:44:20.070 --> 00:44:24.450 align:middle line:84% So what's the expected value of the time until I 00:44:24.450 --> 00:44:25.790 align:middle line:90% end at state 4? 00:44:25.790 --> 00:44:31.070 align:middle line:90% 00:44:31.070 --> 00:44:37.910 align:middle line:84% So with probability 0.6, I'm going to end up at state 2 and 00:44:37.910 --> 00:44:42.650 align:middle line:84% from there on it's going to be expected time mu2, and with 00:44:42.650 --> 00:44:45.850 align:middle line:84% probability 0.4 I'm going to end up at state 3, and from 00:44:45.850 --> 00:44:48.730 align:middle line:84% there it's going to take me so much time. 00:44:48.730 --> 00:44:52.790 align:middle line:84% So this is the expected time it's going to take me after 00:44:52.790 --> 00:44:54.860 align:middle line:90% the first transition. 00:44:54.860 --> 00:44:59.420 align:middle line:84% But we also spent 1 time step for the first transition. 00:44:59.420 --> 00:45:02.610 align:middle line:84% The total time to get there is the time of the first 00:45:02.610 --> 00:45:06.330 align:middle line:84% transition, which is 1, plus the expected time starting 00:45:06.330 --> 00:45:07.600 align:middle line:90% from the next state. 00:45:07.600 --> 00:45:10.210 align:middle line:84% This expression here is the expected time starting from 00:45:10.210 --> 00:45:13.640 align:middle line:84% the next state, but we also need to account for the first 00:45:13.640 --> 00:45:15.840 align:middle line:90% transition, so we add 1. 00:45:15.840 --> 00:45:18.010 align:middle line:84% And this is going to be our mu1. 00:45:18.010 --> 00:45:21.150 align:middle line:90% 00:45:21.150 --> 00:45:25.370 align:middle line:84% So once more we have a linear equation that ties together 00:45:25.370 --> 00:45:27.930 align:middle line:90% the different mu's. 00:45:27.930 --> 00:45:32.070 align:middle line:84% And the equation starting from state 4 in this case, of 00:45:32.070 --> 00:45:35.620 align:middle line:84% course is going to be simple, starting from that state the 00:45:35.620 --> 00:45:38.280 align:middle line:84% expected number of steps it takes you to get there for the 00:45:38.280 --> 00:45:42.220 align:middle line:84% first time is of course, 0 because you're already there. 00:45:42.220 --> 00:45:45.640 align:middle line:84% So for that state this is fine, and for all the other 00:45:45.640 --> 00:45:48.280 align:middle line:84% states you get an equation of this form. 00:45:48.280 --> 00:45:51.410 align:middle line:84% Now we're going to have an equation for every state. 00:45:51.410 --> 00:45:53.760 align:middle line:84% It's a system of linear equations, once more we can 00:45:53.760 --> 00:45:57.290 align:middle line:84% solve them, and this gives us the expected times until our 00:45:57.290 --> 00:46:03.890 align:middle line:84% chain gets absorbed in this absorbing state. 00:46:03.890 --> 00:46:06.650 align:middle line:84% And it's nice to know that this system of equations 00:46:06.650 --> 00:46:09.510 align:middle line:90% always has a unique solution. 00:46:09.510 --> 00:46:13.220 align:middle line:84% OK so this was the expected time to absorption. 00:46:13.220 --> 00:46:16.720 align:middle line:84% For this case where we had this scene absorbing state. 00:46:16.720 --> 00:46:23.740 align:middle line:84% Suppose that we have our transient states and that we 00:46:23.740 --> 00:46:27.020 align:middle line:84% have multiple recurrent classes, or 00:46:27.020 --> 00:46:28.590 align:middle line:90% multiple absorbing states. 00:46:28.590 --> 00:46:33.850 align:middle line:90% 00:46:33.850 --> 00:46:37.470 align:middle line:84% Suppose you've got the picture like this. 00:46:37.470 --> 00:46:42.040 align:middle line:84% And we want to calculate the expected time until we get 00:46:42.040 --> 00:46:44.110 align:middle line:90% here or there. 00:46:44.110 --> 00:46:47.830 align:middle line:84% Expected time until we get to an absorbing state. 00:46:47.830 --> 00:46:50.320 align:middle line:90% What's the trick? 00:46:50.320 --> 00:46:55.010 align:middle line:84% Well, we can lump both of these states together and 00:46:55.010 --> 00:47:00.660 align:middle line:84% think of them as just one bad state, one place for which 00:47:00.660 --> 00:47:03.730 align:middle line:84% we're interested in how long it takes us to get there. 00:47:03.730 --> 00:47:10.250 align:middle line:84% So lump them as one state, and accordingly kind of merge all 00:47:10.250 --> 00:47:11.570 align:middle line:90% of those probabilities. 00:47:11.570 --> 00:47:15.470 align:middle line:84% So starting from here, my probability that the next I 00:47:15.470 --> 00:47:18.690 align:middle line:84% end up in this lump and they get absorbed is going to be 00:47:18.690 --> 00:47:21.630 align:middle line:84% this probability plus that probability. 00:47:21.630 --> 00:47:24.010 align:middle line:84% So we would change that picture. 00:47:24.010 --> 00:47:30.080 align:middle line:84% Think of this as being just one big state. 00:47:30.080 --> 00:47:36.080 align:middle line:84% And sort of add those two probabilities together to come 00:47:36.080 --> 00:47:39.360 align:middle line:84% up with a single probability, which is the probability that 00:47:39.360 --> 00:47:42.590 align:middle line:84% starting from here next time I find myself at 00:47:42.590 --> 00:47:44.440 align:middle line:90% some absorbing state. 00:47:44.440 --> 00:47:48.510 align:middle line:84% So once you know how to deal with a situation like this, 00:47:48.510 --> 00:47:52.030 align:middle line:84% you can also find expected times to absorption for the 00:47:52.030 --> 00:47:55.190 align:middle line:84% case where you've got multiple absorbing states. 00:47:55.190 --> 00:47:57.990 align:middle line:84% You just lump all of those multiple absorbing states into 00:47:57.990 --> 00:47:59.240 align:middle line:90% a single one. 00:47:59.240 --> 00:48:01.550 align:middle line:90% 00:48:01.550 --> 00:48:05.040 align:middle line:84% Finally, there's a kind of related 00:48:05.040 --> 00:48:09.600 align:middle line:90% quantity that's of interest. 00:48:09.600 --> 00:48:13.480 align:middle line:84% The question is almost the same as in the previous slide, 00:48:13.480 --> 00:48:16.950 align:middle line:84% except that here we do not have any absorbing states. 00:48:16.950 --> 00:48:20.335 align:middle line:84% Rather, we have a single recurrent class of states. 00:48:20.335 --> 00:48:25.320 align:middle line:90% 00:48:25.320 --> 00:48:27.840 align:middle line:90% You start at some state i. 00:48:27.840 --> 00:48:31.780 align:middle line:84% You have a special state, that is state s. 00:48:31.780 --> 00:48:35.600 align:middle line:84% And you ask the question, how long is it going to take me 00:48:35.600 --> 00:48:39.040 align:middle line:84% until I get to s for the first time? 00:48:39.040 --> 00:48:41.140 align:middle line:84% It's a single recurrent class of states. 00:48:41.140 --> 00:48:43.730 align:middle line:84% So you know that the state keeps circulating here and it 00:48:43.730 --> 00:48:46.610 align:middle line:84% keeps visiting all of the possible states. 00:48:46.610 --> 00:48:49.510 align:middle line:84% So eventually this state will be visited. 00:48:49.510 --> 00:48:52.070 align:middle line:84% How long does it take for this to happen? 00:48:52.070 --> 00:48:55.290 align:middle line:90% 00:48:55.290 --> 00:48:55.425 align:middle line:90% Ok. 00:48:55.425 --> 00:48:58.820 align:middle line:84% So we're interested in how long it takes for this to 00:48:58.820 --> 00:49:02.080 align:middle line:84% happen, how long it takes until we get to s for the 00:49:02.080 --> 00:49:03.000 align:middle line:90% first time. 00:49:03.000 --> 00:49:06.480 align:middle line:84% And we don't care about what happens afterwards. 00:49:06.480 --> 00:49:10.110 align:middle line:84% So we might as well change this picture and remove the 00:49:10.110 --> 00:49:15.470 align:middle line:84% transitions out of s and to make them self transitions. 00:49:15.470 --> 00:49:18.140 align:middle line:90% Is the answer going to change? 00:49:18.140 --> 00:49:19.280 align:middle line:90% No. 00:49:19.280 --> 00:49:22.890 align:middle line:84% The only thing that we changed was what happens 00:49:22.890 --> 00:49:25.760 align:middle line:90% after you get to s. 00:49:25.760 --> 00:49:28.810 align:middle line:84% But what happens after you get to s doesn't matter. 00:49:28.810 --> 00:49:31.730 align:middle line:84% The question we're dealing with is how long does it take 00:49:31.730 --> 00:49:33.900 align:middle line:90% us to get to s. 00:49:33.900 --> 00:49:37.010 align:middle line:84% So essentially, it's after we do this transformation -- 00:49:37.010 --> 00:49:40.740 align:middle line:84% it's the same question as before, what's the time it 00:49:40.740 --> 00:49:43.990 align:middle line:84% takes until eventually we hit this state. 00:49:43.990 --> 00:49:46.390 align:middle line:84% And it's now in this new picture, this state is an 00:49:46.390 --> 00:49:48.410 align:middle line:90% absorbing state. 00:49:48.410 --> 00:49:50.710 align:middle line:84% Or you can just think from first principles. 00:49:50.710 --> 00:49:55.720 align:middle line:84% Starting from the state itself, s, it takes you 0 time 00:49:55.720 --> 00:49:57.430 align:middle line:90% steps until you get to s. 00:49:57.430 --> 00:50:01.610 align:middle line:84% Starting from anywhere else, you need one transition and 00:50:01.610 --> 00:50:05.050 align:middle line:84% then after the first transition you find yourself 00:50:05.050 --> 00:50:09.560 align:middle line:84% at state j with probability Pij and from then on you are 00:50:09.560 --> 00:50:13.350 align:middle line:84% going to take expected time Tj until you get to that 00:50:13.350 --> 00:50:14.710 align:middle line:90% terminal state s. 00:50:14.710 --> 00:50:17.340 align:middle line:84% So once more these equations have a unique solution, you 00:50:17.340 --> 00:50:19.510 align:middle line:84% can solve them and find the answer. 00:50:19.510 --> 00:50:22.530 align:middle line:84% And finally, there's a related question, which is the mean 00:50:22.530 --> 00:50:24.210 align:middle line:90% recurrence time of s. 00:50:24.210 --> 00:50:30.490 align:middle line:84% In that question you start at s, the chain will move 00:50:30.490 --> 00:50:34.090 align:middle line:84% randomly, and you ask how long is it going to take until I 00:50:34.090 --> 00:50:37.160 align:middle line:84% come back to s for the next time. 00:50:37.160 --> 00:50:38.390 align:middle line:90% So notice the difference. 00:50:38.390 --> 00:50:42.670 align:middle line:84% Here we're talking the first time after time 0, whereas 00:50:42.670 --> 00:50:45.500 align:middle line:84% here it's just the first time anywhere. 00:50:45.500 --> 00:50:51.250 align:middle line:84% So here if you start from s, Ts* is not 0. 00:50:51.250 --> 00:50:54.450 align:middle line:84% You want to do at least one transition and that's how long 00:50:54.450 --> 00:50:57.360 align:middle line:84% it's going to take me until it gets back to s. 00:50:57.360 --> 00:51:00.900 align:middle line:84% Well, how long does it take me until I get back to s? 00:51:00.900 --> 00:51:06.060 align:middle line:84% I do my first transition, and then after my first transition 00:51:06.060 --> 00:51:11.920 align:middle line:84% I calculate the expected time from the next state how long 00:51:11.920 --> 00:51:15.350 align:middle line:84% it's going to take me until I come back to s. 00:51:15.350 --> 00:51:20.200 align:middle line:84% So all of these equations that I wrote down, they all kind of 00:51:20.200 --> 00:51:22.500 align:middle line:90% look the same. 00:51:22.500 --> 00:51:23.620 align:middle line:90% But they are different. 00:51:23.620 --> 00:51:27.210 align:middle line:84% So you can either memorize all of these equations, or instead 00:51:27.210 --> 00:51:30.580 align:middle line:84% what's better is to just to get the basic idea. 00:51:30.580 --> 00:51:33.140 align:middle line:84% That is, to calculate probabilities or expected 00:51:33.140 --> 00:51:35.830 align:middle line:84% values you use the total probability or total 00:51:35.830 --> 00:51:38.260 align:middle line:84% expectation theorem and conditional the first 00:51:38.260 --> 00:51:40.810 align:middle line:84% transition and take it from there. 00:51:40.810 --> 00:51:43.230 align:middle line:84% So you're going to get a little bit of practice with 00:51:43.230 --> 00:51:47.276 align:middle line:84% these skills in recitation tomorrow, and of course it's 00:51:47.276 --> 00:51:48.730 align:middle line:90% in your problem set as well. 00:51:48.730 --> 00:51:49.980 align:middle line:90%