WEBVTT 00:00:00.000 --> 00:00:00.040 align:middle line:90% 00:00:00.040 --> 00:00:02.460 align:middle line:84% The following content is provided under a Creative 00:00:02.460 --> 00:00:03.870 align:middle line:90% Commons license. 00:00:03.870 --> 00:00:06.910 align:middle line:84% Your support will help MIT OpenCourseWare continue to 00:00:06.910 --> 00:00:10.560 align:middle line:84% offer high quality educational resources for free. 00:00:10.560 --> 00:00:13.460 align:middle line:84% To make a donation or view additional materials from 00:00:13.460 --> 00:00:19.290 align:middle line:84% hundreds of MIT courses, visit MIT OpenCourseWare at 00:00:19.290 --> 00:00:20.540 align:middle line:90% ocw.mit.edu. 00:00:20.540 --> 00:00:22.340 align:middle line:90% 00:00:22.340 --> 00:00:25.020 align:middle line:84% PROFESSOR: So we're going to start now with a new chapter. 00:00:25.020 --> 00:00:28.720 align:middle line:84% We're going to talk about Markov processes. 00:00:28.720 --> 00:00:32.390 align:middle line:84% The good news is that this is a subject that is a lot more 00:00:32.390 --> 00:00:37.040 align:middle line:84% intuitive and simple in many ways than, let's say, the 00:00:37.040 --> 00:00:38.410 align:middle line:90% Poisson processes. 00:00:38.410 --> 00:00:40.590 align:middle line:84% So hopefully this will be enjoyable. 00:00:40.590 --> 00:00:42.680 align:middle line:84% So Markov processes is, a general 00:00:42.680 --> 00:00:46.420 align:middle line:90% class of random processes. 00:00:46.420 --> 00:00:49.610 align:middle line:84% In some sense, it's more elaborate than the Bernoulli 00:00:49.610 --> 00:00:52.780 align:middle line:84% and Poisson processes, because now we're going to have 00:00:52.780 --> 00:00:55.820 align:middle line:84% dependencies between difference times, instead of 00:00:55.820 --> 00:00:57.960 align:middle line:90% having memoryless processes. 00:00:57.960 --> 00:01:00.100 align:middle line:84% So the basic idea is the following. 00:01:00.100 --> 00:01:05.060 align:middle line:84% In physics, for example, you write down equations for how a 00:01:05.060 --> 00:01:08.240 align:middle line:84% system evolves that has the general form. 00:01:08.240 --> 00:01:11.780 align:middle line:84% The new state of a system one second later is some function 00:01:11.780 --> 00:01:14.970 align:middle line:90% of old state. 00:01:14.970 --> 00:01:20.020 align:middle line:84% So Newton's equations and all that in physics allow you to 00:01:20.020 --> 00:01:21.490 align:middle line:90% write equations of this kind. 00:01:21.490 --> 00:01:26.260 align:middle line:84% And so if that a particle is moving at a certain velocity 00:01:26.260 --> 00:01:28.790 align:middle line:84% and it's at some location, you can predict when it's going to 00:01:28.790 --> 00:01:30.510 align:middle line:90% be a little later. 00:01:30.510 --> 00:01:34.760 align:middle line:84% Markov processes have the same flavor, except that there's 00:01:34.760 --> 00:01:38.810 align:middle line:84% also some randomness thrown inside the equation. 00:01:38.810 --> 00:01:42.720 align:middle line:84% So that's what Markov process essentially is. 00:01:42.720 --> 00:01:47.820 align:middle line:84% It describes the evolution of the system, or some variables, 00:01:47.820 --> 00:01:51.450 align:middle line:84% but in the presence of some noise so that the motion 00:01:51.450 --> 00:01:55.110 align:middle line:90% itself is a bit random. 00:01:55.110 --> 00:01:58.880 align:middle line:84% So this is a pretty general framework. 00:01:58.880 --> 00:02:02.850 align:middle line:84% So pretty much any useful or interesting random process 00:02:02.850 --> 00:02:06.280 align:middle line:84% that you can think about, you can always described it as a 00:02:06.280 --> 00:02:09.050 align:middle line:84% Markov process if you define properly the 00:02:09.050 --> 00:02:10.720 align:middle line:90% notion of the state. 00:02:10.720 --> 00:02:13.960 align:middle line:84% So what we're going to do is we're going to introduce the 00:02:13.960 --> 00:02:17.630 align:middle line:84% class of Markov processes by, example, by talking about the 00:02:17.630 --> 00:02:19.900 align:middle line:84% checkout counter in a supermarket. 00:02:19.900 --> 00:02:22.650 align:middle line:84% Then we're going to abstract from our example so that we 00:02:22.650 --> 00:02:25.580 align:middle line:90% get a more general definition. 00:02:25.580 --> 00:02:28.410 align:middle line:84% And then we're going to do a few things, such as how to 00:02:28.410 --> 00:02:32.250 align:middle line:84% predict what's going to happen n time steps later, if we 00:02:32.250 --> 00:02:34.210 align:middle line:90% start at the particular state. 00:02:34.210 --> 00:02:36.460 align:middle line:84% And then talk a little bit about some structural 00:02:36.460 --> 00:02:40.640 align:middle line:84% properties of Markov processes or Markov chains. 00:02:40.640 --> 00:02:42.050 align:middle line:90% So here's our example. 00:02:42.050 --> 00:02:44.770 align:middle line:90% 00:02:44.770 --> 00:02:49.670 align:middle line:84% You go to the checkout counter at the supermarket, and you 00:02:49.670 --> 00:02:54.460 align:middle line:84% stand there and watch the customers who come. 00:02:54.460 --> 00:02:59.090 align:middle line:84% So customers come, they get in queue, and customers get 00:02:59.090 --> 00:03:01.250 align:middle line:90% served one at a time. 00:03:01.250 --> 00:03:03.730 align:middle line:84% So the discussion is going to be in terms of supermarket 00:03:03.730 --> 00:03:06.950 align:middle line:84% checkout counters, but the same story applies to any 00:03:06.950 --> 00:03:08.420 align:middle line:90% service system. 00:03:08.420 --> 00:03:12.470 align:middle line:84% You may have a server, jobs arrive to that server, they 00:03:12.470 --> 00:03:15.580 align:middle line:84% get put into the queue, and the server processes those 00:03:15.580 --> 00:03:17.750 align:middle line:90% jobs one at a time. 00:03:17.750 --> 00:03:20.480 align:middle line:84% Now to make a probabilistic model, we need to make some 00:03:20.480 --> 00:03:23.240 align:middle line:84% assumption about the customer arrivals and the customer 00:03:23.240 --> 00:03:24.290 align:middle line:90% departures. 00:03:24.290 --> 00:03:26.150 align:middle line:84% And we want to keep things as simple as 00:03:26.150 --> 00:03:28.160 align:middle line:90% possible to get started. 00:03:28.160 --> 00:03:31.220 align:middle line:84% So let's assume that customers arrive according to a 00:03:31.220 --> 00:03:34.230 align:middle line:84% Bernoulli process with some parameter b. 00:03:34.230 --> 00:03:37.380 align:middle line:84% So essentially, that's the same as the assumption that 00:03:37.380 --> 00:03:40.590 align:middle line:84% the time between consecutive customer arrivals is a 00:03:40.590 --> 00:03:45.280 align:middle line:84% geometric random variable with parameter b. 00:03:45.280 --> 00:03:49.740 align:middle line:84% Another way of thinking about the arrival process-- 00:03:49.740 --> 00:03:54.260 align:middle line:84% that's not how it happens, but it's helpful, mathematically, 00:03:54.260 --> 00:03:58.330 align:middle line:84% is to think of someone who's flipping a coin with bias 00:03:58.330 --> 00:03:59.510 align:middle line:90% equal to b. 00:03:59.510 --> 00:04:02.750 align:middle line:84% And whenever the coin lands heads, 00:04:02.750 --> 00:04:05.050 align:middle line:90% then a customer arrives. 00:04:05.050 --> 00:04:08.960 align:middle line:84% So it's as if there's a coin flip being done by nature that 00:04:08.960 --> 00:04:11.580 align:middle line:84% decides the arrivals of the customers. 00:04:11.580 --> 00:04:16.110 align:middle line:84% So we know that coin flipping to determine the customer 00:04:16.110 --> 00:04:19.550 align:middle line:84% arrivals is the same as having geometric inter-arrival times. 00:04:19.550 --> 00:04:23.200 align:middle line:84% We know that from our study of the Bernoulli process. 00:04:23.200 --> 00:04:23.630 align:middle line:90% OK. 00:04:23.630 --> 00:04:27.790 align:middle line:84% And now how about the customer service times. 00:04:27.790 --> 00:04:29.610 align:middle line:90% We're going to assume that-- 00:04:29.610 --> 00:04:30.660 align:middle line:90% OK. 00:04:30.660 --> 00:04:34.120 align:middle line:84% If there is no customer in queue, no one being served, 00:04:34.120 --> 00:04:37.040 align:middle line:84% then of course, no one is going to 00:04:37.040 --> 00:04:38.530 align:middle line:90% depart from the queue. 00:04:38.530 --> 00:04:42.270 align:middle line:84% But if there a customer in queue, then that customer 00:04:42.270 --> 00:04:45.200 align:middle line:84% starts being served, and is going to be served for a 00:04:45.200 --> 00:04:46.870 align:middle line:90% random amount of time. 00:04:46.870 --> 00:04:50.820 align:middle line:84% And we make the assumption that the time it takes for the 00:04:50.820 --> 00:04:54.470 align:middle line:84% clerk to serve the customer has a geometric distribution 00:04:54.470 --> 00:04:57.270 align:middle line:90% with some known parameter q. 00:04:57.270 --> 00:05:00.630 align:middle line:84% So the time it takes to serve a customer is random, because 00:05:00.630 --> 00:05:04.060 align:middle line:84% it's random how many items they got in their cart, and 00:05:04.060 --> 00:05:06.990 align:middle line:84% how many coupons they have to unload and so on. 00:05:06.990 --> 00:05:09.730 align:middle line:90% So it's random. 00:05:09.730 --> 00:05:13.230 align:middle line:84% In the real world, it has some probability distribution. 00:05:13.230 --> 00:05:16.240 align:middle line:84% Let's not care exactly about what it would be in the real 00:05:16.240 --> 00:05:18.830 align:middle line:84% world, but as a modeling approximation or just to get 00:05:18.830 --> 00:05:22.200 align:middle line:84% started, let's pretend that customer service time are well 00:05:22.200 --> 00:05:25.150 align:middle line:84% described by a geometric distribution, 00:05:25.150 --> 00:05:27.500 align:middle line:90% with a parameter q. 00:05:27.500 --> 00:05:31.010 align:middle line:84% An equivalent way of thinking about the customer service, 00:05:31.010 --> 00:05:32.640 align:middle line:84% mathematically, would be, again, in 00:05:32.640 --> 00:05:34.350 align:middle line:90% terms of coin flipping. 00:05:34.350 --> 00:05:38.490 align:middle line:84% That is, the clerk has a coin with a bias, and at each time 00:05:38.490 --> 00:05:40.620 align:middle line:90% slot the clerk flips the coin. 00:05:40.620 --> 00:05:43.470 align:middle line:84% With probability q, service is over. 00:05:43.470 --> 00:05:49.100 align:middle line:84% With probability 1-q, you continue the service process. 00:05:49.100 --> 00:05:52.370 align:middle line:84% An assumption that we're going to make is that the coin flips 00:05:52.370 --> 00:05:56.220 align:middle line:84% that happen here to determine the arrivals, they're all 00:05:56.220 --> 00:05:57.860 align:middle line:90% independent of each other. 00:05:57.860 --> 00:06:02.610 align:middle line:84% The coin flips that determine the end of service are also 00:06:02.610 --> 00:06:03.930 align:middle line:90% independent from each other. 00:06:03.930 --> 00:06:07.160 align:middle line:84% But also the coin flips involved here are independent 00:06:07.160 --> 00:06:09.740 align:middle line:84% from the coin flips that happened there. 00:06:09.740 --> 00:06:15.190 align:middle line:84% So how arrivals happen is independent with what happens 00:06:15.190 --> 00:06:17.490 align:middle line:90% at the service process. 00:06:17.490 --> 00:06:18.330 align:middle line:90% OK. 00:06:18.330 --> 00:06:21.240 align:middle line:84% So suppose now you want to answer a 00:06:21.240 --> 00:06:23.200 align:middle line:90% question such as the following. 00:06:23.200 --> 00:06:24.830 align:middle line:90% The time is 7:00 PM. 00:06:24.830 --> 00:06:28.640 align:middle line:84% What's the probability that the customer will be departing 00:06:28.640 --> 00:06:30.990 align:middle line:90% at this particular time? 00:06:30.990 --> 00:06:33.870 align:middle line:90% Well, you say, it depends. 00:06:33.870 --> 00:06:37.740 align:middle line:84% If the queue is empty at that time, then you're certain that 00:06:37.740 --> 00:06:40.200 align:middle line:84% you're not going to have a customer departure. 00:06:40.200 --> 00:06:44.970 align:middle line:84% But if the queue is not empty, then there is probability q 00:06:44.970 --> 00:06:47.880 align:middle line:84% that a departure will happen at that time. 00:06:47.880 --> 00:06:52.230 align:middle line:84% So the answer to a question like this has something to do 00:06:52.230 --> 00:06:54.460 align:middle line:84% with the state of the system at that time. 00:06:54.460 --> 00:06:56.770 align:middle line:90% It depends what the queue is. 00:06:56.770 --> 00:07:02.770 align:middle line:84% And if I ask you, will the queue be empty at 7:10? 00:07:02.770 --> 00:07:06.870 align:middle line:84% Well, the answer to that question depends on whether at 00:07:06.870 --> 00:07:10.630 align:middle line:84% 7 o'clock whether the queue was huge or not. 00:07:10.630 --> 00:07:14.770 align:middle line:84% So knowing something about the state of the queue right now 00:07:14.770 --> 00:07:17.480 align:middle line:84% gives me relevant information about what may 00:07:17.480 --> 00:07:19.790 align:middle line:90% happen in the future. 00:07:19.790 --> 00:07:22.770 align:middle line:84% So what is the state of the system? 00:07:22.770 --> 00:07:26.080 align:middle line:84% Therefore we're brought to start using this term. 00:07:26.080 --> 00:07:28.720 align:middle line:84% So the state basically corresponds to 00:07:28.720 --> 00:07:31.980 align:middle line:90% anything that's relevant. 00:07:31.980 --> 00:07:34.900 align:middle line:84% Anything that's happening right now that's kind of 00:07:34.900 --> 00:07:38.120 align:middle line:84% relevant to what may happen in the future. 00:07:38.120 --> 00:07:41.360 align:middle line:84% Knowing the size of the queue right now, is useful 00:07:41.360 --> 00:07:45.700 align:middle line:84% information for me to make predictions about what may 00:07:45.700 --> 00:07:49.100 align:middle line:84% happen 2 minutes later from now. 00:07:49.100 --> 00:07:52.510 align:middle line:84% So in this particular example, a reasonable choice for the 00:07:52.510 --> 00:07:56.410 align:middle line:84% state is to just count how many customers 00:07:56.410 --> 00:07:58.950 align:middle line:90% we have in the queue. 00:07:58.950 --> 00:08:02.330 align:middle line:84% And let's assume that our supermarket building is not 00:08:02.330 --> 00:08:05.230 align:middle line:84% too big, so it can only hold 10 people. 00:08:05.230 --> 00:08:07.480 align:middle line:84% So we're going to limit the states. 00:08:07.480 --> 00:08:11.230 align:middle line:84% Instead of going from 0 to infinity, we're going to 00:08:11.230 --> 00:08:13.580 align:middle line:90% truncate our model at ten. 00:08:13.580 --> 00:08:18.240 align:middle line:84% So we have 11 possible states, corresponding to 0 customers 00:08:18.240 --> 00:08:22.620 align:middle line:84% in queue, 1 customer in queue, 2 customers, and so on, all 00:08:22.620 --> 00:08:24.190 align:middle line:90% the way up to 10. 00:08:24.190 --> 00:08:27.600 align:middle line:84% So these are the different possible states of the system, 00:08:27.600 --> 00:08:33.340 align:middle line:84% assuming that the store cannot handle more than 10 customers. 00:08:33.340 --> 00:08:37.360 align:middle line:84% So this is the first step, to write down the set of possible 00:08:37.360 --> 00:08:38.950 align:middle line:90% states for our system. 00:08:38.950 --> 00:08:41.820 align:middle line:84% Then the next thing to do is to start describing the 00:08:41.820 --> 00:08:45.570 align:middle line:84% possible transitions between the states. 00:08:45.570 --> 00:08:48.750 align:middle line:84% At any given time step, what are the 00:08:48.750 --> 00:08:50.030 align:middle line:90% things that can happen? 00:08:50.030 --> 00:08:53.180 align:middle line:84% We can have a customer arrival, which 00:08:53.180 --> 00:08:55.830 align:middle line:90% moves the state 1 higher. 00:08:55.830 --> 00:08:58.560 align:middle line:84% We can have a customer departure, which moves the 00:08:58.560 --> 00:09:00.320 align:middle line:90% state 1 lower. 00:09:00.320 --> 00:09:03.080 align:middle line:84% There's a possibility that nothing happens, in which case 00:09:03.080 --> 00:09:04.710 align:middle line:90% the state stays the same. 00:09:04.710 --> 00:09:06.470 align:middle line:84% And there's also the possibility of having 00:09:06.470 --> 00:09:10.800 align:middle line:84% simultaneously an arrival and a departure, in which case the 00:09:10.800 --> 00:09:12.700 align:middle line:90% state again stays the same. 00:09:12.700 --> 00:09:16.290 align:middle line:84% So let's write some representative probabilities. 00:09:16.290 --> 00:09:19.630 align:middle line:84% If we have 2 customers, the probability that during this 00:09:19.630 --> 00:09:22.800 align:middle line:84% step we go down, this is the probability that we have a 00:09:22.800 --> 00:09:26.940 align:middle line:84% service completion, but to no customer arrival. 00:09:26.940 --> 00:09:30.060 align:middle line:84% So this is the probability associated with this 00:09:30.060 --> 00:09:31.730 align:middle line:90% transition. 00:09:31.730 --> 00:09:37.280 align:middle line:84% The other possibility is that there's a customer arrival, 00:09:37.280 --> 00:09:40.910 align:middle line:84% which happens with probability p, and we do not have a 00:09:40.910 --> 00:09:45.120 align:middle line:84% customer departure, and so the probability of that particular 00:09:45.120 --> 00:09:47.690 align:middle line:90% transition is this number. 00:09:47.690 --> 00:09:50.960 align:middle line:84% And then finally, the probability that we stay in 00:09:50.960 --> 00:09:55.530 align:middle line:84% the same state, this can happen in 2 possible ways. 00:09:55.530 --> 00:10:00.360 align:middle line:84% One way is that we have an arrival and a departure 00:10:00.360 --> 00:10:01.930 align:middle line:90% simultaneously. 00:10:01.930 --> 00:10:05.690 align:middle line:84% And the other possibility is that we have no arrival and no 00:10:05.690 --> 00:10:09.670 align:middle line:84% departure, so that the state stays the same. 00:10:09.670 --> 00:10:11.870 align:middle line:84% So these transition probabilities would be the 00:10:11.870 --> 00:10:15.420 align:middle line:84% same starting from any other states, state 3, or 00:10:15.420 --> 00:10:17.050 align:middle line:90% state 9, and so on. 00:10:17.050 --> 00:10:20.010 align:middle line:84% Transition probabilities become a little different at 00:10:20.010 --> 00:10:23.750 align:middle line:84% the borders, at the boundaries of this diagram, because if 00:10:23.750 --> 00:10:27.350 align:middle line:84% you're in a state 0, then you cannot have any customer 00:10:27.350 --> 00:10:28.130 align:middle line:90% departures. 00:10:28.130 --> 00:10:31.940 align:middle line:84% There's no one to be served, but there is a probability p 00:10:31.940 --> 00:10:36.040 align:middle line:84% that the customer arrives, in which case the number of 00:10:36.040 --> 00:10:38.110 align:middle line:84% customers in the system goes to 1. 00:10:38.110 --> 00:10:41.150 align:middle line:84% Then probability 1-p, nothing happens. 00:10:41.150 --> 00:10:46.020 align:middle line:84% Similarly with departures, if the system is full, there's no 00:10:46.020 --> 00:10:47.780 align:middle line:90% room for another arrival. 00:10:47.780 --> 00:10:50.300 align:middle line:84% But we may have a departure that happens with probability 00:10:50.300 --> 00:10:55.250 align:middle line:84% q, and nothing happens with probability 1-q. 00:10:55.250 --> 00:11:00.260 align:middle line:84% So this is the full transition diagram annotated with 00:11:00.260 --> 00:11:02.150 align:middle line:90% transition probabilities. 00:11:02.150 --> 00:11:05.970 align:middle line:84% And this is a complete description of a discrete 00:11:05.970 --> 00:11:10.000 align:middle line:84% time, finite state Markov chain. 00:11:10.000 --> 00:11:13.010 align:middle line:84% So this is a complete probabilistic model. 00:11:13.010 --> 00:11:15.520 align:middle line:84% Once you have all of these pieces of information, you can 00:11:15.520 --> 00:11:18.370 align:middle line:84% start calculating things, and trying to predict what's going 00:11:18.370 --> 00:11:20.140 align:middle line:90% to happen in the future. 00:11:20.140 --> 00:11:24.460 align:middle line:84% Now let us abstract from this example and come up with a 00:11:24.460 --> 00:11:27.530 align:middle line:90% more general definition. 00:11:27.530 --> 00:11:37.010 align:middle line:84% So we have this concept of the state which describes the 00:11:37.010 --> 00:11:40.560 align:middle line:84% current situation in the system that we're looking at. 00:11:40.560 --> 00:11:44.440 align:middle line:84% The current state is random, so we're going to think of it 00:11:44.440 --> 00:11:50.570 align:middle line:84% as a random variable Xn is the state, and transitions after 00:11:50.570 --> 00:11:52.560 align:middle line:90% the system started operating. 00:11:52.560 --> 00:11:56.450 align:middle line:84% So the system starts operating at some initial state X0, and 00:11:56.450 --> 00:12:00.190 align:middle line:84% after n transitions, it moves to state Xn. 00:12:00.190 --> 00:12:03.020 align:middle line:84% Now we have a set of possible states. 00:12:03.020 --> 00:12:06.930 align:middle line:84% State 1 state 2, state 3, and in general, 00:12:06.930 --> 00:12:10.680 align:middle line:90% state i and state j. 00:12:10.680 --> 00:12:13.870 align:middle line:84% To keep things simple, we assume that the set of 00:12:13.870 --> 00:12:16.700 align:middle line:84% possible states is a finite set. 00:12:16.700 --> 00:12:19.350 align:middle line:84% As you can imagine, we can have systems in which the 00:12:19.350 --> 00:12:21.160 align:middle line:84% state space is going to be infinite. 00:12:21.160 --> 00:12:23.240 align:middle line:84% It could be discrete, or continuous. 00:12:23.240 --> 00:12:25.870 align:middle line:84% But all that is more difficult and more complicated. 00:12:25.870 --> 00:12:29.110 align:middle line:84% It makes sense to start from the simplest possible setting 00:12:29.110 --> 00:12:33.770 align:middle line:84% where we just deal with the finite state space. 00:12:33.770 --> 00:12:39.430 align:middle line:84% And time is discrete, so we can think of this state in the 00:12:39.430 --> 00:12:42.660 align:middle line:84% beginning, after 1 transition, 2 transitions, and so on. 00:12:42.660 --> 00:12:46.600 align:middle line:84% So we're in discrete time and we have finite in many states. 00:12:46.600 --> 00:12:49.900 align:middle line:84% So the system starts somewhere, and at every time 00:12:49.900 --> 00:12:54.506 align:middle line:84% step, the state is, let's say, here. 00:12:54.506 --> 00:12:59.850 align:middle line:84% A whistle blows, and the state jumps to a random next state. 00:12:59.850 --> 00:13:05.120 align:middle line:84% So it may move here, or it may move there, or it may move 00:13:05.120 --> 00:13:08.510 align:middle line:84% here, or it might stay in the place. 00:13:08.510 --> 00:13:11.400 align:middle line:84% So one possible transition is the transition before you 00:13:11.400 --> 00:13:13.880 align:middle line:84% jump, and just land in the same place 00:13:13.880 --> 00:13:15.760 align:middle line:90% where you started from. 00:13:15.760 --> 00:13:19.410 align:middle line:84% Now we want to describe the statistics of these 00:13:19.410 --> 00:13:20.490 align:middle line:90% transitions. 00:13:20.490 --> 00:13:23.760 align:middle line:84% If I am at that state, how likely is it to that, next 00:13:23.760 --> 00:13:26.885 align:middle line:84% time, I'm going to find myself at that state? 00:13:26.885 --> 00:13:30.390 align:middle line:84% Well, we describe the statistics of this transition 00:13:30.390 --> 00:13:35.730 align:middle line:84% by writing down a transition probability, the transition 00:13:35.730 --> 00:13:41.420 align:middle line:84% probability of going from state 3 to state 1. 00:13:41.420 --> 00:13:44.180 align:middle line:84% So this transition probability is to be thought of as a 00:13:44.180 --> 00:13:45.960 align:middle line:90% conditional probability. 00:13:45.960 --> 00:13:49.620 align:middle line:84% Given that right now I am at state i what is the 00:13:49.620 --> 00:13:55.650 align:middle line:84% probability that next time I find myself at state j? 00:13:55.650 --> 00:14:00.100 align:middle line:84% So given that right now I am at state 3, P31 is the 00:14:00.100 --> 00:14:02.090 align:middle line:84% probability that the next time I'm going to find 00:14:02.090 --> 00:14:04.740 align:middle line:90% myself at state 1. 00:14:04.740 --> 00:14:09.340 align:middle line:84% Similarly here, we would have a probability P3i, which is 00:14:09.340 --> 00:14:12.710 align:middle line:84% the probability that given that right now I'm at state 3, 00:14:12.710 --> 00:14:17.680 align:middle line:84% next time I'm going to find myself at state i. 00:14:17.680 --> 00:14:21.390 align:middle line:84% Now one can write such conditional probabilities down 00:14:21.390 --> 00:14:25.110 align:middle line:84% in principle, but we need to make-- 00:14:25.110 --> 00:14:29.040 align:middle line:84% so you might think of this as a definition here, but we need 00:14:29.040 --> 00:14:34.050 align:middle line:84% to make one additional big assumption, and this is the 00:14:34.050 --> 00:14:36.360 align:middle line:84% assumption that to make a process 00:14:36.360 --> 00:14:38.540 align:middle line:90% to be a Markov process. 00:14:38.540 --> 00:14:41.210 align:middle line:84% This is the so-called Markov property, and 00:14:41.210 --> 00:14:43.770 align:middle line:90% here's what it says. 00:14:43.770 --> 00:14:46.760 align:middle line:84% Let me describe it first in words here. 00:14:46.760 --> 00:14:52.230 align:middle line:84% Every time that I find myself at state 3, the probability 00:14:52.230 --> 00:14:56.380 align:middle line:84% that next time I'm going to find myself at state 1 is this 00:14:56.380 --> 00:15:00.890 align:middle line:84% particular number, no matter how I got there. 00:15:00.890 --> 00:15:04.870 align:middle line:84% That is, this transition probability is not affected by 00:15:04.870 --> 00:15:06.560 align:middle line:90% the past of the process. 00:15:06.560 --> 00:15:11.930 align:middle line:84% It doesn't care about what path I used to find 00:15:11.930 --> 00:15:14.150 align:middle line:90% myself at state 3. 00:15:14.150 --> 00:15:17.060 align:middle line:84% Mathematically, it means the following. 00:15:17.060 --> 00:15:19.700 align:middle line:84% You have this transition probability that from state i 00:15:19.700 --> 00:15:21.480 align:middle line:90% jump to state j. 00:15:21.480 --> 00:15:24.530 align:middle line:84% Suppose that I gave you some additional information, that I 00:15:24.530 --> 00:15:27.780 align:middle line:84% told you everything else that happened in the past of the 00:15:27.780 --> 00:15:30.410 align:middle line:84% process, everything that happened, how did you 00:15:30.410 --> 00:15:32.570 align:middle line:90% get to state i? 00:15:32.570 --> 00:15:35.550 align:middle line:84% The assumption we're making is that this information about 00:15:35.550 --> 00:15:39.940 align:middle line:84% the past has no bearing in making predictions about the 00:15:39.940 --> 00:15:44.890 align:middle line:84% future, as long as you know where you are right now. 00:15:44.890 --> 00:15:49.300 align:middle line:84% So if I tell you, right now, you are at state i, and by the 00:15:49.300 --> 00:15:53.010 align:middle line:84% way, you got there by following a particular path, 00:15:53.010 --> 00:15:56.940 align:middle line:84% you can ignore the extra information of the particular 00:15:56.940 --> 00:15:58.410 align:middle line:90% path that you followed. 00:15:58.410 --> 00:16:01.610 align:middle line:84% You only take into account where you are right now. 00:16:01.610 --> 00:16:05.800 align:middle line:84% So every time you find yourself at that state, no 00:16:05.800 --> 00:16:10.050 align:middle line:84% matter how you got there, you will find yourself next time 00:16:10.050 --> 00:16:12.980 align:middle line:84% at state 1 with probability P31. 00:16:12.980 --> 00:16:17.740 align:middle line:84% So the past has no bearing into the future, as long as 00:16:17.740 --> 00:16:21.600 align:middle line:84% you know where you are sitting right now. 00:16:21.600 --> 00:16:27.390 align:middle line:84% For this property to happen, you need to choose your state 00:16:27.390 --> 00:16:29.650 align:middle line:90% carefully in the right way. 00:16:29.650 --> 00:16:32.950 align:middle line:84% In that sense, the states needs to include any 00:16:32.950 --> 00:16:36.310 align:middle line:84% information that's relevant about the 00:16:36.310 --> 00:16:38.280 align:middle line:90% future of the system. 00:16:38.280 --> 00:16:41.580 align:middle line:84% Anything that's not in the state is not going to play a 00:16:41.580 --> 00:16:45.340 align:middle line:84% role, but the state needs to have all the information 00:16:45.340 --> 00:16:48.580 align:middle line:84% that's relevant in determining what kind of transitions are 00:16:48.580 --> 00:16:50.080 align:middle line:90% going to happen next. 00:16:50.080 --> 00:16:54.690 align:middle line:84% So to take an example, before you go to Markov process, just 00:16:54.690 --> 00:16:57.660 align:middle line:84% from the deterministic world, if you have a ball that's 00:16:57.660 --> 00:17:01.630 align:middle line:84% flying up in the air, and you want to make predictions about 00:17:01.630 --> 00:17:02.730 align:middle line:90% the future. 00:17:02.730 --> 00:17:06.369 align:middle line:84% If I tell you that the state of the ball is the position of 00:17:06.369 --> 00:17:11.579 align:middle line:84% the ball at the particular time, is that enough for you 00:17:11.579 --> 00:17:15.240 align:middle line:84% to make predictions where the ball is going to go next? 00:17:15.240 --> 00:17:15.700 align:middle line:90% No. 00:17:15.700 --> 00:17:19.460 align:middle line:84% You need to know both the position and the velocity. 00:17:19.460 --> 00:17:21.710 align:middle line:84% If you know position and velocity, you can make 00:17:21.710 --> 00:17:23.490 align:middle line:90% predictions about the future. 00:17:23.490 --> 00:17:27.520 align:middle line:84% So the state of a ball that's flying is position together 00:17:27.520 --> 00:17:29.450 align:middle line:90% with velocity. 00:17:29.450 --> 00:17:32.430 align:middle line:84% If you were to just take position, that would not be 00:17:32.430 --> 00:17:36.290 align:middle line:84% enough information, because if I tell you current position, 00:17:36.290 --> 00:17:39.640 align:middle line:84% and then I tell you past position, you could use the 00:17:39.640 --> 00:17:42.120 align:middle line:84% information from the past position to complete the 00:17:42.120 --> 00:17:43.930 align:middle line:84% trajectory and to make the prediction. 00:17:43.930 --> 00:17:47.870 align:middle line:84% So information from the past is useful if you don't know 00:17:47.870 --> 00:17:48.580 align:middle line:90% the velocity. 00:17:48.580 --> 00:17:53.650 align:middle line:84% But if both position and velocity, you don't care how 00:17:53.650 --> 00:17:56.220 align:middle line:84% you got there, or what time you started. 00:17:56.220 --> 00:17:58.660 align:middle line:84% From position and velocity, you can make predictions about 00:17:58.660 --> 00:17:59.800 align:middle line:90% the future. 00:17:59.800 --> 00:18:04.330 align:middle line:84% So there's a certain art, or a certain element of thinking, a 00:18:04.330 --> 00:18:07.400 align:middle line:84% non-mechanical aspect into problems of this kind, to 00:18:07.400 --> 00:18:11.840 align:middle line:84% figure out which is the right state variable. 00:18:11.840 --> 00:18:14.670 align:middle line:84% When you define the state of your system, you need to 00:18:14.670 --> 00:18:19.870 align:middle line:84% define it in such a way that includes all information that 00:18:19.870 --> 00:18:23.735 align:middle line:84% has been accumulated that has some relevance for the future. 00:18:23.735 --> 00:18:27.380 align:middle line:90% 00:18:27.380 --> 00:18:31.250 align:middle line:84% So the general process for coming up with a Markov model 00:18:31.250 --> 00:18:34.970 align:middle line:84% is to first make this big decision of what your state 00:18:34.970 --> 00:18:37.480 align:middle line:90% variable is going to be. 00:18:37.480 --> 00:18:41.570 align:middle line:84% Then you write down if it may be a picture of 00:18:41.570 --> 00:18:43.150 align:middle line:90% the different states. 00:18:43.150 --> 00:18:45.720 align:middle line:84% Then you identify the possible transitions. 00:18:45.720 --> 00:18:48.810 align:middle line:84% So sometimes the diagram that you're going to have will not 00:18:48.810 --> 00:18:50.970 align:middle line:90% include all the possible arcs. 00:18:50.970 --> 00:18:54.040 align:middle line:84% You would only show those arcs that correspond to transitions 00:18:54.040 --> 00:18:54.770 align:middle line:90% that are possible. 00:18:54.770 --> 00:18:57.850 align:middle line:84% For example, in the supermarket example, we did 00:18:57.850 --> 00:19:01.660 align:middle line:84% not have a transition from state 2 to state 5, because 00:19:01.660 --> 00:19:02.590 align:middle line:90% that cannot happen. 00:19:02.590 --> 00:19:05.360 align:middle line:84% You can only have 1 arrival at any time. 00:19:05.360 --> 00:19:08.330 align:middle line:84% So in the diagram, we only showed the possible 00:19:08.330 --> 00:19:09.280 align:middle line:90% transitions. 00:19:09.280 --> 00:19:12.200 align:middle line:84% And for each of the possible transitions, then you work 00:19:12.200 --> 00:19:15.060 align:middle line:84% with the description of the model to figure out the 00:19:15.060 --> 00:19:17.380 align:middle line:84% correct transition probability. 00:19:17.380 --> 00:19:21.090 align:middle line:84% So you got the diagram by writing down transition 00:19:21.090 --> 00:19:22.340 align:middle line:90% probabilities. 00:19:22.340 --> 00:19:26.890 align:middle line:90% 00:19:26.890 --> 00:19:30.930 align:middle line:84% OK, so suppose you got your Markov model. 00:19:30.930 --> 00:19:32.900 align:middle line:90% What will you do with it? 00:19:32.900 --> 00:19:34.900 align:middle line:84% Well, what do we need models for? 00:19:34.900 --> 00:19:38.580 align:middle line:84% We need models in order to make predictions, to make 00:19:38.580 --> 00:19:39.890 align:middle line:90% probabilistic predictions. 00:19:39.890 --> 00:19:42.750 align:middle line:84% So for example, I tell you that the process started in 00:19:42.750 --> 00:19:43.790 align:middle line:90% that state. 00:19:43.790 --> 00:19:46.070 align:middle line:90% You let it run for some time. 00:19:46.070 --> 00:19:49.980 align:middle line:84% Where do you think it's going to be 10 time steps from now? 00:19:49.980 --> 00:19:52.540 align:middle line:84% That's a question that you might want to answer. 00:19:52.540 --> 00:19:55.660 align:middle line:84% Since the process is random, there's no way for you to tell 00:19:55.660 --> 00:19:58.610 align:middle line:84% me exactly where it's going to be. 00:19:58.610 --> 00:20:00.480 align:middle line:84% But maybe you can give me probabilities. 00:20:00.480 --> 00:20:02.880 align:middle line:84% You can tell me, with so much probability, the 00:20:02.880 --> 00:20:04.240 align:middle line:90% state would be there. 00:20:04.240 --> 00:20:06.080 align:middle line:84% With so much probability, the state would be 00:20:06.080 --> 00:20:07.680 align:middle line:90% there, and so on. 00:20:07.680 --> 00:20:12.010 align:middle line:84% So our first exercise is to calculate those probabilities 00:20:12.010 --> 00:20:16.720 align:middle line:84% about what may happen to the process a number of steps in 00:20:16.720 --> 00:20:18.790 align:middle line:90% the future. 00:20:18.790 --> 00:20:21.800 align:middle line:84% It's handy to have some notation in here. 00:20:21.800 --> 00:20:25.700 align:middle line:84% So somebody tells us that this process starts at the 00:20:25.700 --> 00:20:27.560 align:middle line:90% particular state i. 00:20:27.560 --> 00:20:31.800 align:middle line:84% We let the process run for n transitions. 00:20:31.800 --> 00:20:36.190 align:middle line:84% It may land at some state j, but that state j at which it's 00:20:36.190 --> 00:20:38.060 align:middle line:84% going to land is going to be random. 00:20:38.060 --> 00:20:40.440 align:middle line:84% So we want to give probabilities. 00:20:40.440 --> 00:20:44.750 align:middle line:84% Tell me, with what probability the state, n times steps 00:20:44.750 --> 00:20:49.100 align:middle line:84% later, is going to be that particular state j? 00:20:49.100 --> 00:20:54.830 align:middle line:84% The shorthand notation is to use this symbol here for the 00:20:54.830 --> 00:20:58.730 align:middle line:84% n-step transition probabilities that you find 00:20:58.730 --> 00:21:02.610 align:middle line:84% yourself at state j given that you started at state i. 00:21:02.610 --> 00:21:05.930 align:middle line:84% So the way these two indices are ordered, the way to think 00:21:05.930 --> 00:21:09.130 align:middle line:84% about them is that from i, you go to j. 00:21:09.130 --> 00:21:13.040 align:middle line:84% So the probability that from i you go to j if you have n 00:21:13.040 --> 00:21:16.210 align:middle line:90% steps in front of you. 00:21:16.210 --> 00:21:18.890 align:middle line:84% Some of these transition probabilities are, of course 00:21:18.890 --> 00:21:20.190 align:middle line:90% easy to write. 00:21:20.190 --> 00:21:29.530 align:middle line:84% For example, in 0 transitions, you're going to be exactly 00:21:29.530 --> 00:21:30.860 align:middle line:90% where you started. 00:21:30.860 --> 00:21:35.590 align:middle line:84% So this probability is going to be equal to 1 if i is equal 00:21:35.590 --> 00:21:40.870 align:middle line:84% to j, And 0 if i is different than j. 00:21:40.870 --> 00:21:43.360 align:middle line:84% That's an easy one to write down. 00:21:43.360 --> 00:21:48.250 align:middle line:84% If you have only 1 transition, what's the probability that 1 00:21:48.250 --> 00:21:51.740 align:middle line:84% step later you find yourself in state j given that you 00:21:51.740 --> 00:21:54.310 align:middle line:90% started at state i? 00:21:54.310 --> 00:21:56.830 align:middle line:90% What is this? 00:21:56.830 --> 00:22:00.450 align:middle line:84% These are just the ordinary 1-step transition 00:22:00.450 --> 00:22:03.180 align:middle line:84% probabilities that we are given in the description of 00:22:03.180 --> 00:22:04.780 align:middle line:90% the problem. 00:22:04.780 --> 00:22:08.965 align:middle line:84% So by definition, the 1-step transition probabilities are 00:22:08.965 --> 00:22:10.215 align:middle line:90% of this form. 00:22:10.215 --> 00:22:14.070 align:middle line:90% 00:22:14.070 --> 00:22:17.980 align:middle line:84% This equality is correct just because of the way that we 00:22:17.980 --> 00:22:20.680 align:middle line:90% defined those two quantities. 00:22:20.680 --> 00:22:24.670 align:middle line:84% Now we want to say something about the n-step transition 00:22:24.670 --> 00:22:28.760 align:middle line:84% probabilities when n is a bigger number. 00:22:28.760 --> 00:22:31.320 align:middle line:90% 00:22:31.320 --> 00:22:31.700 align:middle line:90% OK. 00:22:31.700 --> 00:22:36.450 align:middle line:84% So here, we're going to use the total probability theorem. 00:22:36.450 --> 00:22:39.750 align:middle line:84% So we're going to condition in two different scenarios, and 00:22:39.750 --> 00:22:43.580 align:middle line:84% break up the calculation of this quantity, by considering 00:22:43.580 --> 00:22:46.850 align:middle line:84% the different ways that this event can happen. 00:22:46.850 --> 00:22:49.110 align:middle line:84% So what is the event of interest? 00:22:49.110 --> 00:22:51.040 align:middle line:84% The event of interest is the following. 00:22:51.040 --> 00:22:54.070 align:middle line:90% At time 0 we start i. 00:22:54.070 --> 00:22:57.310 align:middle line:84% We are interested in landing at time n at the 00:22:57.310 --> 00:22:59.640 align:middle line:90% particular state j. 00:22:59.640 --> 00:23:03.860 align:middle line:84% Now this event can happen in several different ways, in 00:23:03.860 --> 00:23:05.120 align:middle line:90% lots of different ways. 00:23:05.120 --> 00:23:08.630 align:middle line:84% But let us group them into subgroups. 00:23:08.630 --> 00:23:12.640 align:middle line:84% One group, or one sort of scenario, is the following. 00:23:12.640 --> 00:23:18.200 align:middle line:84% During the first n-1 time steps, things happen, and 00:23:18.200 --> 00:23:20.750 align:middle line:90% somehow you end up at state 1. 00:23:20.750 --> 00:23:24.180 align:middle line:84% And then from state 1, in the next time step you make a 00:23:24.180 --> 00:23:27.160 align:middle line:90% transition to state j. 00:23:27.160 --> 00:23:32.770 align:middle line:84% This particular arc here actually corresponds to lots 00:23:32.770 --> 00:23:36.600 align:middle line:84% and lots of different possible scenarios, or different spots, 00:23:36.600 --> 00:23:38.110 align:middle line:90% or different transitions. 00:23:38.110 --> 00:23:43.770 align:middle line:84% In n-1 time steps, there's lots of possible ways by which 00:23:43.770 --> 00:23:46.010 align:middle line:90% you could end up at state 1. 00:23:46.010 --> 00:23:48.650 align:middle line:84% Different paths through the state space. 00:23:48.650 --> 00:23:51.630 align:middle line:84% But all of them together collectively have a 00:23:51.630 --> 00:23:55.360 align:middle line:84% probability, which is the (n-1)-step transition 00:23:55.360 --> 00:24:02.200 align:middle line:84% probability, that from state i, you end up at state 1 00:24:02.200 --> 00:24:05.960 align:middle line:84% And then there's other possible scenarios. 00:24:05.960 --> 00:24:10.120 align:middle line:84% Perhaps in the first n-1 time steps, you follow the 00:24:10.120 --> 00:24:13.370 align:middle line:84% trajectory that took you at state m. 00:24:13.370 --> 00:24:17.430 align:middle line:84% And then from state m, you did this transition, and you ended 00:24:17.430 --> 00:24:18.980 align:middle line:90% up at state j. 00:24:18.980 --> 00:24:22.580 align:middle line:84% So this diagram breaks up the set of all possible 00:24:22.580 --> 00:24:27.360 align:middle line:84% trajectories from i to j into different collections, where 00:24:27.360 --> 00:24:31.340 align:middle line:84% each collection has to do with which one happens to be the 00:24:31.340 --> 00:24:37.070 align:middle line:84% state just before the last time step, just before time n. 00:24:37.070 --> 00:24:40.040 align:middle line:84% And we're going to condition on the state at time n-1. 00:24:40.040 --> 00:24:42.620 align:middle line:90% 00:24:42.620 --> 00:24:48.180 align:middle line:84% So the total probability of ending up at state j is the 00:24:48.180 --> 00:24:53.090 align:middle line:84% sum of the probabilities of the different scenarios -- 00:24:53.090 --> 00:24:56.380 align:middle line:84% the different ways that you can get to state j. 00:24:56.380 --> 00:25:00.650 align:middle line:84% If we look at that type of scenario, what's the 00:25:00.650 --> 00:25:03.030 align:middle line:84% probability of that scenario happening? 00:25:03.030 --> 00:25:08.290 align:middle line:84% With probability Ri1(n-1), I find myself at 00:25:08.290 --> 00:25:10.810 align:middle line:90% state 1 at time n-1. 00:25:10.810 --> 00:25:15.000 align:middle line:84% This is just by the definition of these multi-step transition 00:25:15.000 --> 00:25:16.160 align:middle line:90% probabilities. 00:25:16.160 --> 00:25:17.990 align:middle line:84% This is the number of transitions. 00:25:17.990 --> 00:25:22.690 align:middle line:84% The probability that from state i, I end up at state 1. 00:25:22.690 --> 00:25:27.130 align:middle line:84% And then given that I found myself at state 1, with 00:25:27.130 --> 00:25:31.350 align:middle line:84% probability P1j, that's the transition probability, next 00:25:31.350 --> 00:25:34.710 align:middle line:84% time I'm going to find myself at state j. 00:25:34.710 --> 00:25:39.610 align:middle line:84% So the product of these two is the total probability of my 00:25:39.610 --> 00:25:43.500 align:middle line:84% getting from state i to state j through state 00:25:43.500 --> 00:25:47.340 align:middle line:90% 1 at the time before. 00:25:47.340 --> 00:25:53.160 align:middle line:84% Now where exactly did we use the Markov assumption here? 00:25:53.160 --> 00:25:57.750 align:middle line:84% No matter which particular path we used to get from i to 00:25:57.750 --> 00:26:01.660 align:middle line:84% state 1, the probability that next I'm going to make this 00:26:01.660 --> 00:26:05.510 align:middle line:84% transition is that same number, P1j. 00:26:05.510 --> 00:26:09.170 align:middle line:84% So that number does not depend on the particular path that I 00:26:09.170 --> 00:26:11.240 align:middle line:84% followed in order to get there. 00:26:11.240 --> 00:26:15.090 align:middle line:84% If we didn't have the Markov assumption, we should have 00:26:15.090 --> 00:26:18.610 align:middle line:84% considered all possible individual trajectories here, 00:26:18.610 --> 00:26:21.360 align:middle line:84% and then we would need to use the transition probability 00:26:21.360 --> 00:26:23.840 align:middle line:84% that corresponds to that particular trajectory. 00:26:23.840 --> 00:26:26.130 align:middle line:84% But because of the Markov assumption, the only thing 00:26:26.130 --> 00:26:29.930 align:middle line:84% that matters is that right now we are at state 1. 00:26:29.930 --> 00:26:33.100 align:middle line:84% It does not matter how we got there. 00:26:33.100 --> 00:26:37.240 align:middle line:84% So now once you see this scenario, then this scenario, 00:26:37.240 --> 00:26:40.160 align:middle line:84% and that scenario, and you add the probabilities of these 00:26:40.160 --> 00:26:43.820 align:middle line:84% different scenarios, you end up with this formula here, 00:26:43.820 --> 00:26:45.540 align:middle line:90% which is a recursion. 00:26:45.540 --> 00:26:49.810 align:middle line:84% It tells us that once you have computed the (n-1)-step 00:26:49.810 --> 00:26:53.830 align:middle line:84% transition probabilities, then you can compute also the 00:26:53.830 --> 00:26:55.990 align:middle line:84% n-step transition probabilities. 00:26:55.990 --> 00:27:01.390 align:middle line:84% This is a recursion that you execute or you run for all i's 00:27:01.390 --> 00:27:03.320 align:middle line:90% and j's simultaneously. 00:27:03.320 --> 00:27:04.180 align:middle line:90% That is fixed. 00:27:04.180 --> 00:27:08.280 align:middle line:84% And for a particular n, you calculate this quantity for 00:27:08.280 --> 00:27:10.620 align:middle line:90% all possible i's, j's, k's. 00:27:10.620 --> 00:27:13.710 align:middle line:84% You have all of those quantities, and then you use 00:27:13.710 --> 00:27:16.730 align:middle line:84% this equation to find those numbers again for all the 00:27:16.730 --> 00:27:20.340 align:middle line:90% possible i's and j's. 00:27:20.340 --> 00:27:26.620 align:middle line:84% Now this is formula which is always true, and there's a big 00:27:26.620 --> 00:27:28.810 align:middle line:90% idea behind the formula. 00:27:28.810 --> 00:27:32.050 align:middle line:84% And now there's variations of this formula, depending on 00:27:32.050 --> 00:27:33.610 align:middle line:84% whether you're interested in something 00:27:33.610 --> 00:27:35.200 align:middle line:90% that's slightly different. 00:27:35.200 --> 00:27:42.070 align:middle line:84% So for example, if you were to have a random initial state, 00:27:42.070 --> 00:27:44.850 align:middle line:84% somebody gives you the probability distribution of 00:27:44.850 --> 00:27:48.300 align:middle line:84% the initial state, so you're told that with probability 00:27:48.300 --> 00:27:51.250 align:middle line:84% such and such, you're going to start at state 1. 00:27:51.250 --> 00:27:52.760 align:middle line:84% With that probability, you're going to start at 00:27:52.760 --> 00:27:54.200 align:middle line:90% state 2, and so on. 00:27:54.200 --> 00:27:56.560 align:middle line:84% And you want to find the probability at the time n you 00:27:56.560 --> 00:27:58.530 align:middle line:90% find yourself at state j. 00:27:58.530 --> 00:28:01.880 align:middle line:84% Well again, total probability theorem, you condition on the 00:28:01.880 --> 00:28:03.120 align:middle line:90% initial state. 00:28:03.120 --> 00:28:05.430 align:middle line:84% With this probability you find yourself at that particular 00:28:05.430 --> 00:28:08.570 align:middle line:84% initial state, and given that this is your initial state, 00:28:08.570 --> 00:28:11.840 align:middle line:84% this is the probability that n time steps later you find 00:28:11.840 --> 00:28:14.980 align:middle line:90% yourself at state j. 00:28:14.980 --> 00:28:20.080 align:middle line:84% Now building again on the same idea, you can run every 00:28:20.080 --> 00:28:23.330 align:middle line:84% recursion of this kind by conditioning 00:28:23.330 --> 00:28:24.950 align:middle line:90% at different times. 00:28:24.950 --> 00:28:26.200 align:middle line:90% So here's a variation. 00:28:26.200 --> 00:28:29.260 align:middle line:90% 00:28:29.260 --> 00:28:31.520 align:middle line:90% You start at state i. 00:28:31.520 --> 00:28:36.240 align:middle line:84% After 1 time step, you find yourself at state 1, with 00:28:36.240 --> 00:28:40.630 align:middle line:84% probability pi1, and you find yourself at state m with 00:28:40.630 --> 00:28:43.930 align:middle line:90% probability Pim. 00:28:43.930 --> 00:28:49.250 align:middle line:84% And once that happens, then you're going to follow some 00:28:49.250 --> 00:28:51.070 align:middle line:90% trajectories. 00:28:51.070 --> 00:28:54.390 align:middle line:84% And there is a possibility that you're going to end up at 00:28:54.390 --> 00:28:58.285 align:middle line:90% state j after n-1 time steps. 00:28:58.285 --> 00:29:02.160 align:middle line:90% 00:29:02.160 --> 00:29:05.160 align:middle line:84% This scenario can happen in many possible ways. 00:29:05.160 --> 00:29:08.130 align:middle line:84% There's lots of possible paths from state 1 to state j. 00:29:08.130 --> 00:29:12.680 align:middle line:84% There's many paths from state 1 to state j. 00:29:12.680 --> 00:29:15.940 align:middle line:84% What is the collective probability of all these 00:29:15.940 --> 00:29:17.190 align:middle line:90% transitions? 00:29:17.190 --> 00:29:19.250 align:middle line:90% 00:29:19.250 --> 00:29:23.150 align:middle line:84% This is the event that, starting from state 1, I end 00:29:23.150 --> 00:29:27.560 align:middle line:84% up at state j in n-1 time steps. 00:29:27.560 --> 00:29:34.240 align:middle line:84% So this one has here probability R1j of n-1. 00:29:34.240 --> 00:29:37.980 align:middle line:90% And similarly down here. 00:29:37.980 --> 00:29:41.350 align:middle line:84% And then by using the same way of thinking as before, we get 00:29:41.350 --> 00:29:48.650 align:middle line:84% the formula that Rij(n) is the sum over all k's of Pik, and 00:29:48.650 --> 00:29:49.940 align:middle line:90% then the Rkj(n-1). 00:29:49.940 --> 00:29:54.800 align:middle line:90% 00:29:54.800 --> 00:29:59.050 align:middle line:84% So this formula looks almost the same as this one, but it's 00:29:59.050 --> 00:30:00.810 align:middle line:90% actually different. 00:30:00.810 --> 00:30:05.570 align:middle line:84% The indices and the way things work out are a bit different, 00:30:05.570 --> 00:30:08.500 align:middle line:84% but the basic idea is the same. 00:30:08.500 --> 00:30:10.940 align:middle line:84% Here we use the total probability theory by 00:30:10.940 --> 00:30:15.770 align:middle line:84% conditioning on the state just 1 step before the end of our 00:30:15.770 --> 00:30:17.020 align:middle line:90% time horizon. 00:30:17.020 --> 00:30:21.260 align:middle line:84% Here we use total probability theorem by conditioning on the 00:30:21.260 --> 00:30:24.300 align:middle line:84% state right after the first transition. 00:30:24.300 --> 00:30:28.340 align:middle line:84% So this generally idea has different variations. 00:30:28.340 --> 00:30:30.920 align:middle line:84% They're all valid, and depending on the context that 00:30:30.920 --> 00:30:34.600 align:middle line:84% you're dealing with, you might want to work with one of these 00:30:34.600 --> 00:30:37.130 align:middle line:90% or another. 00:30:37.130 --> 00:30:40.070 align:middle line:84% So let's illustrate these calculations 00:30:40.070 --> 00:30:42.090 align:middle line:90% in terms of an example. 00:30:42.090 --> 00:30:46.910 align:middle line:84% So in this example, we just have 2 states, and somebody 00:30:46.910 --> 00:30:49.510 align:middle line:84% gives us transition probabilities to be those 00:30:49.510 --> 00:30:51.740 align:middle line:90% particular numbers. 00:30:51.740 --> 00:30:55.530 align:middle line:84% Let's write down the equations. 00:30:55.530 --> 00:31:02.760 align:middle line:84% So the probability that starting from state 1, I find 00:31:02.760 --> 00:31:06.580 align:middle line:84% myself at state 1 n time steps later. 00:31:06.580 --> 00:31:09.270 align:middle line:90% This can happen in 2 ways. 00:31:09.270 --> 00:31:15.440 align:middle line:84% At time n-1, I might find myself at state 2. 00:31:15.440 --> 00:31:21.370 align:middle line:84% And then from state 2, I make a transition back to state 1, 00:31:21.370 --> 00:31:24.050 align:middle line:84% which happens with probability-- 00:31:24.050 --> 00:31:25.260 align:middle line:90% why'd I put 2 there -- 00:31:25.260 --> 00:31:27.890 align:middle line:90% anyway, 0.2. 00:31:27.890 --> 00:31:32.230 align:middle line:84% And another way is that from state 1, I go to state 1 in 00:31:32.230 --> 00:31:38.170 align:middle line:84% n-1 steps, and then from state 1 I stay where I am, which 00:31:38.170 --> 00:31:42.830 align:middle line:90% happens with probability 0.5. 00:31:42.830 --> 00:31:44.810 align:middle line:90% So this is for R11(n). 00:31:44.810 --> 00:31:48.730 align:middle line:90% 00:31:48.730 --> 00:31:54.740 align:middle line:84% Now R12(n), we can write a similar 00:31:54.740 --> 00:31:56.780 align:middle line:90% recursion for this one. 00:31:56.780 --> 00:32:00.010 align:middle line:84% On the other hand, seems these are probabilities. 00:32:00.010 --> 00:32:02.270 align:middle line:84% The state at time n is going to be either 00:32:02.270 --> 00:32:04.420 align:middle line:90% state 1 or state 2. 00:32:04.420 --> 00:32:09.270 align:middle line:84% So these 2 numbers need to add to 1, so we can just write 00:32:09.270 --> 00:32:10.520 align:middle line:90% this as 1 - R11(n). 00:32:10.520 --> 00:32:13.080 align:middle line:90% 00:32:13.080 --> 00:32:19.660 align:middle line:84% And this is an enough of a recursion to propagate R11 and 00:32:19.660 --> 00:32:22.010 align:middle line:90% R12 as time goes on. 00:32:22.010 --> 00:32:24.830 align:middle line:90% 00:32:24.830 --> 00:32:29.105 align:middle line:84% So after n-1 transitions, either I find myself in state 00:32:29.105 --> 00:32:33.910 align:middle line:84% 2, and then there's a point to transition that I go to 1, or 00:32:33.910 --> 00:32:37.810 align:middle line:84% I find myself in state 1, which with that probability, 00:32:37.810 --> 00:32:41.230 align:middle line:84% and from there, I have probability 0.5 of staying 00:32:41.230 --> 00:32:42.880 align:middle line:90% where I am. 00:32:42.880 --> 00:32:45.830 align:middle line:90% Now let's start calculating. 00:32:45.830 --> 00:32:49.500 align:middle line:84% As we discussed before, if I start at state 1, after 0 00:32:49.500 --> 00:32:53.320 align:middle line:84% transitions I'm certain to be at state , and I'm certain not 00:32:53.320 --> 00:32:55.100 align:middle line:90% to be at state 1. 00:32:55.100 --> 00:32:59.390 align:middle line:84% If I start from state 1, I'm certain to not to be at state 00:32:59.390 --> 00:33:01.980 align:middle line:84% at that time, and I'm certain that I am right 00:33:01.980 --> 00:33:03.520 align:middle line:90% now, it's state 1. 00:33:03.520 --> 00:33:09.970 align:middle line:84% After I make transition, starting from state 1, there's 00:33:09.970 --> 00:33:13.790 align:middle line:84% probability 0.5 that I stay at state 1. 00:33:13.790 --> 00:33:17.830 align:middle line:84% And there's probability 0.5 that I stay at state 2. 00:33:17.830 --> 00:33:22.060 align:middle line:84% If I were to start from state 2, the probability that I go 00:33:22.060 --> 00:33:25.690 align:middle line:84% to 1 in 1 time step is this transition that has 00:33:25.690 --> 00:33:30.160 align:middle line:84% probability 0.2, and the other 0.8. 00:33:30.160 --> 00:33:30.510 align:middle line:90% OK. 00:33:30.510 --> 00:33:33.850 align:middle line:84% So the calculation now becomes more interesting, if we want 00:33:33.850 --> 00:33:36.920 align:middle line:90% to calculate the next term. 00:33:36.920 --> 00:33:41.890 align:middle line:84% How likely is that at time 2, I find myself at state 1? 00:33:41.890 --> 00:33:44.610 align:middle line:90% 00:33:44.610 --> 00:33:50.060 align:middle line:84% In order to be here at state 1, this can happen in 2 ways. 00:33:50.060 --> 00:33:54.510 align:middle line:84% Either the first transition left me there, and the second 00:33:54.510 --> 00:33:57.620 align:middle line:90% transition is the same. 00:33:57.620 --> 00:34:01.380 align:middle line:84% So these correspond to this 0.5, that the first transition 00:34:01.380 --> 00:34:04.660 align:middle line:84% took me there, and the next transition was 00:34:04.660 --> 00:34:07.140 align:middle line:90% also of the same kind. 00:34:07.140 --> 00:34:08.880 align:middle line:90% That's one possibility. 00:34:08.880 --> 00:34:10.510 align:middle line:90% But there's another scenario. 00:34:10.510 --> 00:34:15.100 align:middle line:84% In order to be at state 1 at time 2 -- this can 00:34:15.100 --> 00:34:17.239 align:middle line:90% also happen this way. 00:34:17.239 --> 00:34:19.690 align:middle line:84% So that's the event that, after 1 00:34:19.690 --> 00:34:22.449 align:middle line:90% transition, I got there. 00:34:22.449 --> 00:34:26.920 align:middle line:84% And the next transition happened to be this one. 00:34:26.920 --> 00:34:31.020 align:middle line:84% So this corresponds to 0.5 times 0.2. 00:34:31.020 --> 00:34:34.250 align:middle line:84% It corresponds to taking the 1-step transition probability 00:34:34.250 --> 00:34:39.070 align:middle line:84% of getting there, times the probability that from state 2 00:34:39.070 --> 00:34:43.070 align:middle line:84% I move to state 1, which in this case, is 0.2. 00:34:43.070 --> 00:34:47.480 align:middle line:84% So basically we take this number, multiplied with 0.2, 00:34:47.480 --> 00:34:50.250 align:middle line:90% and then add those 2 numbers. 00:34:50.250 --> 00:34:54.090 align:middle line:84% And after you add them, you get 0.35. 00:34:54.090 --> 00:34:59.090 align:middle line:84% And similarly here, you're going to get 0.65. 00:34:59.090 --> 00:35:02.390 align:middle line:84% And now to continue with the recursion, we keep doing the 00:35:02.390 --> 00:35:02.960 align:middle line:90% same thing. 00:35:02.960 --> 00:35:08.290 align:middle line:84% We take this number times 0.5 plus this number times 0.2. 00:35:08.290 --> 00:35:10.780 align:middle line:84% Add them up, you get the next entry. 00:35:10.780 --> 00:35:15.390 align:middle line:84% Keep doing that, keep doing that, and eventually you will 00:35:15.390 --> 00:35:19.970 align:middle line:84% notice that the numbers start settling into a 00:35:19.970 --> 00:35:23.810 align:middle line:90% limiting value at 2/7. 00:35:23.810 --> 00:35:25.690 align:middle line:90% And let's verify this. 00:35:25.690 --> 00:35:29.220 align:middle line:84% If this number is 2/7, what is the next number going to be? 00:35:29.220 --> 00:35:33.720 align:middle line:90% 00:35:33.720 --> 00:35:41.100 align:middle line:84% The next number is going to be 2/7 -- (not 2.7) -- 00:35:41.100 --> 00:35:42.410 align:middle line:90% it's going to be 2/7. 00:35:42.410 --> 00:35:44.870 align:middle line:84% That's the probability that I find myself at that state, 00:35:44.870 --> 00:35:46.890 align:middle line:90% times 0.5-- 00:35:46.890 --> 00:35:51.200 align:middle line:84% that's the next transition that takes me to state 1 -- 00:35:51.200 --> 00:35:53.020 align:middle line:90% plus 5/7-- 00:35:53.020 --> 00:35:56.700 align:middle line:84% that would be the remaining probability that I find myself 00:35:56.700 --> 00:35:58.180 align:middle line:90% in state 2 -- 00:35:58.180 --> 00:36:02.760 align:middle line:90% times 1/5. 00:36:02.760 --> 00:36:06.360 align:middle line:90% 00:36:06.360 --> 00:36:11.210 align:middle line:84% And so that gives me, again, 2/7. 00:36:11.210 --> 00:36:15.840 align:middle line:84% So this calculation basically illustrates, if this number 00:36:15.840 --> 00:36:19.360 align:middle line:84% has become 2/7, then the next number is 00:36:19.360 --> 00:36:21.570 align:middle line:90% also going to be 2/7. 00:36:21.570 --> 00:36:24.260 align:middle line:84% And of course this number here is going to have to be 5/7. 00:36:24.260 --> 00:36:26.900 align:middle line:90% 00:36:26.900 --> 00:36:32.020 align:middle line:84% And this one would have to be again, the same, 5/7. 00:36:32.020 --> 00:36:36.620 align:middle line:84% So the probability that I find myself at state 1, after a 00:36:36.620 --> 00:36:41.830 align:middle line:84% long time has elapsed, settles into some steady state value. 00:36:41.830 --> 00:36:44.140 align:middle line:84% So that's an interesting phenomenon. 00:36:44.140 --> 00:36:46.850 align:middle line:90% We just make this observation. 00:36:46.850 --> 00:36:50.390 align:middle line:84% Now we can also do the calculation about the 00:36:50.390 --> 00:36:53.660 align:middle line:84% probability, starting from state 2. 00:36:53.660 --> 00:36:57.050 align:middle line:84% And here, you do the calculations -- 00:36:57.050 --> 00:36:58.460 align:middle line:90% I'm not going to do them. 00:36:58.460 --> 00:37:02.040 align:middle line:84% But after you do them, you find this probability also 00:37:02.040 --> 00:37:07.405 align:middle line:84% settles to 2/7 and this one also settles to 5/7. 00:37:07.405 --> 00:37:11.130 align:middle line:90% 00:37:11.130 --> 00:37:15.320 align:middle line:84% So these numbers here are the same as those numbers. 00:37:15.320 --> 00:37:19.770 align:middle line:84% What's the difference between these? 00:37:19.770 --> 00:37:24.790 align:middle line:84% This is the probability that I find myself at state 1 given 00:37:24.790 --> 00:37:27.050 align:middle line:90% that I started at 1. 00:37:27.050 --> 00:37:30.890 align:middle line:84% This is the probability that I find myself at state 1 given 00:37:30.890 --> 00:37:34.220 align:middle line:90% that I started at state 2. 00:37:34.220 --> 00:37:39.110 align:middle line:84% These probabilities are the same, no matter where I 00:37:39.110 --> 00:37:40.460 align:middle line:90% started from. 00:37:40.460 --> 00:37:45.790 align:middle line:84% So this numerical example sort of illustrates the idea that 00:37:45.790 --> 00:37:51.070 align:middle line:84% after the chain has run for a long time, what the state of 00:37:51.070 --> 00:37:55.010 align:middle line:84% the chain is, does not care about the initial 00:37:55.010 --> 00:37:56.530 align:middle line:90% state of the chain. 00:37:56.530 --> 00:38:02.590 align:middle line:84% So if you start here, you know that you're going to stay here 00:38:02.590 --> 00:38:05.350 align:middle line:84% for some time, a few transitions, because this 00:38:05.350 --> 00:38:07.090 align:middle line:90% probability is kind of small. 00:38:07.090 --> 00:38:10.440 align:middle line:84% So the initial state does that's tell you something. 00:38:10.440 --> 00:38:13.710 align:middle line:84% But in the very long run, transitions of this kind are 00:38:13.710 --> 00:38:14.340 align:middle line:90% going to happen. 00:38:14.340 --> 00:38:17.510 align:middle line:84% Transitions of that kind are going to happen. 00:38:17.510 --> 00:38:20.920 align:middle line:84% There's a lot of randomness that comes in, and that 00:38:20.920 --> 00:38:24.820 align:middle line:84% randomness washes out any information that could come 00:38:24.820 --> 00:38:28.580 align:middle line:84% from the initial state of the system. 00:38:28.580 --> 00:38:33.290 align:middle line:84% We describe this situation by saying that the Markov chain 00:38:33.290 --> 00:38:37.210 align:middle line:84% eventually enters a steady state. 00:38:37.210 --> 00:38:41.050 align:middle line:84% Where a steady state, what does it mean it? 00:38:41.050 --> 00:38:46.630 align:middle line:84% Does it mean the state itself becomes steady and 00:38:46.630 --> 00:38:48.750 align:middle line:90% stops at one place? 00:38:48.750 --> 00:38:52.490 align:middle line:84% No, the state of the chain keeps jumping forever. 00:38:52.490 --> 00:38:55.380 align:middle line:84% The state of the chain will keep making transitions, will 00:38:55.380 --> 00:38:58.780 align:middle line:84% keep going back and forth between 1 and 2. 00:38:58.780 --> 00:39:02.920 align:middle line:84% So the state itself, the Xn, does not become 00:39:02.920 --> 00:39:04.970 align:middle line:90% steady in any sense. 00:39:04.970 --> 00:39:07.950 align:middle line:84% What becomes steady are the probabilities 00:39:07.950 --> 00:39:09.860 align:middle line:90% that describe Xn. 00:39:09.860 --> 00:39:12.900 align:middle line:84% That is, after a long time elapses, the probability that 00:39:12.900 --> 00:39:19.700 align:middle line:84% you find yourself at state 1 becomes a constant 2/7, and 00:39:19.700 --> 00:39:21.520 align:middle line:84% the probability that you find yourself in 00:39:21.520 --> 00:39:23.810 align:middle line:90% state 2 becomes a constant. 00:39:23.810 --> 00:39:28.000 align:middle line:84% So jumps will keep happening, but at any given time, if you 00:39:28.000 --> 00:39:30.590 align:middle line:84% ask what's the probability that right now I am at state 00:39:30.590 --> 00:39:34.630 align:middle line:84% 1, the answer is going to be 2/7. 00:39:34.630 --> 00:39:37.650 align:middle line:84% Incidentally, do the numbers sort of makes sense? 00:39:37.650 --> 00:39:42.270 align:middle line:84% Why is this number bigger than that number? 00:39:42.270 --> 00:39:46.500 align:middle line:84% Well, this state is a little more sticky than that state. 00:39:46.500 --> 00:39:50.000 align:middle line:84% Once you enter here, it's kind of harder to get out. 00:39:50.000 --> 00:39:53.380 align:middle line:84% So when you enter here, you spend a lot of time here. 00:39:53.380 --> 00:39:56.240 align:middle line:84% This one is easier to get out, because the probability is 00:39:56.240 --> 00:40:00.370 align:middle line:84% 0.5, so when you enter there, you tend to get out faster. 00:40:00.370 --> 00:40:04.150 align:middle line:84% So you keep moving from one to the other, but you tend to 00:40:04.150 --> 00:40:08.510 align:middle line:84% spend more time on that state, and this is reflected in this 00:40:08.510 --> 00:40:10.930 align:middle line:84% probability being bigger than that one. 00:40:10.930 --> 00:40:14.540 align:middle line:84% So no matter where you start, there's 5/7 probability of 00:40:14.540 --> 00:40:18.650 align:middle line:84% being here, 2/7 probability being there. 00:40:18.650 --> 00:40:20.480 align:middle line:84% So there were some really nice things that 00:40:20.480 --> 00:40:24.730 align:middle line:90% happened in this example. 00:40:24.730 --> 00:40:28.830 align:middle line:84% The question is, whether things are always as nice for 00:40:28.830 --> 00:40:30.410 align:middle line:90% general Markov chains. 00:40:30.410 --> 00:40:33.380 align:middle line:84% The two nice things that happened where the following-- 00:40:33.380 --> 00:40:36.020 align:middle line:84% as we keep doing this calculation, this number 00:40:36.020 --> 00:40:37.660 align:middle line:90% settles to something. 00:40:37.660 --> 00:40:39.620 align:middle line:90% The limit exists. 00:40:39.620 --> 00:40:42.520 align:middle line:84% The other thing that happens is that this number is the 00:40:42.520 --> 00:40:45.740 align:middle line:84% same as that number, which means that the initial state 00:40:45.740 --> 00:40:47.280 align:middle line:90% does not matter. 00:40:47.280 --> 00:40:50.130 align:middle line:90% Is this always the case? 00:40:50.130 --> 00:40:54.570 align:middle line:84% Is it always the case that as n goes to infinity, the 00:40:54.570 --> 00:40:58.490 align:middle line:84% transition probabilities converge to something? 00:40:58.490 --> 00:41:02.680 align:middle line:84% And if they do converge to something, is it the case that 00:41:02.680 --> 00:41:07.830 align:middle line:84% the limit is not affected by the initial state i at which 00:41:07.830 --> 00:41:09.400 align:middle line:90% the chain started? 00:41:09.400 --> 00:41:12.970 align:middle line:84% So mathematically speaking, the question we are raising is 00:41:12.970 --> 00:41:19.180 align:middle line:84% whether Rij(n) converges to something. 00:41:19.180 --> 00:41:25.440 align:middle line:84% And whether that something to which it converges to has only 00:41:25.440 --> 00:41:26.780 align:middle line:90% to do with j. 00:41:26.780 --> 00:41:30.680 align:middle line:84% It's the probability that you find yourself at state j, and 00:41:30.680 --> 00:41:34.170 align:middle line:84% that probability doesn't care about the initial state. 00:41:34.170 --> 00:41:36.950 align:middle line:84% So it's the question of whether the initial state gets 00:41:36.950 --> 00:41:39.970 align:middle line:90% forgotten in the long run. 00:41:39.970 --> 00:41:46.260 align:middle line:84% So the answer is that usually, or for nice chains, both of 00:41:46.260 --> 00:41:49.420 align:middle line:90% these things will be true. 00:41:49.420 --> 00:41:51.500 align:middle line:84% You get the limit which does not depend 00:41:51.500 --> 00:41:52.900 align:middle line:90% on the initial state. 00:41:52.900 --> 00:41:59.020 align:middle line:84% But if your chain has some peculiar or unique structure, 00:41:59.020 --> 00:42:01.160 align:middle line:90% this might not happen. 00:42:01.160 --> 00:42:03.905 align:middle line:84% So let's think first about the issue of convergence. 00:42:03.905 --> 00:42:06.980 align:middle line:90% 00:42:06.980 --> 00:42:12.720 align:middle line:84% So convergence, as n goes to infinity at a steady value, 00:42:12.720 --> 00:42:14.520 align:middle line:90% really means the following. 00:42:14.520 --> 00:42:18.710 align:middle line:84% If I tell you a lot of time has passed, then you tell me, 00:42:18.710 --> 00:42:21.220 align:middle line:84% OK, the state of the probabilities are equal to 00:42:21.220 --> 00:42:25.630 align:middle line:84% that value without having to consult your clock. 00:42:25.630 --> 00:42:29.340 align:middle line:84% If you don't have convergence, it means that Rij can keep 00:42:29.340 --> 00:42:32.230 align:middle line:84% going up and down, without settling to something. 00:42:32.230 --> 00:42:35.640 align:middle line:84% So in order for you to tell me the value of Rij, you need to 00:42:35.640 --> 00:42:38.200 align:middle line:84% consult your clock to check if, right now, 00:42:38.200 --> 00:42:40.670 align:middle line:90% it's up or is it down. 00:42:40.670 --> 00:42:43.230 align:middle line:84% So there's some kind of periodic behavior that you 00:42:43.230 --> 00:42:46.020 align:middle line:84% might get when you do not get convergence, and this example 00:42:46.020 --> 00:42:47.490 align:middle line:90% here illustrates it. 00:42:47.490 --> 00:42:50.120 align:middle line:84% So what's happened in this example? 00:42:50.120 --> 00:42:54.470 align:middle line:84% Starting from state 2, next time you go here, or there, 00:42:54.470 --> 00:42:56.760 align:middle line:90% with probability half. 00:42:56.760 --> 00:43:00.820 align:middle line:84% And then next time, no matter where you are, you move back 00:43:00.820 --> 00:43:02.200 align:middle line:90% to state 2. 00:43:02.200 --> 00:43:05.950 align:middle line:84% So this chain has some randomness, but the randomness 00:43:05.950 --> 00:43:08.050 align:middle line:90% is kind of limited type. 00:43:08.050 --> 00:43:09.260 align:middle line:90% You go out, you come in. 00:43:09.260 --> 00:43:10.500 align:middle line:90% You go out, you come in. 00:43:10.500 --> 00:43:14.290 align:middle line:84% So there's a periodic pattern that gets repeated. 00:43:14.290 --> 00:43:19.850 align:middle line:84% It means that if you start at state 2 after an even number 00:43:19.850 --> 00:43:24.100 align:middle line:84% of steps, you are certain to be back at state 2. 00:43:24.100 --> 00:43:26.730 align:middle line:90% So this probability here is 1. 00:43:26.730 --> 00:43:30.430 align:middle line:84% On the other hand, if the number of transitions is odd, 00:43:30.430 --> 00:43:33.900 align:middle line:84% there's no way that you can be at your initial state. 00:43:33.900 --> 00:43:37.150 align:middle line:84% If you start here, at even times you would be here, at 00:43:37.150 --> 00:43:39.570 align:middle line:84% odd times you would be there or there. 00:43:39.570 --> 00:43:42.170 align:middle line:90% So this probability is 0. 00:43:42.170 --> 00:43:46.540 align:middle line:84% As n goes to infinity, these probabilities, the n-step 00:43:46.540 --> 00:43:49.220 align:middle line:84% transition probability does not converge to anything. 00:43:49.220 --> 00:43:51.950 align:middle line:84% It keeps alternating between 0 and 1. 00:43:51.950 --> 00:43:54.090 align:middle line:90% So convergence fails. 00:43:54.090 --> 00:43:57.440 align:middle line:84% This is the main mechanism by which convergence can fail if 00:43:57.440 --> 00:43:59.720 align:middle line:84% your chain has a periodic structure. 00:43:59.720 --> 00:44:03.520 align:middle line:84% And we're going to discuss next time that, if periodicity 00:44:03.520 --> 00:44:08.790 align:middle line:84% absent, then we don't have an issue with convergence. 00:44:08.790 --> 00:44:13.950 align:middle line:84% The second question if we have convergence, whether the 00:44:13.950 --> 00:44:16.850 align:middle line:90% initial state matters or not. 00:44:16.850 --> 00:44:19.560 align:middle line:84% In the previous chain, where you could keep going back and 00:44:19.560 --> 00:44:22.830 align:middle line:84% forth between states 1 and 2 numerically, one finds that 00:44:22.830 --> 00:44:25.040 align:middle line:84% the initial state does not matter. 00:44:25.040 --> 00:44:27.460 align:middle line:84% But you can think of situations where the initial 00:44:27.460 --> 00:44:30.070 align:middle line:90% state does matter. 00:44:30.070 --> 00:44:33.120 align:middle line:90% Look at this chain here. 00:44:33.120 --> 00:44:37.520 align:middle line:84% If you start at state 1, you stay at state 1 forever. 00:44:37.520 --> 00:44:39.840 align:middle line:90% There's no way to escape. 00:44:39.840 --> 00:44:46.040 align:middle line:84% So this means that R11(n) is 1 for all n. 00:44:46.040 --> 00:44:50.390 align:middle line:84% If you start at state 3, you will be moving between stage 3 00:44:50.390 --> 00:44:54.400 align:middle line:84% and 4, but there's no way to go in that direction, so 00:44:54.400 --> 00:44:57.730 align:middle line:84% there's no way that you go to state 1. 00:44:57.730 --> 00:45:01.410 align:middle line:84% And for that reason, R31 is 0 for all n. 00:45:01.410 --> 00:45:06.570 align:middle line:90% 00:45:06.570 --> 00:45:16.100 align:middle line:84% OK So this is a case where the initial state matters. 00:45:16.100 --> 00:45:21.640 align:middle line:84% R11 goes to a limit, as n goes to infinity, 00:45:21.640 --> 00:45:22.720 align:middle line:90% because it's constant. 00:45:22.720 --> 00:45:25.460 align:middle line:84% It's always 1 so the limit is 1. 00:45:25.460 --> 00:45:28.050 align:middle line:90% R31 also has a limit. 00:45:28.050 --> 00:45:30.010 align:middle line:90% It's 0 for all times. 00:45:30.010 --> 00:45:32.830 align:middle line:84% So these are the long term probabilities of finding 00:45:32.830 --> 00:45:34.730 align:middle line:90% yourself at state 1. 00:45:34.730 --> 00:45:37.990 align:middle line:84% But those long-term probabilities are affected by 00:45:37.990 --> 00:45:39.470 align:middle line:90% where you started. 00:45:39.470 --> 00:45:41.830 align:middle line:84% If you start here, you're sure that's, in the long term, 00:45:41.830 --> 00:45:42.860 align:middle line:90% you'll be here. 00:45:42.860 --> 00:45:45.260 align:middle line:84% If you start here, you're sure that, in the long term, you 00:45:45.260 --> 00:45:47.120 align:middle line:90% will not be there. 00:45:47.120 --> 00:45:50.550 align:middle line:84% So the initial state does matter here. 00:45:50.550 --> 00:45:53.780 align:middle line:84% And this is a situation where certain states are not 00:45:53.780 --> 00:45:57.120 align:middle line:84% accessible from certain other states, so it has something to 00:45:57.120 --> 00:46:00.150 align:middle line:84% do with the graph structure of our Markov chain. 00:46:00.150 --> 00:46:04.640 align:middle line:84% Finally let's answer this question here, at 00:46:04.640 --> 00:46:07.560 align:middle line:90% least for large n's. 00:46:07.560 --> 00:46:12.140 align:middle line:84% What do you think is going to happen in the long term if you 00:46:12.140 --> 00:46:14.555 align:middle line:90% start at state 2? 00:46:14.555 --> 00:46:18.840 align:middle line:84% If you start at state 2, you may stay at state 2 for a 00:46:18.840 --> 00:46:22.860 align:middle line:84% random amount of time, but eventually this transition 00:46:22.860 --> 00:46:25.680 align:middle line:84% will happen, or that transition would happen. 00:46:25.680 --> 00:46:30.720 align:middle line:84% Because of the symmetry, you are as likely to escape from 00:46:30.720 --> 00:46:34.170 align:middle line:84% state 2 in this direction, or in that direction, so there's 00:46:34.170 --> 00:46:37.510 align:middle line:84% probability 1/2 that, when the transition happens, the 00:46:37.510 --> 00:46:40.020 align:middle line:84% transition happens in that direction. 00:46:40.020 --> 00:46:47.660 align:middle line:84% So for large N, you're certain that the 00:46:47.660 --> 00:46:50.640 align:middle line:90% transition does happen. 00:46:50.640 --> 00:46:54.380 align:middle line:84% And given that the transition has happened, it has 00:46:54.380 --> 00:46:57.560 align:middle line:84% probability 1/2 that it has gone that particular way. 00:46:57.560 --> 00:47:00.380 align:middle line:84% So clearly here, you see that the probability of finding 00:47:00.380 --> 00:47:03.770 align:middle line:84% yourself in a particular state is very much affected by where 00:47:03.770 --> 00:47:05.400 align:middle line:90% you started from. 00:47:05.400 --> 00:47:09.310 align:middle line:84% So what we want to do next is to abstract from these two 00:47:09.310 --> 00:47:13.550 align:middle line:84% examples and describe the general structural properties 00:47:13.550 --> 00:47:16.360 align:middle line:84% that have to do with periodicity, and that have to 00:47:16.360 --> 00:47:18.990 align:middle line:84% do with what happened here with certain states, not being 00:47:18.990 --> 00:47:20.720 align:middle line:90% accessible from the others. 00:47:20.720 --> 00:47:23.530 align:middle line:84% We're going to leave periodicity for next time. 00:47:23.530 --> 00:47:25.380 align:middle line:84% But let's talk about the second kind of 00:47:25.380 --> 00:47:28.540 align:middle line:90% phenomenon that we have. 00:47:28.540 --> 00:47:32.420 align:middle line:84% So here, what we're going to do is to classify the states 00:47:32.420 --> 00:47:35.230 align:middle line:84% in a transition diagram into two types, 00:47:35.230 --> 00:47:38.000 align:middle line:90% recurrent and transient. 00:47:38.000 --> 00:47:41.330 align:middle line:84% So a state is said to be recurrent if the 00:47:41.330 --> 00:47:43.560 align:middle line:90% following is true. 00:47:43.560 --> 00:47:49.470 align:middle line:84% If you start from the state i, you can go to some places, but 00:47:49.470 --> 00:47:55.390 align:middle line:84% no matter where you go, there is a way of coming back. 00:47:55.390 --> 00:47:59.390 align:middle line:84% So what's an example for the recurrent state? 00:47:59.390 --> 00:48:02.000 align:middle line:90% This one. 00:48:02.000 --> 00:48:04.510 align:middle line:84% Starting from here, you can go elsewhere. 00:48:04.510 --> 00:48:06.510 align:middle line:90% You can go to state 7. 00:48:06.510 --> 00:48:08.540 align:middle line:90% You can go to state 6. 00:48:08.540 --> 00:48:11.050 align:middle line:84% That's all where you can go to. 00:48:11.050 --> 00:48:15.730 align:middle line:84% But no matter where you go, there is a path that can take 00:48:15.730 --> 00:48:17.190 align:middle line:90% you back there. 00:48:17.190 --> 00:48:20.640 align:middle line:84% So no matter where you go, there is a chance, and there 00:48:20.640 --> 00:48:23.470 align:middle line:84% is a way for returning where you started. 00:48:23.470 --> 00:48:25.770 align:middle line:84% Those states we call recurrent. 00:48:25.770 --> 00:48:28.750 align:middle line:90% And by this, 8 is recurrent. 00:48:28.750 --> 00:48:31.960 align:middle line:90% All of these are recurrent. 00:48:31.960 --> 00:48:34.080 align:middle line:84% So this is recurrent, this is recurrent. 00:48:34.080 --> 00:48:36.570 align:middle line:84% And this state 5 is also recurrent. 00:48:36.570 --> 00:48:40.080 align:middle line:84% You cannot go anywhere from 5 except to 5 itself. 00:48:40.080 --> 00:48:43.860 align:middle line:84% Wherever you can go, you can go back to where you start. 00:48:43.860 --> 00:48:45.830 align:middle line:90% So this is recurrent. 00:48:45.830 --> 00:48:49.190 align:middle line:84% If it is not the recurrent, we say that it is transient. 00:48:49.190 --> 00:48:50.890 align:middle line:90% So what does transient mean? 00:48:50.890 --> 00:48:53.900 align:middle line:84% You need to take this definition, and reverse it. 00:48:53.900 --> 00:48:58.860 align:middle line:84% Transient means that, starting from i, there is a place to 00:48:58.860 --> 00:49:05.010 align:middle line:84% which you could go, and from which you cannot return. 00:49:05.010 --> 00:49:07.160 align:middle line:84% If it's recurrent, anywhere you go, you 00:49:07.160 --> 00:49:09.170 align:middle line:90% can always come back. 00:49:09.170 --> 00:49:12.100 align:middle line:84% Transient means there are places where you can go from 00:49:12.100 --> 00:49:14.270 align:middle line:90% which you cannot come back. 00:49:14.270 --> 00:49:18.320 align:middle line:84% So state 1 is recurrent - because starting from here, 00:49:18.320 --> 00:49:20.880 align:middle line:84% there's a possibility that you get there, and then 00:49:20.880 --> 00:49:22.310 align:middle line:90% there's no way back. 00:49:22.310 --> 00:49:26.120 align:middle line:84% State 4 is recurrent, starting from 4, there's somewhere you 00:49:26.120 --> 00:49:28.520 align:middle line:90% can go and-- 00:49:28.520 --> 00:49:30.260 align:middle line:90% sorry, transient, correct. 00:49:30.260 --> 00:49:33.180 align:middle line:84% State 4 is transient starting from here, there are places 00:49:33.180 --> 00:49:36.670 align:middle line:84% where you could go, and from which you cannot come back. 00:49:36.670 --> 00:49:40.380 align:middle line:84% And in this particular diagram, all these 4 states 00:49:40.380 --> 00:49:43.110 align:middle line:90% are transients. 00:49:43.110 --> 00:49:49.800 align:middle line:84% Now if the state is transient, it means that there is a way 00:49:49.800 --> 00:49:53.150 align:middle line:84% to go somewhere where you're going to get stuck and not to 00:49:53.150 --> 00:49:54.840 align:middle line:90% be able to come. 00:49:54.840 --> 00:49:59.350 align:middle line:84% As long as your state keeps circulating around here, 00:49:59.350 --> 00:50:02.460 align:middle line:84% eventually one of these transitions is going to 00:50:02.460 --> 00:50:05.820 align:middle line:84% happen, and once that happens, then there's no way that you 00:50:05.820 --> 00:50:06.960 align:middle line:90% can come back. 00:50:06.960 --> 00:50:11.360 align:middle line:84% So that transient state will be visited only a finite 00:50:11.360 --> 00:50:12.650 align:middle line:90% number of times. 00:50:12.650 --> 00:50:15.100 align:middle line:84% You will not be able to return to it. 00:50:15.100 --> 00:50:17.760 align:middle line:84% And in the long run, you're certain that you're going to 00:50:17.760 --> 00:50:22.150 align:middle line:84% get out of the transient states, and get to some class 00:50:22.150 --> 00:50:25.100 align:middle line:84% of recurrent states, and get stuck forever. 00:50:25.100 --> 00:50:29.050 align:middle line:84% So, let's see, in this diagram, if I start here, 00:50:29.050 --> 00:50:32.580 align:middle line:84% could I stay in this lump of states forever? 00:50:32.580 --> 00:50:35.780 align:middle line:84% Well as long as I'm staying in this type of states, I would 00:50:35.780 --> 00:50:40.000 align:middle line:84% keep visiting states 1 and 2 Each time that I visit state 00:50:40.000 --> 00:50:41.480 align:middle line:90% 2, there's going to be positive 00:50:41.480 --> 00:50:43.550 align:middle line:90% probability that I escape. 00:50:43.550 --> 00:50:47.935 align:middle line:84% So in the long run, if I were to stay here, I would visit 00:50:47.935 --> 00:50:50.130 align:middle line:84% state 2 an infinite number of times, and I would get 00:50:50.130 --> 00:50:52.210 align:middle line:90% infinite chances to escape. 00:50:52.210 --> 00:50:56.840 align:middle line:84% But if you have infinite chances to escape, eventually 00:50:56.840 --> 00:50:57.810 align:middle line:90% you will escape. 00:50:57.810 --> 00:51:01.760 align:middle line:84% So you are certain that with probability 1, starting from 00:51:01.760 --> 00:51:05.050 align:middle line:84% here, you're going to move either to those states, or to 00:51:05.050 --> 00:51:06.150 align:middle line:90% those states. 00:51:06.150 --> 00:51:09.710 align:middle line:84% So starting from transient states, you only stay at the 00:51:09.710 --> 00:51:14.660 align:middle line:84% transient states for random but finite amount of time. 00:51:14.660 --> 00:51:19.000 align:middle line:84% And after that happens, you end up in a class 00:51:19.000 --> 00:51:20.180 align:middle line:90% of recurrent states. 00:51:20.180 --> 00:51:23.090 align:middle line:84% And when I say class, what they mean is that, in this 00:51:23.090 --> 00:51:26.440 align:middle line:84% picture, I divide the recurrent states into 2 00:51:26.440 --> 00:51:28.280 align:middle line:90% classes, or categories. 00:51:28.280 --> 00:51:30.220 align:middle line:90% What's special about them? 00:51:30.220 --> 00:51:31.300 align:middle line:90% These states are recurrent. 00:51:31.300 --> 00:51:33.000 align:middle line:90% These states are recurrent. 00:51:33.000 --> 00:51:35.150 align:middle line:84% But there's no communication between the 2. 00:51:35.150 --> 00:51:36.780 align:middle line:84% If you start here, you're stuck here. 00:51:36.780 --> 00:51:39.750 align:middle line:84% If you start here, you are stuck there. 00:51:39.750 --> 00:51:42.970 align:middle line:84% And this is a case where the initial state does matter, 00:51:42.970 --> 00:51:45.180 align:middle line:84% because if you start here, you get stuck here. 00:51:45.180 --> 00:51:47.130 align:middle line:84% You start here, you get stuck there. 00:51:47.130 --> 00:51:49.970 align:middle line:84% So depending on the initial state, that's going to affect 00:51:49.970 --> 00:51:52.590 align:middle line:84% the long term behavior of your chain. 00:51:52.590 --> 00:51:55.470 align:middle line:84% So the guess you can make at this point is that, for the 00:51:55.470 --> 00:51:59.210 align:middle line:84% initial state to not matter, we should not have multiple 00:51:59.210 --> 00:52:00.160 align:middle line:90% recurrent classes. 00:52:00.160 --> 00:52:01.710 align:middle line:90% We should have only 1. 00:52:01.710 --> 00:52:04.030 align:middle line:84% But we're going to get back to this point next time. 00:52:04.030 --> 00:52:05.280 align:middle line:90%