WEBVTT
00:00:01.030 --> 00:00:03.580
Markov processes
can be very general.
00:00:03.580 --> 00:00:06.330
They can run in continuous
or discrete time,
00:00:06.330 --> 00:00:10.250
can have a discrete or a
continuous state space.
00:00:10.250 --> 00:00:12.340
In this class, we'll
restrict ourselves
00:00:12.340 --> 00:00:15.670
to discrete time discrete
state Markov chains.
00:00:15.670 --> 00:00:17.480
These are the
simplest cases and are
00:00:17.480 --> 00:00:19.380
the best to build our intuition.
00:00:19.380 --> 00:00:26.360
So the state space is discrete,
here, finite with m states,
00:00:26.360 --> 00:00:29.160
and time is discrete.
00:00:29.160 --> 00:00:31.890
That is, at any
discrete point in time,
00:00:31.890 --> 00:00:34.280
the process is in
one of these m states
00:00:34.280 --> 00:00:39.770
and let's say here,
at any given time s.
00:00:39.770 --> 00:00:42.520
And again, time is
discrete, so think
00:00:42.520 --> 00:00:44.680
about the following process.
00:00:44.680 --> 00:00:48.660
You have someone hitting
a drum, indicating
00:00:48.660 --> 00:00:50.730
that a transition occurs.
00:00:50.730 --> 00:00:53.510
And what it means is that
the chain that was here
00:00:53.510 --> 00:00:55.380
will then jump.
00:00:55.380 --> 00:01:00.220
Let's say to another
state j at the next time.
00:01:00.220 --> 00:01:05.190
So when the Markov chain
jumps it can jump on itself
00:01:05.190 --> 00:01:08.690
or jump to another
state like here or here.
00:01:11.780 --> 00:01:15.660
And then at time s plus
1 someone hit the drum
00:01:15.660 --> 00:01:19.170
and you jump again,
and so on and so forth.
00:01:19.170 --> 00:01:22.620
You can think of a very active
frog jumping from lilies
00:01:22.620 --> 00:01:26.445
to lilies on the pond and
following a regular drumbeat.
00:01:29.450 --> 00:01:32.570
So what is left to define
are the various probabilities
00:01:32.570 --> 00:01:36.300
of transitions, such as
the transition from i to j,
00:01:36.300 --> 00:01:39.039
and the notation
we're going to use
00:01:39.039 --> 00:01:43.890
is pij, which by definition
is that transition probability
00:01:43.890 --> 00:01:44.390
here.
00:01:50.509 --> 00:01:54.940
So given that you are in
state i at time s, what
00:01:54.940 --> 00:01:59.060
is the probability that you end
up in state j at time s plus 1.
00:01:59.060 --> 00:02:03.120
Notice that these transition
priorities here, pij,
00:02:03.120 --> 00:02:04.880
are not function of s.
00:02:04.880 --> 00:02:09.280
So irrespective of what the
time s that we're talking about,
00:02:09.280 --> 00:02:11.770
these transitions
priorities are the same.
00:02:11.770 --> 00:02:16.710
So this is what we mean by a
time-homogeneous Markov chain.
00:02:16.710 --> 00:02:23.490
In other words, these are
valid for s equal 0, 1, 2,
00:02:23.490 --> 00:02:25.630
and so on and so forth.
00:02:25.630 --> 00:02:28.415
So the defining feature
of a Markov chain
00:02:28.415 --> 00:02:30.730
is the Markov property.
00:02:30.730 --> 00:02:32.200
And the Markov
property essentially
00:02:32.200 --> 00:02:35.990
says that the past is not
really important in order
00:02:35.990 --> 00:02:39.590
to predict the future, as long
as you know where you are now.
00:02:39.590 --> 00:02:42.540
Another way of saying
it is that if you
00:02:42.540 --> 00:02:47.329
look at the probability
of going next in state j
00:02:47.329 --> 00:02:51.360
given that you
are now in state i
00:02:51.360 --> 00:02:54.130
and that I give
you, in addition,
00:02:54.130 --> 00:02:55.480
the entire trajectories.
00:02:55.480 --> 00:03:00.340
So I tell you that it was in i0
at that time, and so on and so
00:03:00.340 --> 00:03:04.460
forth, all the way
up to time s minus 1,
00:03:04.460 --> 00:03:06.850
where it was in is minus 1.
00:03:06.850 --> 00:03:09.605
So it gives you the
entire trajectory
00:03:09.605 --> 00:03:12.940
of the chain up to s,
and now I'm asking you,
00:03:12.940 --> 00:03:16.170
what is the probability that
you're going to go to s plus 1?
00:03:16.170 --> 00:03:21.079
The Markov property here simply
says that that probability here
00:03:21.079 --> 00:03:24.260
is again, pij.
00:03:24.260 --> 00:03:25.990
So in other words,
all this information
00:03:25.990 --> 00:03:30.460
here is of no use to
compute this probability.
00:03:30.460 --> 00:03:33.610
Now, note that these transition
probabilities are really
00:03:33.610 --> 00:03:35.240
probabilities, in
the following sense.
00:03:35.240 --> 00:03:35.740
Right?
00:03:35.740 --> 00:03:40.270
So you are in i and then
at the next time step,
00:03:40.270 --> 00:03:42.560
you will definitely
jump with probability 1.
00:03:42.560 --> 00:03:44.570
And where you're going
to jump will depend,
00:03:44.570 --> 00:03:47.829
but the summation
of all possibilities
00:03:47.829 --> 00:03:49.950
have to sum up to 1.
00:03:49.950 --> 00:03:54.520
So from j equals 1
to n has to be 1.
00:03:54.520 --> 00:03:57.740
So now that we have introduced
the main ingredients,
00:03:57.740 --> 00:04:01.140
usually we are very interested
in knowing what a Markov
00:04:01.140 --> 00:04:03.440
chain is going to
do in the long run.
00:04:03.440 --> 00:04:05.390
We are interested in
finding the probability
00:04:05.390 --> 00:04:09.680
that the chain is in a state
j after n transitions, given
00:04:09.680 --> 00:04:11.690
that it is now in state i.
00:04:11.690 --> 00:04:13.830
Now because of the
time-homogeneous,
00:04:13.830 --> 00:04:15.200
this is the same thing as that.
00:04:15.200 --> 00:04:18.490
In other words, the current
time could be in any time s,
00:04:18.490 --> 00:04:20.480
we just have to add s here.
00:04:20.480 --> 00:04:24.490
And again, that is nothing
else than this property.
00:04:24.490 --> 00:04:30.544
So we are interested in
calculating rij of n for any n.
00:04:30.544 --> 00:04:35.560
For n equals 1, this is
nothing else than rij of 1
00:04:35.560 --> 00:04:38.800
is the same as this
transition probabilities
00:04:38.800 --> 00:04:40.280
that we have defined.
00:04:40.280 --> 00:04:43.490
But for n greater than or
equals to 2, what we are seeing
00:04:43.490 --> 00:04:46.520
is the introduction of
a key recursion here.
00:04:46.520 --> 00:04:49.380
And this is how you would
be able to calculate
00:04:49.380 --> 00:04:50.980
these probabilities.
00:04:50.980 --> 00:04:53.805
Now, how did we come
up with this recursion?
00:04:53.805 --> 00:04:56.860
Well, it's based on a
classical divide and conquer
00:04:56.860 --> 00:05:00.210
and essentially, the use of
the total property theorem.
00:05:00.210 --> 00:05:02.720
Essentially, you have
the time step here.
00:05:02.720 --> 00:05:04.544
This is the current time s.
00:05:04.544 --> 00:05:05.960
You are interested
in what's going
00:05:05.960 --> 00:05:09.350
to happen at n plus
s and n steps later.
00:05:09.350 --> 00:05:11.354
Here you are in state i.
00:05:11.354 --> 00:05:12.770
You are interested
in knowing what
00:05:12.770 --> 00:05:15.650
is the probability of being
in state j at that time.
00:05:15.650 --> 00:05:18.970
And what you simply do
is you look at the step n
00:05:18.970 --> 00:05:21.840
plus s minus 1, just
before the last one.
00:05:21.840 --> 00:05:24.960
And then you say, well, let
me do a divide and conquer.
00:05:24.960 --> 00:05:27.450
This is k here, and
I'm going to look
00:05:27.450 --> 00:05:30.360
at evaluating that probability.
00:05:30.360 --> 00:05:32.860
And then once I have
that, I will simply
00:05:32.860 --> 00:05:35.290
multiply it by this
transition here.
00:05:35.290 --> 00:05:37.610
And what happened
is that this here
00:05:37.610 --> 00:05:40.730
is nothing else than this
calculation that we have here.
00:05:40.730 --> 00:05:42.310
And that's the same thing.
00:05:42.310 --> 00:05:44.850
And here, this is
the probability
00:05:44.850 --> 00:05:48.450
of one step transition.
00:05:48.450 --> 00:05:51.090
And, of course, we have
conditioned on the fact
00:05:51.090 --> 00:05:52.670
that we would be
in a state k here,
00:05:52.670 --> 00:05:55.620
but k could be any of
these states, right?
00:05:55.620 --> 00:05:57.960
And they are m of
them, and this is
00:05:57.960 --> 00:06:01.560
why we saw from k equals 1 to m.
00:06:01.560 --> 00:06:04.610
So essentially this is
how this key recursion has
00:06:04.610 --> 00:06:06.440
been put together,
and we have used,
00:06:06.440 --> 00:06:09.790
of course, the Markov
property in order to do that.
00:06:09.790 --> 00:06:12.650
Let's do now a little bit of
warm up in terms of calculation
00:06:12.650 --> 00:06:14.940
and apply these concepts.