WEBVTT
00:00:00.820 --> 00:00:03.330
Let us now abstract from
our previous example
00:00:03.330 --> 00:00:06.630
and provide a general definition
of what a discrete time,
00:00:06.630 --> 00:00:09.480
finite state Markov chain is.
00:00:09.480 --> 00:00:13.230
First, central in the
description of a Markov process
00:00:13.230 --> 00:00:17.020
is the concept of a state, which
describes the current situation
00:00:17.020 --> 00:00:20.640
of a system we
are interested in.
00:00:20.640 --> 00:00:24.280
For example, in the case of
the checkout counter example,
00:00:24.280 --> 00:00:26.580
the number of
customers in the queue
00:00:26.580 --> 00:00:28.970
provided the right
level of information
00:00:28.970 --> 00:00:33.260
needed to define a useful state.
00:00:33.260 --> 00:00:37.070
Time is assumed to
be discrete, that is,
00:00:37.070 --> 00:00:40.390
divided in discrete time steps.
00:00:40.390 --> 00:00:46.340
The system starts at time
0 in an initial state,
00:00:46.340 --> 00:00:48.650
and at each
successive time step,
00:00:48.650 --> 00:00:50.640
the system goes from
its current state
00:00:50.640 --> 00:00:53.510
to a next one chosen
with some randomness.
00:00:53.510 --> 00:00:57.740
As a result, after
n such transitions,
00:00:57.740 --> 00:01:00.590
the state of the
system will be random,
00:01:00.590 --> 00:01:04.260
and so we can think of
it as a random variable.
00:01:04.260 --> 00:01:07.210
Let Xn be this random variable.
00:01:07.210 --> 00:01:11.485
That is, Xn represents the
state in which the system is
00:01:11.485 --> 00:01:14.539
after n transitions from an
initial state in which it
00:01:14.539 --> 00:01:17.210
started to operate.
00:01:17.210 --> 00:01:22.120
As a shortcut, we
may often say that Xn
00:01:22.120 --> 00:01:25.620
is the state of the
system at time n.
00:01:25.620 --> 00:01:28.500
We suppose that there
is a finite number
00:01:28.500 --> 00:01:32.200
of possible states for
the system to be in.
00:01:32.200 --> 00:01:36.610
Here, we have drawn a portion
of a finite state space
00:01:36.610 --> 00:01:42.910
with m possible states
labeled 1 to m, using i
00:01:42.910 --> 00:01:47.670
and j as generic labels.
00:01:47.670 --> 00:01:50.210
Of course, we could think of
systems with an infinite number
00:01:50.210 --> 00:01:52.690
of states, either
discrete or continuous,
00:01:52.690 --> 00:01:54.759
but this is a bit
more complicated,
00:01:54.759 --> 00:01:57.150
and so in this course,
we restrict ourselves
00:01:57.150 --> 00:01:59.360
to a finite state space.
00:01:59.360 --> 00:02:06.210
Note that the initial
state could itself
00:02:06.210 --> 00:02:10.228
be fixed or chosen randomly.
00:02:10.228 --> 00:02:15.580
Assume now that the system
started in state three.
00:02:15.580 --> 00:02:17.360
What will happen next?
00:02:17.360 --> 00:02:20.090
The system will evolve
according to one
00:02:20.090 --> 00:02:23.500
of the possible transitions
out of state three,
00:02:23.500 --> 00:02:27.230
for example, one of these arcs.
00:02:27.230 --> 00:02:31.950
Note here that we don't have
an arc from three to four.
00:02:31.950 --> 00:02:34.920
As a convention, we
only include arcs
00:02:34.920 --> 00:02:38.380
for transitions that can happen.
00:02:38.380 --> 00:02:41.530
Remember the checkout
counter example.
00:02:41.530 --> 00:02:43.900
Because of our assumptions
that no more than one
00:02:43.900 --> 00:02:46.350
person can join the
queue at any time,
00:02:46.350 --> 00:02:48.230
we didn't have arcs
of the type going
00:02:48.230 --> 00:02:55.260
from one to three
or from two to ten.
00:02:55.260 --> 00:02:58.500
Also, because of the customers
being served one at a time,
00:02:58.500 --> 00:03:01.430
departures were limited
to one person at a time,
00:03:01.430 --> 00:03:04.170
and so no arcs of the
type going from two
00:03:04.170 --> 00:03:08.352
to zero or from nine to two.
00:03:08.352 --> 00:03:11.790
So the next transition
out of three
00:03:11.790 --> 00:03:16.090
can be thought of a random
jump where, from state three,
00:03:16.090 --> 00:03:23.180
the system will jump to either
state one, state two, state j,
00:03:23.180 --> 00:03:26.530
or jump unto itself.
00:03:26.530 --> 00:03:29.950
These will be the
only possibilities.
00:03:29.950 --> 00:03:32.710
We want to describe the
statistics of these jumps,
00:03:32.710 --> 00:03:35.560
and we will use
conditional probabilities.
00:03:35.560 --> 00:03:37.220
Given that at time
zero, the system
00:03:37.220 --> 00:03:39.190
is in state three,
what is the probability
00:03:39.190 --> 00:03:42.480
that it will be in state j next?
00:03:42.480 --> 00:03:46.400
These will be called
transition probabilities.
00:03:46.400 --> 00:03:50.620
For example, the probability
of going from three to one
00:03:50.620 --> 00:03:53.340
will be p31.
00:03:53.340 --> 00:04:01.930
Here, p32, here,
p33, and here, p3j.
00:04:06.770 --> 00:04:08.930
Note that these are
the only possibilities.
00:04:08.930 --> 00:04:26.310
As a result, you have p31
plus p32 plus p33 plus p3j
00:04:26.310 --> 00:04:27.150
will be 1.
00:04:30.340 --> 00:04:32.780
Assume that the system
continued to evolve,
00:04:32.780 --> 00:04:38.850
and after various
different steps, come back
00:04:38.850 --> 00:04:41.890
in three at time n.
00:04:44.990 --> 00:04:47.750
Again, what will happen next?
00:04:47.750 --> 00:04:50.720
Well, this property here
says that the probability
00:04:50.720 --> 00:04:57.190
of going from state three
to one is again p31,
00:04:57.190 --> 00:04:59.200
the same as before.
00:04:59.200 --> 00:05:04.650
In other words,
here, we will say
00:05:04.650 --> 00:05:08.980
that the chain is
time homogeneous.
00:05:12.090 --> 00:05:15.660
That is, these
transition probabilities
00:05:15.660 --> 00:05:18.490
will be the same
irrespective of the time.
00:05:18.490 --> 00:05:22.380
So this is true for all n.
00:05:22.380 --> 00:05:24.260
And the summation that
we have written here
00:05:24.260 --> 00:05:27.370
for the special case
is of course general.
00:05:27.370 --> 00:05:33.920
What we have is that the
probability of i to j,
00:05:33.920 --> 00:05:36.510
If you sum all of
these probabilities
00:05:36.510 --> 00:05:43.690
for all possible j's,
you will have one.
00:05:43.690 --> 00:05:46.950
Now, in order for this
probabilistic specification
00:05:46.950 --> 00:05:49.920
to make sense and
be coherent, we
00:05:49.920 --> 00:05:52.159
need to make a big assumption
about the evolution
00:05:52.159 --> 00:05:54.040
of the state of the system.
00:05:54.040 --> 00:05:56.700
This assumption, the
so-called Markov property,
00:05:56.700 --> 00:06:03.950
given in words here and in
mathematical statement here,
00:06:03.950 --> 00:06:08.730
is in fact, the defining nature
of what a Markov process is.
00:06:08.730 --> 00:06:12.940
In words, what it says is
that every time the system
00:06:12.940 --> 00:06:16.980
finds itself in state three,
the transition probability
00:06:16.980 --> 00:06:20.660
of going to state
one will always
00:06:20.660 --> 00:06:24.350
be p31, no matter how
the system evolved
00:06:24.350 --> 00:06:27.380
in the past up to
being in state three.
00:06:27.380 --> 00:06:29.980
In other words, no matter
what path the system
00:06:29.980 --> 00:06:32.040
has followed up to
the current state,
00:06:32.040 --> 00:06:34.110
the next state
transition probability
00:06:34.110 --> 00:06:37.290
will be the same,
independent of that past.
00:06:37.290 --> 00:06:43.830
Mathematically, conditionally
on knowing your current state,
00:06:43.830 --> 00:06:47.740
having more information
about the past state
00:06:47.740 --> 00:06:51.000
variables does not change
the transition probability
00:06:51.000 --> 00:06:52.500
to your next state.
00:06:52.500 --> 00:06:56.580
In other words, the probability
distribution of the next state,
00:06:56.580 --> 00:07:00.460
X n+1, depends on the past
only through the value
00:07:00.460 --> 00:07:02.670
of the present state, Xn.
00:07:02.670 --> 00:07:06.530
So you can see that as the
definition of the transition
00:07:06.530 --> 00:07:09.860
probability and that
property, that equality
00:07:09.860 --> 00:07:13.550
from here to here, being
the Markov property.
00:07:13.550 --> 00:07:17.040
For this property to hold
in any modeling application,
00:07:17.040 --> 00:07:19.380
you need to choose
your state carefully.
00:07:19.380 --> 00:07:21.430
You want to ensure that
the information contained
00:07:21.430 --> 00:07:23.470
in the description of
your state captures
00:07:23.470 --> 00:07:26.110
all the relevant information
to make predictions
00:07:26.110 --> 00:07:28.070
about the future evolution.
00:07:28.070 --> 00:07:31.240
Now, given a system,
how to properly define
00:07:31.240 --> 00:07:33.070
the state variables
which will allow
00:07:33.070 --> 00:07:36.210
us to model its evolution
as a Markov process
00:07:36.210 --> 00:07:38.770
is somewhat of an
art, and there are
00:07:38.770 --> 00:07:41.490
no cookbook recipes to do it.
00:07:41.490 --> 00:07:45.710
However, with a little bit
of experience and practice,
00:07:45.710 --> 00:07:50.250
one quickly gets the required
intuition to do this properly.
00:07:50.250 --> 00:07:53.480
You will be able to
do so in that class.