WEBVTT
00:00:00.500 --> 00:00:04.230
In this lecture, we introduce
Markov chains, a general class
00:00:04.230 --> 00:00:06.860
of random processes
with many applications
00:00:06.860 --> 00:00:10.380
dealing with the evolution
of dynamical systems.
00:00:10.380 --> 00:00:12.230
As opposed to the
Bernoulli and Poisson
00:00:12.230 --> 00:00:15.420
processes, which are
memoryless in a sense
00:00:15.420 --> 00:00:17.860
that the future does
not depend on the past,
00:00:17.860 --> 00:00:20.350
Markov chains are more
elaborate, as they
00:00:20.350 --> 00:00:23.980
allow some dependencies
between different times.
00:00:23.980 --> 00:00:27.640
However, these dependencies
are of simple and restricted
00:00:27.640 --> 00:00:31.330
nature, captured by the
so-called Markov property.
00:00:31.330 --> 00:00:34.710
Conditional on the current state
of the Markov chain, its future
00:00:34.710 --> 00:00:37.420
and past evolutions
are independent.
00:00:37.420 --> 00:00:39.830
As mentioned in
the unit overview,
00:00:39.830 --> 00:00:41.760
we will only consider
discrete time
00:00:41.760 --> 00:00:46.170
Markov chains that evolve
within finite state spaces.
00:00:46.170 --> 00:00:49.270
This allows us to concentrate
on the main concepts
00:00:49.270 --> 00:00:52.760
without having to deal with
some required technical details
00:00:52.760 --> 00:00:57.470
needed to study general Markov
processes under continuous time
00:00:57.470 --> 00:01:02.850
and general, possibly
uncountable, state spaces.
00:01:02.850 --> 00:01:05.550
We will first introduced
the basic concepts,
00:01:05.550 --> 00:01:08.270
using the simple example
of a checkout counter
00:01:08.270 --> 00:01:12.610
at a supermarket, an example
of a simple queuing system.
00:01:12.610 --> 00:01:14.570
We will then abstract
from the example
00:01:14.570 --> 00:01:17.200
and give some general
definitions, including
00:01:17.200 --> 00:01:21.570
the central notions of states,
transition probabilities,
00:01:21.570 --> 00:01:26.090
Markov property, and
transition probability graphs.
00:01:26.090 --> 00:01:28.920
Afterwards, we will look
at various questions,
00:01:28.920 --> 00:01:32.560
such as predicting what
will happen in n-steps
00:01:32.560 --> 00:01:36.310
in the future, given the
current state of our system.
00:01:36.310 --> 00:01:40.180
We will define n-step
transition probabilities exactly
00:01:40.180 --> 00:01:43.180
and show how to calculate
them efficiency.
00:01:43.180 --> 00:01:45.530
We will also discuss
what could happen
00:01:45.530 --> 00:01:49.570
when we let the Markov chain
run for a very long time.
00:01:49.570 --> 00:01:51.710
We will end this
lecture by introducing
00:01:51.710 --> 00:01:55.039
the notions of recurrent
and transient states
00:01:55.039 --> 00:01:59.570
and their importance in studying
Markov chains in the long run.