Finite Horizon Problems

Lecture 1 (PDF)

 Introduction to Dynamic Programming
 Examples of Dynamic Programming
 Significance of Feedback

Lecture 2 (PDF)

 The Basic Problem
 Principle of Optimality
 The General Dynamic Programming Algorithm
 State Augmentation

Lecture 3 (PDF)

 Deterministic FiniteState Problem
 Backward Shortest Path Algorithm
 Forward Shortest Path Algorithm
 Alternative Shortest Path Algorithms

Lecture 4 (PDF)

 Examples of Stochastic Dynamic Programming Problems
 LinearQuadratic Problems
 Inventory Control

Lecture 5 (PDF)

 Stopping Problems
 Scheduling Problems
 Minimax Control

Lecture 6 (PDF)

 Problems with Imperfect State Info
 Reduction to the Perfect State Info Cas
 Linear Quadratic Problems
 Separation of Estimation and Control

Lecture 7 (PDF)

 Imperfect State Information
 Sufficient Statistics
 Conditional State Distribution as a Sufficient Statistic
 FiniteState Analysis

Lecture 8 (PDF)

 Suboptimal Control
 Cost Approximation Methods: Classification
 Certainty Equivalent Control
 Limited Lookahead Policies
 Performance Bounds
 Problem Approximation Approach
 Parametric CostToGo Approximation

Lecture 9 (PDF)

 Rollout Algorithms
 Cost Improvement Property
 Discrete Deterministic Problems
 Approximations to Rollout Algorithms
 Model Predictive Control (MPS)
 Discretization of Continuous Time
 Discretization of Continuous Space
 Other Suboptimal Approaches

Simple Infinite Horizon Problems

Lecture 10 (PDF)

 Infinite Horizon Problems
 Stochastic Shortest Path (SSP) Problems
 Bellman’s Equation
 Dynamic Programming – Value Iteration
 Discounted Problems as a Special Case of SSP

Lecture 11 (PDF)

 Review of Stochastic Shortest Path Problems
 Computation Methods for SSP
 Computational Methods for Discounted Problems

Lecture 12 (PDF)

 Average Cost Per Stage Problems
 Connection With Stochastic Shortest Path Problems
 Bellman’s Equation
 Value Iteration, Policy Iteration

Lecture 13 (PDF)

 Control of ContinuousTime Markov Chains: SemiMarkov Problems
 Problem Formulation: Equivalence to DiscreteTime Problems
 Discounted Problems
 Average Cost Problems

Advanced Infinite Horizon Problems

Lecture 14 (PDF)

 Introduction to Advanced Infinite Horizon Dynamic Programming and Approximation Methods

Lecture 15 (PDF)

 Review of Basic Theory of Discounted Problems
 Monotonicity of Contraction Properties
 Contraction Mappings in Dynamic Programming
 Discounted Problems: Countable State Space with Unbounded Costs
 Generalized Discounted Dynamic Programming
 An Introduction to Abstract Dynamic Programming

Lecture 16 (PDF)

 Review of Computational Theory of Discounted Problems
 Value Iteration (VI)
 Policy Iteration (PI)
 Optimistic PI
 Computational Methods for Generalized Discounted Dynamic Programming
 Asynchronous Algorithms

Lecture 17 (PDF)

 Undiscounted Problems
 Stochastic Shortest Path Problems
 Proper and Improper Policies
 Analysis and Computational Methods for SSP
 Pathologies of SSP
 SSP Under Weak Conditions

Lecture 18 (PDF)

 Undiscounted Total Cost Problems
 Positive and Negative Cost Problems
 Deterministic Optimal Cost Problems
 Adaptive (Linear Quadratic) Dynamic Programming
 Affine Monotomic and Risk Sensitive Problems

Lecture 19 (PDF)

 Introduction to approximate Dynamic Programming
 Approximation in Policy Space
 Approximation in Value Space, Rollout / Simulationbased Single Policy Iteration
 Approximation in Value Space Using Problem Approximation

Lecture 20 (PDF)

 Discounted Problems
 Approximate (fitted) VI
 Approximate PI
 The Projected Equation
 Contraction Properties: Error Bounds
 Matrix Form of the Projected Equation
 Simulationbased Implementation
 LSTD, LSPE, and TD Methods

Lecture 21 (PDF)

 Review of Approximate Policy Iteration
 Projected Equation Methods for Policy Evaluation
 SimulationBased Implementation Issues, Multistep Projected Equation Methods
 BiasVariance Tradeoff
 ExplorationEnhanced Implementations, Oscillations

Lecture 22 (PDF)

 Aggregation as an Approximation Methodology
 Aggregate Problem
 Simulationbased Aggregation
 QLearning

Lecture 23 (PDF)

 Additional Topics in Advanced Dynamic Programming
 Stochastic Shortest Path Problems
 Average Cost Problems
 Generalizations
 Basis Function Adaptation
 Gradientbased Approximation in Policy Space
 An Overview
