|
LEC #
|
TOPICS
|
|
Finite Horizon Problems
|
|
Lecture 1 (PDF)
|
- Introduction to Dynamic Programming
- Examples of Dynamic Programming
- Significance of Feedback
|
|
Lecture 2 (PDF)
|
- The Basic Problem
- Principle of Optimality
- The General Dynamic Programming Algorithm
- State Augmentation
|
|
Lecture 3 (PDF)
|
- Deterministic Finite-State Problem
- Backward Shortest Path Algorithm
- Forward Shortest Path Algorithm
- Alternative Shortest Path Algorithms
|
|
Lecture 4 (PDF)
|
- Examples of Stochastic Dynamic Programming Problems
- Linear-Quadratic Problems
- Inventory Control
|
|
Lecture 5 (PDF)
|
- Stopping Problems
- Scheduling Problems
- Minimax Control
|
|
Lecture 6 (PDF)
|
- Problems with Imperfect State Info
- Reduction to the Perfect State Info Cas
- Linear Quadratic Problems
- Separation of Estimation and Control
|
|
Lecture 7 (PDF)
|
- Imperfect State Information
- Sufficient Statistics
- Conditional State Distribution as a Sufficient Statistic
- Finite-State Analysis
|
|
Lecture 8 (PDF)
|
- Suboptimal Control
- Cost Approximation Methods: Classification
- Certainty Equivalent Control
- Limited Lookahead Policies
- Performance Bounds
- Problem Approximation Approach
- Parametric Cost-To-Go Approximation
|
|
Lecture 9 (PDF)
|
- Rollout Algorithms
- Cost Improvement Property
- Discrete Deterministic Problems
- Approximations to Rollout Algorithms
- Model Predictive Control (MPS)
- Discretization of Continuous Time
- Discretization of Continuous Space
- Other Suboptimal Approaches
|
|
Simple Infinite Horizon Problems
|
|
Lecture 10 (PDF)
|
- Infinite Horizon Problems
- Stochastic Shortest Path (SSP) Problems
- Bellman’s Equation
- Dynamic Programming – Value Iteration
- Discounted Problems as a Special Case of SSP
|
|
Lecture 11 (PDF)
|
- Review of Stochastic Shortest Path Problems
- Computation Methods for SSP
- Computational Methods for Discounted Problems
|
|
Lecture 12 (PDF)
|
- Average Cost Per Stage Problems
- Connection With Stochastic Shortest Path Problems
- Bellman’s Equation
- Value Iteration, Policy Iteration
|
|
Lecture 13 (PDF)
|
- Control of Continuous-Time Markov Chains: Semi-Markov Problems
- Problem Formulation: Equivalence to Discrete-Time Problems
- Discounted Problems
- Average Cost Problems
|
|
Advanced Infinite Horizon Problems
|
|
Lecture 14 (PDF)
|
- Introduction to Advanced Infinite Horizon Dynamic Programming and Approximation Methods
|
|
Lecture 15 (PDF)
|
- Review of Basic Theory of Discounted Problems
- Monotonicity of Contraction Properties
- Contraction Mappings in Dynamic Programming
- Discounted Problems: Countable State Space with Unbounded Costs
- Generalized Discounted Dynamic Programming
- An Introduction to Abstract Dynamic Programming
|
|
Lecture 16 (PDF)
|
- Review of Computational Theory of Discounted Problems
- Value Iteration (VI)
- Policy Iteration (PI)
- Optimistic PI
- Computational Methods for Generalized Discounted Dynamic Programming
- Asynchronous Algorithms
|
|
Lecture 17 (PDF)
|
- Undiscounted Problems
- Stochastic Shortest Path Problems
- Proper and Improper Policies
- Analysis and Computational Methods for SSP
- Pathologies of SSP
- SSP Under Weak Conditions
|
|
Lecture 18 (PDF)
|
- Undiscounted Total Cost Problems
- Positive and Negative Cost Problems
- Deterministic Optimal Cost Problems
- Adaptive (Linear Quadratic) Dynamic Programming
- Affine Monotomic and Risk Sensitive Problems
|
|
Lecture 19 (PDF)
|
- Introduction to approximate Dynamic Programming
- Approximation in Policy Space
- Approximation in Value Space, Rollout / Simulation-based Single Policy Iteration
- Approximation in Value Space Using Problem Approximation
|
|
Lecture 20 (PDF)
|
- Discounted Problems
- Approximate (fitted) VI
- Approximate PI
- The Projected Equation
- Contraction Properties: Error Bounds
- Matrix Form of the Projected Equation
- Simulation-based Implementation
- LSTD, LSPE, and TD Methods
|
|
Lecture 21 (PDF)
|
- Review of Approximate Policy Iteration
- Projected Equation Methods for Policy Evaluation
- Simulation-Based Implementation Issues, Multistep Projected Equation Methods
- Bias-Variance Tradeoff
- Exploration-Enhanced Implementations, Oscillations
|
|
Lecture 22 (PDF)
|
- Aggregation as an Approximation Methodology
- Aggregate Problem
- Simulation-based Aggregation
- Q-Learning
|
|
Lecture 23 (PDF)
|
- Additional Topics in Advanced Dynamic Programming
- Stochastic Shortest Path Problems
- Average Cost Problems
- Generalizations
- Basis Function Adaptation
- Gradient-based Approximation in Policy Space
- An Overview
|
