Dynamic Programming and Stochastic Control | Electrical Engineering and Computer Science | MIT OpenCourseWare

Lecture Notes

6.231 Fall 2015 Complete Lecture Notes

6.231 Fall 2015 Lecture 1: Introduction to Dynamic Programming, Examples of Dynamic Programming, Significance of Feedback

6.231 Fall 2015 Lecture 10: Infinite Horizon Problems, Stochastic Shortest Path (SSP) Problems, Bellman’s Equation, Dynamic Programming – Value Iteration, Discounted Problems as a Special Case of SSP

6.231 Fall 2015 Lecture 11: Review of Stochastic Shortest Path Problems, Computation Methods for SSP, Computational Methods for Discounted Problems

6.231 Fall 2015 Lecture 12: Average Cost Per Stage Problems, Connection With Stochastic Shortest Path Problems, Bellman’s Equation, Value Iteration, Policy Iteration

6.231 Fall 2015 Lecture 13: Control of Continuous-Time Markov Chains: Semi-Markov Problems, Problem Formulation: Equivalence to Discrete-Time Problems, Problem Formulation: Equivalence to Discrete-Time Problems, Average Cost Problems

6.231 Fall 2015 Lecture 14: Introduction to Advanced Infinite Horizon Dynamic Programming and Approximation Methods

6.231 Fall 2015 Lecture 15: Review of Basic Theory of Discounted Problems, Monotonicity of Contraction Properties, Contraction Mappings in Dynamic Programming, Discounted Problems: Countable State Space with Unbounded Costs, Generalized Discounted Dynamic Programming, An Introduction to Abstract Dynamic Programming

6.231 Fall 2015 Lecture 16: Review of Computational Theory of Discounted Problems, Value Iteration (VI), Optimistic PI, Computational Methods for Generalized Discounted Dynamic Programming, Asynchronous Algorithms

6.231 Fall 2015 Lecture 17: Undiscounted Problems, Stochastic Shortest Path Problems, Proper and Improper Policies, Analysis and Computational Methods for SSP, Pathologies of SSP, SSP Under Weak Conditions

6.231 Fall 2015 Lecture 18: Undiscounted Total Cost Problems, Positive and Negative Cost Problems, Proper and Improper Policies, Adaptive (Linear Quadratic) Dynamic Programming, Affine Monotomic and Risk Sensitive Problems

6.231 Fall 2015 Lecture 19: Introduction to affirmative Dynamic Programming, Approximation in Policy Space, Approximation in Value Space, Rollout/Simulation-based Single Policy Iteration, Approximation in Value Space Using Problem Approximation

6.231 Fall 2015 Lecture 2: The Basic Problem, Principle of Optimality, The General Dynamic Programming Algorithm, State Augmentation

6.231 Fall 2015 Lecture 20: Discounted Problems, Approximate (fitted) VI, Approximate PI, The Projected Equation, Contraction Properties: Error Bounds, Matrix Form of the Projected Equation, Simulation-based Implementation, LSTD and LSPE Methods

6.231 Fall 2015 Lecture 21: Review of Approximate Policy Iteration, Projected Equation methods for Policy Evaluation, Simulation-Based Implementation Issues, Multistep Projected Equation Methods, Bias-Variance Tradeoff, Exploration-Enhanced Implementations, Oscillations

6.231 Fall 2015 Lecture 22: Aggregation as an Approximation Methodology, Aggregate Problem, Simulation-based Aggregation, Q-Learning

6.231 Fall 2015 Lecture 23: Additional Topics in Advanced Dynamic Programming, Stochastic Shortest Path Problems, Average Cost Problems, Generalizations, Basis Function Adaption, Gradient-based Approximation in Policy Space, An Overview

6.231 Fall 2015 Lecture 3: Deterministic Finite-State Problem, Backward Shortest Path Algorithm, Forward Shortest Path Algorithm, Alternative Shortest Path Algorithms

6.231 Fall 2015 Lecture 4: Examples of Stochastic Dynamic Programming Problems, Linear-Quadratic Problems, Inventory Control

6.231 Fall 2015 Lecture 5: Stopping Problems, Scheduling Problems, Minimax Control

6.231 Fall 2015 Lecture 6: Problems with Imperfect State Info, Reduction to the Perfect State Info Cast, Linear Quadratic Problems, Separation of Estimation and Control

6.231 Fall 2015 Lecture 7: Imperfect State Information, Sufficient Statistics, Conditional State Distribution as a Sufficient Statistic, Finite-State Analysis

6.231 Fall 2015 Lecture 8: Suboptimal Control, Cost Approximation Methods: Classification, Certainty Equivalent Control, Limited Lookahead Policies, Performance Bounds, Problem Approximation Approach, Parametric Cost-To-Go Approximation

6.231 Fall 2015 Lecture 9: Rollout Algorithms, Cost Improvement Property, Discrete Deterministic Problems, Approximations to Rollout Algorithms, Model Predictive Control (MPS), Discretization of Continuous Time, Discretization of Continuous Space, Other Suboptimal Approaches

Course Info

Instructor

Prof. Dimitri Bertsekas

Departments

Electrical Engineering and Computer Science

As Taught In

Fall 2015

Level

Graduate

Topics

Learning Resource Types

Exam Solutions

Exams

Lecture Notes

Other Video

Problem Sets

Download Course

MIT Open Learning

Over 2,500 courses & materials

Freely sharing knowledge with learners and educators around the world. Learn more

© 2001–2026 Massachusetts Institute of Technology

Accessibility

Creative Commons License

Terms and Conditions

Proud member of:

© 2001–2026 Massachusetts Institute of Technology