Decision Making in Large Scale Systems

Readings

Bertsekas = Bertsekas, Dimitri P. Dynamic Programming and Optimal Control. 2 vols. Belmont, MA: Athena Scientific, 2007. ISBN: 9781886529083.

Bertsekas and Tsitsiklis = Bertsekas, Dimitri P., and John N. Tsitsiklis. Neuro-Dynamic Programming. Belmont, MA: Athena Scientific, 1996. ISBN: 9781886529106.

LEC #	TOPICS	READINGS
1	Markov Decision Processes Finite-Horizon Problems: Backwards Induction Discounted-Cost Problems: Cost-to-Go Function, Bellman’s Equation	Bertsekas Vol. 1, Chapter 1.
2	Value Iteration Existence and Uniqueness of Bellman’s Equation Solution Gauss-Seidel Value Iteration	Bertsekas Vol. 2, Chapter 1. Bertsekas and Tsitsiklis, Chapter 2.
3	Optimality of Policies derived from the Cost-to-Go Function Policy Iteration Asynchronous Policy Iteration	Bertsekas Vol. 2, Chapter 1. Bertsekas and Tsitsiklis, Chapter 2.
4	Average-Cost Problems Relationship with Discounted-Cost Problems Bellman’s Equation Blackwell Optimality	Bertsekas Vol. 2, Chapter 4.
5	Average-Cost Problems Computational Methods	Bertsekas Vol. 2, Chapter 4.
6	Application of Value Iteration to Optimization of Multiclass Queueing Networks Introduction to Simulation-based Methods Real-Time Value Iteration	Chen, R. R., and S. P. Meyn. “Value Iteration and Optimization of Multiclass Queueing Networks."Queueing Systems 32 (1999): 65-97. Bertsekas and Tsitsiklis, Chapter 5.
7	Q-Learning Stochastic Approximations	Bertsekas and Tsitsiklis, Chapters 4 and 5.
8	Stochastic Approximations: Lyapunov Function Analysis The ODE Method Convergence of Q-Learning	Bertsekas and Tsitsiklis, Chapters 4 and 5.
9	Exploration versus Exploitation: The Complexity of Reinforcement Learning	Kearns, M. , and S. Singh. “Near-Optional Reinforcement Learning in Polynomial Time.” Machine Learning 49, no. 2 (Nov 2002): 209-232.
10	Introduction to Value Function Approximation Curse of Dimensionality Approximation Architectures	Bertsekas and Tsitsiklis, Chapter 6.
11	Model Selection and Complexity	Hastie, Tibshirani, and Friedmann. Chapter 7 in The Elements of Statistical Learning. New York: Springer, 2003. ISBN: 9780387952840.
12	Introduction to Value Function Approximation Algorithms Performance Bounds	Bertsekas and Tsitsiklis, Chapter 6.
13	Temporal-Difference Learning with Value Function Approximation	Bertsekas and Tsitsiklis, Chapter 6.
14	Temporal-Difference Learning with Value Function Approximation (cont.)	Bertsekas and Tsitsiklis, Chapter 6. de Farias, D. P., and B. Van Roy. “On the Existence of Fixed Points for Approximate Value Iteration and Temporal-Difference Learning.”
15	Temporal-Difference Learning with Value Function Approximation (cont.) Optimal Stopping Problems General Control Problems	Bertsekas and Tsitsiklis, Chapter 6. de Farias, D. P., and B. Van Roy. “On the Existence of Fixed Points for Approximate Value Iteration and Temporal-Difference Learning.” Bertsekas, Borkar, and Nedic. “Improved temporal Difference Methods with Linear Function Approximation.”
16	Approximate Linear Programming	de Farias, D. P., and B. Van Roy. “The Linear Programming Approach to Approximate Dynamic Programming.”
17	Approximate Linear Programming (cont.)	de Farias, D. P., and B. Van Roy. “The Linear Programming Approach to Approximate Dynamic Programming.”
18	Efficient Solutions for Approximate Linear Programming	de Farias D. P., and B. Van Roy. “On Constraint Sampling in the Linear Programming Approach to Approximate Dynamic Programming.” Calafiori, and Campi. “Uncertain Convex Programs: Randomized Solutions and Confidence Levels.”
19	Efficient Solutions for Approximate Linear Programming: Factored MDPs	Guestrin, et al. “Efficient Solution Algorithms for Factored MDPs.” Schuurmans, and Patrascu. “Direct Value Approximation for Factored MDPs.”
20	Policy Search Methods	Marbach, and Tsitsiklis. “Simulation-Based Optimization of Markov Reward Processes.” (PDF)
21	Policy Search Methods (cont.)	Baxter, and Bartlett. “Infinite-Horizon Policy-Gradient Estimation.”
22	Policy Search Methods for POMDPs Application: Call Admission Control Actor-Critic Methods	Baxter, and Bartlett. “Infinite-Horizon Policy-Gradient Estimation.” Baxter, and Bartlett. “Experiments with Infinite-Horizon Policy-Gradient Estimation.” Konda, and Tsitsiklis. “Actor-Critic Algorithms.” (PDF)
23	Guest Lecture: Prof. Nick Roy Approximate POMDP Compression	Roy, and Gordon. “Exponential Family PCA for Belief Compression in POMDPs.”
24	Policy Search Methods: PEGASUS Application: Helicopter Control	Ng, and Jordan. “PEGASUS: A policy search method for large MDPs and POMDPs.” Ng, et al. “Autonomous Helicopter Flight via Reinforcement Learning.”

Complementary Reading

Even-Dar, and Mansour. “Learning Rates for Q-Learning.’’ Journal of Machine Learning Research 5 (2003): 1-25.

Barron. “Universal approximation bounds for superpositions of a sigmoidal function.” IEEE Transactions on Information Theory 39 (1993): 930-944.

Tesauro. “Temporal-Difference Learning and TD-Gammon’’ Communications of the ACM 38, no. 3 (1995).