AMATH/CSE 579: Intelligent control through learning and optimization

SLN 19363, MW 3:00-4:20, ARC G070

Emo Todorov

Guggenheim 415h
todorov@cs.washington.edu

Course Description

Design of near-optimal controllers for complex dynamical systems, using analytical techniques, machine learning, and optimization. Topics from deterministic and stochastic optimal control, reinforcement learning and dynamic programming, numerical optimization in the context of control, and robotics. Prerequisite: vector calculus, linear algebra, and Matlab. Recommended: differential equations, stochastic processes, and optimization.

Syllabus

See introductory lecture.

Lecture slides

Lecture 1: Introduction

Lecture 2: Markov Decision Processes and Bellman Equations

Lecture 3: Controlled Diffusions and Hamilton-Jacobi-Bellman Equations

Lecture 4: Deterministic Systems and Pontryagin's Maximum Principle

Lecture 5: Linear-Quadratic-Gaussian (LQG) Controllers and Kalman Filters

Lecture 6: Linearly-Solvable Stochastic Optimal Control Problems

Lecture 7: Inverse Optimal Control

Lecture 8: Trajectory-based Optimization

Lecture 9: Numerical Optimization

Lecture 10: Optimal control in locomotion

Lecture 11: Function approximation methods

Lecture 12: Applications to biological movement

General Readings

A. Barto and R. Sutton (1998) Reinforcement learning: An introduction (online book)

E. Todorov (2006) Optimal control theory (book chapter)

D. Bertsekas (2008) Dynamic programming (lecture slides)

R. Tedrake (2009) Underactuated robotics: Learning, planning and control (lecture notes)

B. Van Roy (2004) Approximate dynamic programming (lecture notes)

P. Abbeel (2009) Advanced robotics (lecture slides)

Lecture-specific Readings

Lecture 6:
Todorov (2009) Efficient computation of optimal actions. PNAS 106: 11478-11483

Lecture 7:
Krishnamurthy and Todorov (2010) Inverse optimal control with linearly-solvable MDPs. In proceedings of ICML

Lecture 8:
Tassa, Erez and Smart (2007) Inverse optimal control with linearly-solvable MDPs. In proceedings of NIPS

Todorov and Li (2005) A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems. In proceedings of ACC

Lecture 10:
Muico, Lee, Popovic and Popovic (2009) Contact-aware nonlinear control of dynamics characters. In proceedings of SIGGRAPH

Wampler and Popovic (2009) Optimal gait and form for animal locomotion. In proceedings of SIGGRAPH

Lecture 11:
Bertsekas (2010) Approximate dynamic programming. In Dynamic Programming and Optimal Control, vol 2, 3rd ed

Lecture 12:
Todorov (2004) Optimality principles in sensorimotor control. Nature Neuroscience

Code

Matlab MDP solver: all problem formulations and algorithms

Acrobot dynamics
Animation of acrobot dynamics

Homeworks

Homework 1, due April 28

Default Final Project, due before the final presentations