CptS 580: Reinforcement Learning
Spring 2015

Matthew E. Taylor (Matt)
EME 137
Syllabus: Spring 2015
Reinforcement Learning: An Introduction  
Please use:


1/13 First day of class: Introduction  
1/15 Secord day of class: Introduction, continued Read: Chapter 1
Udacity: Sign up for Machine Learning 3 - Reinforcement Learning
Watch: "Introduction" through "Markov Decision Pocess Four" (8 videos total)
Sign up for Piazza
1/20 Bandits! Read: Chapter 2
Udacity: More About Rewards (1 & 2) + Rewards Quiz
Please write a response to both on Piazza by 5pm on Monday
1/22 MDPs Read up through section 3.5 in the book
Please write a response on Piazza by 5pm on Wednesday
1/27 No class: AAAI  
1/29 No class: AAAI Read up through chapter 4
Finish watching RL 1 - Markov Decision Processes
Please write a response on Piazza by 5pm on Wednesday
2/3 Dynamic Programming  
2/5 No class: NSF  
2/6 Makeup class: 9:30-11am = Dynamic Programming
2/10 Monte Carlo Methods Read up through chapter 5
Watch RL 2: Reinforcement Learning - Three Approaches to RL (5 videos)
Please write a response on Piazza by 5pm on Monday
2/12 Monte Carlo Methods Watch RL 2: "A New Kind of Value Function" - "Learning Incrementally" (5 videos)
No response required
2/17 Temporal Difference Methods Read up through chapter 6
Watch RL 2: "Estimating Q From Transitions 2" - "What Have We Learned" (6 videos)
Please write a response on Piazza by 5pm on Monday on Chapter 6 and/or the last 11 videos
2/19 Temporal Difference Methods  
2/24 Eligibility Traces Read chapter 7, write a Piazza response by 5pm on Monday
2/26 Eligibility Traces  
3/3 Function Approximation. Class video Read chapter 8, write a Piazza response by 5pm on Monday
3/5 Function Approximation. Class video  
3/10 Planning and Model Learning Read chapter 9, write a Piazza response by 5pm on Monday
3/12 Planning and Model Learning  
3/17 No class: Spring Break  
3/19 No class: Spring Break  
3/24 Discussion of RAM-RMAX Read Efficient Reinforcement Learning with Relocatable Action Models and write a response on Piazza by 5pm on Monday.
3/26 Guest Lecture on Multi-agent Learning: Carrie Rebhuhn  
3/31 Shaping Rewards  
4/2 Intrinsic Motivation in RL  
4/7 Transfer Learning  
4/9 Leah 1-3 paragraph final project proposal posted to Piazza by 5pm on 4/8
Read paper on Extrinsic/Intrinsic Motivation (see piazza)
4/14 Yang Please look at this website and paper
4/16 Zhaodong  
4/21 James  
4/23 Sal  
4/28 Duy  
4/30 Final Class Final Project (Draft) Due
5/7   Final Project Due.
For formatting, I suggest you use the AAAI-15 style in either latex or Word.

Possible further topics
  • Current Function Approximation Choices
  • Efficient Model-Learning methods
  • Hierarchical Methods
  • Game Playing
  • Learning in Robotics
  • Transfer Learning
  • Shaping Rewards
  • Learning from Human Rewards
  • Learning from Demonstration
  • Multi-agent RL
  • Partially observable envirnments and/or POMDPs
  • Meta-RL and empirical evaluation of algorithms
  • Least Squares methods (e.g., LSPI)
  • Adaptive Representations / Representation Learning
  • Case Studies: Robot soccer, Helicopter Control, etc.
  • Inverse Reinforcement Learning (IRL)
  • Intrinsicly Motivated Reinforcement Learning
  • Actor-Critic Methods
  • Policy Gradient methods
  • Crowd Sourcing (?)