CS 639 (Spring 2021) Topics in Sequential Decision Making and Learning




Description
In artificial intelligence, sequential decision making refers to agents that make decisions over time.  
Importantly, the world gives feedback to the agent after each decision, and may change the environment 
surrounding the agent.  Earlier agent decisions affect the availability and quality of future decision 
options.  The agent must improve itself from the feedback and make good decisions as time goes on.  
This is in contrast to supervised learning where learning is typically done only once.   The focus of 
the course will be on reinforcement learning, though we will also discuss active learning, multi-armed 
bandits, and stochastic games.  Mathematical maturity (probability and statistics, linear algebra, 
calculus), programming skills (data structure, python), and knowledge of machine learning at the level
of cs540 or cs532 are necessary.  This is an undergraduate level course.

Prereq 
Prerequisites: CS540 OR CS532.

Instructor
Professor Jerry Zhu, jerryzhu@cs.wisc.edu

Time and location




Syllabus
Week 1: Probably Approximately Correct: supervised learning, active learning
	Foundations of Machine Learning. Mohri, Rostamizadeh, Talwalkar. Second Ed, 2018.  Ch 1, 2 (may skip 2.3 and beyond)
	Theory of Active Learning. Hanneke. 2014.  Section 1.3
	Optional: Active Learning. Settles 2012. (download from UW IP address)
Week 2: multi-armed bandits
	Chapter 2.1-2.7 of textbook
	Chapter 1, 4.1-4.5, 6, 7 (may skip proofs) in Bandit Algorithms.  Lattimore and Szepesvari, 2020.
Week 3: contextual bandits, best arm identification
	Chapter 18.1, 19.1-19.2, 33.1 (may skip proofs) in Lattimore and Szepesvari
Week 4: Markov Decision Process
	Ch 1; Part I Tabular Solution Methods intro (p23); Ch 3 in textbook
	1.1, 1.2, 1.4 in Reinforcement Learning: Theory and Algorithms. Agarwal et al 2021. (may skip proofs)
Week 5: Value functions, Bellman equations
	Ch 5, 6 in textbook
Week 6: Planning in MDP (policy iteration, value iteration)
	Cuttlefish exert self-control in a delay of gratification task by Schnell et al. 2021
Week 7: Monte Carlo, temporal difference
	Ch 9 in textbook
Week 8: SARSA, Q-learning, function approximation
	Read Monte Carlo tutorial at least to section 5.2.  Then revisit sections 5.5 and 5.7 in the textbook.
Week 9: Off-policy methods
	Ch 13 in textbook
Week 10: policy gradient
	Read R-max and UCBVI
Week 11: exploration
	An Algorithmic Perspective on Imitation Learning.  Osa et al. Foundations and Trends in Robotics, 2018. 
	Sections 1.1-1.4, 2.2, 2.4, 2.6, 3.1, 3.4.3.3, 4.1
Week 12: imitation learning 
	Algorithmic Game Theory.  Nisan et al. 2007.  Sections 1.1-1.7
Week 13: stochastic games
	An Overview of Multi-Agent Reinforcement Learning from Game Theoretical Perspective. Yang, Wang. 2021. Sections 1-4
Week 14: stochastic games

Textbooks

Reinforcement Learning: An Introduction.  Richard S. Sutton and Andrew G. Barto.  Second Edition. MIT Press, Cambridge, MA, 2018.

Grading: weekly reading summary (40%), math / coding homework (40%), exams (20%)





Homework
All assignments are in Canvas.  There are two kinds of homework: 

1. Weekly reading summary.  Usually posted on Thursdays and due on Mondays at 5pm, students submit a 
paragraph in Canvas in response to each reading assignment.   The reading assignment will specify the 
book chapters or papers to read, and then may either ask a specific question or be open-ended.  Grade 
will be based on evidence that you have done the readings thoughtfully. For open-ended reading summary,
you may pose insightful questions, relate to previous classes, suggest in-depth discussion directions,
summarize key points you learned from reading, etc.   The Monday 5pm deadline enables the instructor 
to potentially incorporate your respnoses in teaching for that week.  

2. Math and coding problems.  This is the traditional homework, assigned about every two weeks.  
Homework is always due the minute before class starts on the due date (usually Thursdays at 9:29am).

Late submissions will not be accepted.  However, we will automatically drop two lowest weekly reading 
summary scores and two lowest math and coding problem scores from your final homework average 
calculation.  These drops are meant for emergency.  We do not provide additional drops, late days, or 
homework extensions.  Grading questions must be raised with the instructor within one week after it is 
returned.  Regrading request for a part of a homework question may trigger the grader to regrade the 
entire homework and could potentially take points off.  Regrading will be done on the original 
submitted work, no changes allowed.  We encourage you to use a study group for doing your homework.  
Students are expected to help each other out, and if desired, form ad-hoc homework groups.

Exam
Final exam: Tuesday May 4 12:25-2:25pm Madison time, via Canvas assignment.

Academic Integrity: 
You are encouraged to discuss with your peers, the TA or the instructors ideas, approaches 
and techniques broadly.  However, all examinations, programming assignments, and written 
homeworks must be written up individually.  For example, code for programming assignments 
must not be developed in groups, nor should code be shared.  Make sure you work through all 
problems yourself, and that your final write-up is your own.  If you feel your peer discussions 
are too deep for comfort, declare it in the homework solution: "I discussed with X,Y,Z the following
specific ideas: A, B, C; therefore our solutions may have similarities on D, E, F..."

You may use books or legit online resources to help solve homework problems, but you must always credit 
all such sources in your writeup and you must never copy material verbatim.

Do not bother to obfuscate plagiarism (e.g. change variable names, code style, etc.)  One application 
of AI is to develop sophisticated plagiarism detection techniques!

Cheating and plagiarism will be dealt with in accordance with University procedures 
(see the UW-Madison Academic Misconduct Rules and Procedures).

Disability Information
The University of Wisconsin-Madison supports the right of all enrolled students to a full and equal 
educational opportunity. The Americans with Disabilities Act (ADA), Wisconsin State Statute (36.12), 
and UW-Madison policy (Faculty Document 1071) require that students with disabilities be reasonably 
accommodated in instruction and campus life. Reasonable accommodations for students with disabilities 
is a shared faculty and student responsibility. Students are expected to inform Professor Zhu of their 
need for instructional accommodations by the end of the third week of the semester, or as soon as 
possible after a disability has been incurred or recognized.  Professor Zhu will work either directly 
with the student or in coordination with the McBurney Center to identify and provide reasonable 
instructional accommodations. Disability information, including instructional accommodations as part 
of a student's educational record, is confidential and protected under FERPA.

Additional Course Information

Class learning outcome
Student will be able to: 
 - gain familiarity with advanced learning paradigms, including active learning, multi-armed bandits, reinforcement learning.
 - implement basic sequential decision making algorithms
 - understand basic theoretical analysis in sequential decision making

Number of credits associated with the course: 3

How credit hours are met by the course: For each 50min of classroom instruction, a minimum of two hours 
of out of class student work is expected.  This course has two 75-minute classes each week over approximately 
15 weeks, which amounts to the standard definition of a 3-credit course.