Course Description: Reinforcement learning is the branch of machine learning that studies how an agent can learn from taking actions and receiving feedback in an unknown environment. In the past decade, reinforcement learning algorithms have demonstrated impressive empirical successes ranging from beating a human world champion at the board game Go to allowing robots to learn difficult manipulation skills. This course aims to introduce students to reinforcement learning and topics at the forefront of reinforcement learning research. The first half of the course provides an introduction to RL fundamentals and then covers advanced topics in the reinforcement learning research literature such as hierarchical and deep reinforcement learning. The course will assume familiarity with probability, statistics, and topics covered in an introductory machine learning class. Familiarity with the Python programming language is recommended.

Course learning outcomes: Students will understand fundamental concepts and algorithms in reinforcement learning. Students will gain familiarity with topics at the forefront of empirical reinforcement learning research. Students will be able to conceive and complete a small scale reinforcement learning research project.

Number of credits associated with the course: 3

How credit hours are met by the course: This class meets for two 75-minute class periods each week over the semester and carries the expectation that students will work on course learning activities (reading, writing, problem sets, studying, etc) for about 3 hours out of classroom for every class period. The syllabus includes more information about meeting times and expectations for student work.

Prerequisite: None but the course will assume familiarity with probability, statistics, linear algebra, and topics covered in an introductory machine learning class (linear regression, neural networks, etc). Familiarity with the Python programming lanaguage is recommended.

Time: TR 9:30AM - 10:45AM

Location: CS Building, Room 1263

Textbook: Reinforcement Learning: An Introduction (2nd edition). Rich Sutton and Andy Barto. MIT Press, 2018. ISBN 9780262039246. (Available for free online: http://incompleteideas.net/book/RLbook2020.pdf)

• Understand and be able to apply fundamental RL concepts and algorithms.
• Understand advanced topics in RL research such as deep RL, hierarchical RL, and multi-agent RL.
• Have completed an RL research project including an implementation component and experimental analysis of implementation.

In the regular lecture time (Tuesday and Thursday 9:30am-10:45am CT), we will have class in CS 1263, during which the instructor will lecture and the class will engage in discussions and Q&A.

Each week will have assigned readings and the expectation is that all students will complete readings before lectures. The instructor will answer questions from the readings in class and lecture on the week's topic.

Each class period will also involve two 10-minute student led presentations on a topic that complements the week's topic. The aim of these presentations is to stimulate discussion or introduce the entire class to a research paper or application that relates to the lecture topic.

We will use Piazza for Q&A outside lectures. Please follow these rules:

• Please check if someone has posted the same / similar question before you; it’s much easier if we build on the thread.
• Use an informative Summary line to help others.

The following weights are used:

• Weekly Readings and Questions: 10%
  • More details here.
• Paper Presentation: 10%
  • More details here.
  • Sign-up for a slot here.
• Class Participation: 10%
  • Attendence, asking and answering questions or making comments in class or Piazza.
• Final Programming Project: 40%
  • Project Proposal: 5%
  • Literature Survey: 10%
  • Final Report and Code: 25%
  • More details on the project page
• Programming Assignments: 30%

McBurney Center students should contact the instructors to specify any special requests for the exams or homework assignments together with the supporting documentation provided by the McBurney Center. We will do our best to accommodate the requests.

All assignments are due when specied by the instructor. Late assignments will have 10% deducted for each 24 hours past the due date. This penalty is capped at 50% after which no credit is received except for weekly reading responses. Weekly reading responses may be turned in up to the final class day with a penalty of up to 50% off. In the event of illness or emergency that prevents an on-time completion, please contact the instructor prior to the deadline.

The culmination of the course is a final project in which you will implement RL algorithms to answer a research question or address a particular application. In either case, the project grade will come from 1) a project proposal, 2) a literature survey, and 3) a final report that describes an implementation and experiments. The experimental section of the report must state a clear hypothesis, describe one or more experiments designed to validate that hypothesis, and report and analyze results of the experiments.

Final project grading questions must be raised with the instructor within 72 hours after it is returned. If a regrade request is submitted for a part of a project, the grader reserves the right to regrade the entire project and could potentially take points off.

Final project timeline

• Proposal Due : October 6 (Tentative)

• Literature Survey Due : November 3, 5pm (Tentative)

• Final Report Due : December 14, 5pm (Tentative)

The instructor will hold office hours virtually or in-person. For Fall 2022, office hours will be from 11am-12pm on Tuesdays (after lecture) or by appointment.