Project Page

A major component of your course grade is based upon the completion of a final project. This project must include an implementation component and scientific evaluation of the implementation.

You have two choices for the type of project you do:

Application Project: Choose a domain of interest, define an RL problem in that domain, and apply an RL algorithm to solve the problem.
Algorithm Project: Propose a modification to an RL algorithm that we read about in class or you read about elsewhere. Demonstrate that the modification leads to some type of improvement over the original algorithm.

By default, projects should be completed individually. If you would like to form a team of up to 2 people, please discuss with me beforehand. You will each need to write a separate proposal, survey, and final report. Each report should clearly describe the contribution of each team member.

Project Proposal

Your first task is to write a project proposal describing what you will do for your project, what your goals are, and how you will evaluate success.

Send the proposal as a plain text email with the subject line "CS 839: Project Proposal."

You will receive full credit if the following questions are answered:

What is the goal of your project?
Why is the goal interesting and non-trivial?
How will you evaluate success?
What (if any) code do you have to build upon?
(If application project) Formalize the problem as an MDP. What are the states, actions, rewards, and discount?
(If algorithms project) Describe the RL environments (i.e., MDPs) you will use to evaluate your proposed modification.
Team members (if any).

The more precise you write your proposal, the more helpful my feedback can be. I'm also happy to discuss projects before the proposal is due.

Literature Survey

Your second task is to write a literature survey that covers the relevant research literature on your project topic. This survey will have three sections:

Summarize the plan for your project and any changes to it since the proposal was submitted. If nothing has changed since the proposal, simply state that.
(Most important) The actual literature survey.
- The survey should be roughly 1-1.5 columns in a two column format.
- The survey must include at least 10 surveyed references. More is fine too.
- For each surveyed reference discussed, say 1) what was done, 2) how it relates to your project, 3) how your proposed project will be novel relative to this work. Example: Smith et al. applied policy gradient RL to learn a walking controller for a quadrupedal robot [2020]. While this project also considers learning of walk controllers, we focus on the more challenging task of bipedal locomotion and will use an actor-critic algorithm.
- You may include other references in the survey without completing these 3 steps. However, you must complete the 3 steps for at least 10 references.
A bibliography with full citations for each included reference.

Note that the survey will become a component in the final project report.

Literature Survey Guidelines:

Use of Latex is highly encouraged for type-setting your survey and final report. Overleaf.com is a great resource for quickly creating Latex projects in a browser.
Citations go at the end of sentences.
Citations are not nouns. Don't say "(Smith et al. 2020) applied policy gradient RL." Say "Smith et al. applied policy gradient RL (2020)."

Final Project Implementation and Report

Implementation:

Submit code and executable.
README with directions for setting up and running your code. MAKE SURE IT IS ABSOLUTELY CLEAR HOW TO RUN YOUR CODE.

Empirical Analysis:

Identifies one empirical question.
Forms and states at least one hypothesis as an answer to the question.
An experiment that allow you to either accept or reject the hypothesis.
An ablation experiment that shows the impact of different components of your implementation.
A sensitivity experiment that shows how sensitive your implementation is to one or more hyper-parameters.

You should carefully consider any feedback you received on homeworks 3 and 4 when conducting your empirical analysis.

Report:

Approximately 4, two-column pages with 1 inch margins.
Includes an introduction that motivates the application or algorithmic modifications.
Includes your literature review in paragraph form (not bullet points!)
Desciption of your approach or application and implementation.
Evaluate good and bad aspects of your implementation.
Empirical analysis as specified above. The analysis should be reproducible, following standards discussed in "Deep Reinforcement Learning that Matters."
Proofread and spell check.
If your implementation requires setting hyper-parameters (e.g., neural network architectures, step sizes), make sure you specify both what values you used and how you arrived at them.