CS 839 Fall 2023: Foundation Models & the Future of Machine Learning

Important Notes

This schedule is tentative and subject to change. Please check back often. In particular, the deadlines for the homework sets/project may change; please see Canvas for the actual deadlines.

Tentative Schedule

Date	Lecture	Readings	Homework Released	Homework Due
Thursday Sept. 7	Introduction and Course Overview
Tuesday Sept. 12	Machine Learning Mini-Review	Patterns, Predictions, and Actions
Thursday Sept. 14	Transformers & Attention	Attention Is All You Need The Illustrated Transformer
Tuesday Sept. 19	Models (Encoder-Only, Encoder-Decoder, Decoder-Only) I	BERT Paper RoBERTa Paper T5 Paper
Thursday Sept. 21	Models (Encoder-Only, Encoder-Decoder, Decoder-Only) II	GPT-3 Paper PALM Paper
Tuesday Sept. 26	Prompting I	Pre-train, Prompt, and Predict Survey Finetuned Language Models Are Zero-Shot Learners	HW 1 Released
Thursday Sept. 28	Prompting II	Prefix-Tuning Parameter-Efficient Prompt Tuning
Tuesday Oct. 3	Reasoning & Chain-of-Thought	CoT Paper Large Language Models are Zero-Shot Reasoners Tree of Thoughts
Thursday Oct. 5	In-Context Learning: Practice and Theory	ICL Survey ICL as Implicit Bayesian Inference What Makes ICL Work?
Tuesday Oct. 10	Fine-Tuning, Specialization, Adaptation	Low-Rank Adaptation of Large Language Models Fast Model Editing at Scale
Thursday Oct. 12	Training	FlashAttention Megatron-LM		Homework 1 Due
Tuesday Oct. 17	Reinforcement Learning from Human Feedback	Hugging Face RLHF Blog Training language models to follow instructions with human feedback
Thursday Oct. 19	Data	The Pile Deduplication Helps LMs
Tuesday Oct. 24	Multimodal and Specialized Foundation Models (CodeLMs and more)	CodeGen Paper Flamingo Paper AlphaFold Paper ClimaX Paper LegalBench Paper
Thursday Oct. 26	Knowledge	Language Models as Knowledge Bases? Unifying LLMs and KGs
Tuesday Oct. 31	Scaling & Scaling Laws	"Compute-Optimal" LLMs Scaling Laws for Neural LMs
Thursday Nov. 2	Project & Presentation Details	Extracting Training Data from LLMs LLMs Can Be Strong Differentially Private Learners Stochastic Parrots Paper Red Teaming Paper	Due: 1/2 page info on presentations
Tuesday Nov. 7	Security, Privacy, Toxicity + Future Areas
To Dec. 13	Student Presentations