Nonparametric Bayesian Reading Group
Fall 2009

Day/time: Wednesdays from 1-2pm
Location: CS 4310 for the rest of the semester.
Mailing list: sign up here
Organizer: Andrew Goldberg

Description: This reading group will introduce the "hot" topic of Nonparametric Bayesian modeling and its applications in machine learning. As we will learn, nonparametric methods make fewer assumptions than their parametric counterparts and can be used to model a wide range of data. After reviewing Bayesian fundamentals, we will discuss several popular building blocks and nonparametric models, including Gaussian Processes, Dirichlet Processes (DP), Dirichlet Process Mixture Models (DPMM), Hierarchical DP, and Indian Buffet Processes. We will then focus in more depth on various approximate inference methods used for fitting or training such models: sampling methods, variational inference, and deterministic inference procedures. Participants will be asked to read papers in advance and take turns leading the discussion.

Prerequisites: None, except it would help to have some basic background in probability and statistics.

Schedule: An up-to-date schedule will be maintained as a Google Calendar (see below), but the order of topics will follow roughly what is mapped out below.


1 mtg: Review of Bayesian Methods

9/9 Room CS 4310:
Tutorial on Bayesian Machine Learning
Zoubin Ghahramani
ICML 2004 tutorial

Bayesian Inference: An Introduction to Principles and Practice in Machine Learning
Michael E. Tipping
In O. Bousquet, U. von Luxburg, and G. Ratsch (Eds.), Advanced Lectures on Machine Learning, pp. 41-62. Springer.

2 mtgs: Introduction to Non-parametric Bayesian and Gaussian Processes

9/16 Room CS 3310

Nonparametric Bayesian Methods
Zoubin Ghahramani
UAI 2005 Tutorial 

Advances in Gaussian Processes
Carl Edward Rasmussen 
NIPS 2006 Tutorial 

Gaussian Process Basics
Video lecture by David MacKay 

9/23 Room CS 4310

Gaussian Processes for Machine Learning
Rasmussen and Williams (textbook available online as a PDF) 
Recommended: Chapters 1-3
Optional: Chapters 4-6

3 mtgs: Dirichlet Processes and Dirichlet Process Mixture Models

9/30 Room CS 4310: Finish GP (classification and choosing a kernel function) and start DP (see references below) 

10/7 Room CS 4310: Continue DP using Teh tutorial

10/14 Room CS 4310: Professor Michael Newton from Statistics will discuss his personal interactions with DP models.

10/21 Room CS 4310: DPMMs for clustering

10/28 Room CS 4310: DPMM inference
Dirichlet Processes---MLSS 2007
Y.W. Teh. Machine Learning Summer School 2007 Tutorial and Practical Course.

Volker Tresp
Dirichlet Processes and Nonparametric Bayesian Modelling

Nonparametric Bayesian Discrete Latent Variable Models for Unsupervised Learning.
Recent PhD thesis from Dilan Gorur with a great introductory chapter on DP.

Neal, R.M. (2000). 
Markov chain sampling methods for Dirichlet process mixture models. 
Journal of Computational and Graphical Statistics, 9, 249-265.

Rasmussen, C.E. (2000). 
The infinite gaussian mixture model. 
In Advances in Neural Information Processing Systems 12. Cambridge, MA: MIT Press.

11/4 Room CS 4310: HDP

1 mtg: Hierarchical Dirichlet Process

Hierarchical Dirichlet Processes 
Teh, Jordan, Beal and Blei, 2006

2 mtgs: Indian Buffet Process

11/11, 11/18 Room CS 4310: Indian Buffet Process led by Dave

Indian Buffet Process tech report (starts with a Dirichlet Process review/comparison)

Slides about IBP

Bayesian nonparametric latent feature models
Griffiths and Ghahramani, 2006

1 mtg: Variational Inference

Structured Bayesian nonparametric models with variational inference (tutorial).
Percy Liang, Dan Klein.
Association for Computational Linguistics (ACL), 2007. 

2 mtgs: MCMC and Sampling methods

An Introduction to MCMC for Machine Learning 
Andrieu, de Freitas, Doucet and Jordan, 2003

More references coming soon

1 mtg: Deterministic approximate inference: Laplace's method, Variational Bayes, and expectation propagation

References coming soon

Other useful references:

Nonparametric Bayes Workshop at ICML/UAI/COLT 2008