Nonparametric Bayesian Reading Group
Fall 2009
Day/time: Wednesdays from 1-2pm
Location: CS 4310 for the rest of the semester.
Mailing list: sign up here
Organizer: Andrew Goldberg
Description: This reading group will introduce the "hot" topic of Nonparametric Bayesian modeling and its applications in machine learning. As we will learn, nonparametric methods make fewer assumptions than their parametric counterparts and can be used to model a wide range of data. After reviewing Bayesian fundamentals, we will discuss several popular building blocks and nonparametric models, including Gaussian Processes, Dirichlet Processes (DP), Dirichlet Process Mixture Models (DPMM), Hierarchical DP, and Indian Buffet Processes. We will then focus in more depth on various approximate inference methods used for fitting or training such models: sampling methods, variational inference, and deterministic inference procedures. Participants will be asked to read papers in advance and take turns leading the discussion.
Prerequisites: None, except it would help to have some basic background in probability and statistics.
Schedule: An up-to-date schedule will be maintained as a Google Calendar (see below), but the order of topics will follow roughly what is mapped out below.
Papers:
1 mtg: Review of Bayesian Methods
9/9 Room CS 4310:
Tutorial on Bayesian Machine Learning
Zoubin Ghahramani
ICML 2004 tutorial
Bayesian Inference: An Introduction to Principles and Practice in Machine Learning
Michael E. Tipping
In O. Bousquet, U. von Luxburg, and G. Ratsch (Eds.), Advanced Lectures on Machine Learning, pp. 41-62. Springer.
2 mtgs: Introduction to Non-parametric Bayesian and Gaussian Processes
9/16 Room CS 3310
Nonparametric Bayesian Methods
Zoubin Ghahramani
UAI 2005 Tutorial
Advances in Gaussian Processes
Carl Edward Rasmussen
NIPS 2006 Tutorial
Gaussian Process Basics
Video lecture by David MacKay
9/23 Room CS 4310
Gaussian Processes for Machine Learning
Rasmussen and Williams (textbook available online as a PDF)
Recommended: Chapters 1-3
Optional: Chapters 4-6
3 mtgs: Dirichlet Processes and Dirichlet Process Mixture Models
9/30 Room CS 4310: Finish GP (classification and choosing a kernel function) and start DP (see references below)
10/7 Room CS 4310: Continue DP using Teh tutorial
10/14 Room CS 4310: Professor Michael Newton from Statistics will discuss his personal interactions with DP models.
10/21 Room CS 4310: DPMMs for clustering
10/28 Room CS 4310: DPMM inference
Dirichlet Processes---MLSS 2007
Y.W. Teh. Machine Learning Summer School 2007 Tutorial and Practical Course.
Volker Tresp
Dirichlet Processes and Nonparametric Bayesian Modelling
Nonparametric Bayesian Discrete Latent Variable Models for Unsupervised Learning.
Recent PhD thesis from Dilan Gorur with a great introductory chapter on DP.
Neal, R.M. (2000).
Markov chain sampling methods for Dirichlet process mixture models.
Journal of Computational and Graphical Statistics, 9, 249-265.
Rasmussen, C.E. (2000).
The infinite gaussian mixture model.
In Advances in Neural Information Processing Systems 12. Cambridge, MA: MIT Press.
11/4 Room CS 4310: HDP
1 mtg: Hierarchical Dirichlet Process
Hierarchical Dirichlet Processes
Teh, Jordan, Beal and Blei, 2006
2 mtgs: Indian Buffet Process
11/11, 11/18 Room CS 4310: Indian Buffet Process led by Dave
Indian Buffet Process tech report (starts with a Dirichlet Process review/comparison)
Slides about IBP
Bayesian nonparametric latent feature models
Griffiths and Ghahramani, 2006
1 mtg: Variational Inference
Structured Bayesian nonparametric models with variational inference (tutorial).
Percy Liang, Dan Klein.
Association for Computational Linguistics (ACL), 2007.
2 mtgs: MCMC and Sampling methods
An Introduction to MCMC for Machine Learning
Andrieu, de Freitas, Doucet and Jordan, 2003
More references coming soon
1 mtg: Deterministic approximate inference: Laplace's method, Variational Bayes, and expectation propagation
References coming soon
Other useful references:
Nonparametric Bayes Workshop at ICML/UAI/COLT 2008