Instructor:Ben Recht
Time: Tue & Thu, 1:00-2:15 PM
Location: 1221 Computer Science
Description: This course will address the design of provably efficient algorithms for data processing that leverage prior information. We will focus on the specific areas of compressed sensing, stochastic algorithms for matrix factorization, rank minimization, and non-parametric machine learning. We will emphasize the pivotal roles of convexity and randomness in problem formulation, estimation guarantees, and algorithm design. The course will provide a unified exposition of these growing research areas and is ideal for advanced graduate students who would like to apply these theoretical and algorithmic developments to their own research.
The course will roughly be broken into the following structure:
Grading: Each student will be required to attend class regularly and scribe lecture notes for at least one class. A final project will also be required. This project will require a class presentation and a written report. The project can survey literature on a related topic not covered in the course or an application of the course techniques to a novel research problem.
Prerequisites: Graduate level courses in probability (like ECE 730) and nonlinear optimization (like CS 726). An advanced level of mathematical maturity is necessary. Familiarity with elementary functional analysis (L2 spaces, Fourier transforms, etc.) will be helpful for the last part of the course. Please consult the instructor if you are unsure about your background.
Lecture 1 (01/19): Introduction. Slides
Lecture 2 (01/21): Introduction to Random Mappings.
Related Readings:
Proof of Whitney's Embedding Theorem pdf.
Lecture 3 (01/26): Random Projections Preserve
Distances. The Johnson-Lindenstrauss Lemma.
Related
Readings: Dasgupta and Gupta.
An Elementary Proof of a Theorem of Johnson and Lindenstrauss.
pdf
Lecture 4 (01/28): Epislon Nets and Embedding
Subspaces.
Related
Readings: Rudelson and Vershynin. The Smallest
Singular Value
of a Random
Rectangular Matrix. Only the first 5 paragraphs
of
Section 2, Proposition 2.1, and its proof.
pdf.
Baraniuk et al.
A Simple Proof of the Restricted Isometry Property for Random
Matrices. pdf
Lecture 5 (02/02): Sparsity and its applications.
Notes: pdf.
Lecture 6 (02/04): Prony's method.
Notes: pdf.
A proof of the invertibility of the Vandermonde System. pdf
Lecture 7 (02/09): l1 minimization, Restricted
Isometry Property.
Notes: pdf.
Lecture 8 (02/11): l1 minimization, Robust
Recovery of Sparse Signals
Lecture 9 (02/16): Matrices with the Restricted Isometry Property.
Lecture 10 (02/18): Algorithms for l1 minimization
Related
Readings:
Wolfe. The Simplex Method for Quadratic Programming. pdf.
Donoho and Tsaig. Fast Solution of l1-norm Minimization Problems When
the Solution May be Sparse. pdf.
Tropp and Wright. Computational Methods for Sparse Solution of
Linear Inverse Problems. pdf.
Lecture 11 (02/23): Matrix Norms and Rank
Lecture 12 (02/25): Rank Minimization in Data Analysis, Hardness Results.
Lecture 13 (03/02): Easily solvable rank minimization problems. Pass efficient approximations.
Lecture 14 (03/04): The nuclear norm heuristic.
Lecture 15 (03/09): RIP for low-rank matrices. The nuclear norm succeeds.
Lecture 16 (03/11): Gaussians obey the RIP for low-rank matrices.
Lecture 17 (03/16): Algorithms for Rank Minimization
Lecture 18 (03/18): Iterative Shrinkage Thresholding for Rank Minimization
Lecture 19 (03/23): Moving beyond Restricted Isometry Properties. Dual Certificates.
Lecture 20 (04/06): Function Fitting. The Bias Variance Tradeoff.
Lecture 21 (04/08): Generalization Bounds
Lecture 22 (04/13): Kernels
Lecture 23 (04/15): Approximation Theory. The curse of dimensionality. The blessing of smoothness.
Upcoming Readings
Project Presentations
(4/20) Jesse Holzer, Benjamin Recht
(4/22) Jingjiang Peng, Yongia Song, Suhail Shergill
(4/27) Laura Balzano, Badri Bhaskar, Yuan Yuan
(4/29) Hyemin Jeon, Chia-Chun Tsai, Alok Deshpande
(5/4) Matt Malloy, Vishnu Katreddy, Nikhil Rao
(5/6) Bo Li, Zhiting Xu, Shih-Hsuan Hsu