Convex Geometry in High-Dimensional Data Analysis
CS838 Topics In Optimization

Instructor:Ben Recht
Time:  Tue & Thu, 1:00-2:15 PM
Location: 1221 Computer Science

Description: This course will address the design of provably efficient algorithms for data processing that leverage prior information.  We will focus on the specific areas of compressed sensing, stochastic algorithms for matrix factorization, rank minimization, and non-parametric machine learning.  We will emphasize the pivotal roles of convexity and randomness in problem formulation, estimation guarantees, and algorithm design.  The course will provide a unified exposition of these growing research areas and is ideal for advanced graduate students who would like to apply these theoretical and algorithmic developments to their own research.

The course will roughly be broken into the following structure:

  1. Foundations: (2 weeks) Embeddings, Encodings, Random Projections, Coverings
  2. Sparsity: (3 weeks) Classical interpolation methods and Prony's method.  Compressed Sensing and L1 Minimization. Restricted Isometries.
  3. Rank: (3 weeks) Krylov Subspaces and Lanczos methods.  Stochastic algorithms for factorization.  Matrix Completion.
  4. Smoothness: (3 weeks) Reproducing Kernel Hilbert Spaces, Elements of Approximation Theory, Atomic Decompositions, Random Features.
  5. Project Presentations (3 weeks)

Grading: Each student will be required to attend class regularly and scribe lecture notes for at least one class.  A final project will also be required.  This project will require a class presentation and a written report.  The project can survey literature on a related topic not covered in the course or an application of the course techniques to a novel research problem.

Prerequisites: Graduate level courses in probability (like ECE 730) and nonlinear optimization (like CS 726). An advanced level of mathematical maturity is necessary. Familiarity with elementary functional analysis (L2 spaces, Fourier transforms, etc.) will be helpful for the last part of the course. Please consult the instructor if you are unsure about your background.

Lecture notes template

Lecture 1 (01/19): Introduction. Slides

Lecture 2 (01/21): Introduction to Random Mappings.
Related Readings: Proof of Whitney's Embedding Theorem pdf.

Lecture 3 (01/26): Random Projections Preserve Distances. The Johnson-Lindenstrauss Lemma.
Related Readings: Dasgupta and Gupta. An Elementary Proof of a Theorem of Johnson and Lindenstrauss. pdf

Lecture 4 (01/28): Epislon Nets and Embedding Subspaces.
Related Readings: Rudelson and Vershynin. The Smallest Singular Value of a Random Rectangular Matrix. Only the first 5 paragraphs of Section 2, Proposition 2.1, and its proof. pdf. Baraniuk et al. A Simple Proof of the Restricted Isometry Property for Random Matrices. pdf

Lecture 5 (02/02): Sparsity and its applications.
Notes: pdf.

Lecture 6 (02/04): Prony's method.
Notes: pdf.
A proof of the invertibility of the Vandermonde System. pdf

Lecture 7 (02/09): l1 minimization, Restricted Isometry Property.
Notes: pdf.

Lecture 8 (02/11): l1 minimization, Robust Recovery of Sparse Signals

Lecture 9 (02/16): Matrices with the Restricted Isometry Property.

Lecture 10 (02/18): Algorithms for l1 minimization
Related Readings: Wolfe. The Simplex Method for Quadratic Programming. pdf. Donoho and Tsaig. Fast Solution of l1-norm Minimization Problems When the Solution May be Sparse. pdf. Tropp and Wright. Computational Methods for Sparse Solution of Linear Inverse Problems. pdf.

Lecture 11 (02/23): Matrix Norms and Rank

Lecture 12 (02/25): Rank Minimization in Data Analysis, Hardness Results.

Lecture 13 (03/02): Easily solvable rank minimization problems. Pass efficient approximations.

Lecture 14 (03/04): The nuclear norm heuristic.

Lecture 15 (03/09): RIP for low-rank matrices. The nuclear norm succeeds.

Lecture 16 (03/11): Gaussians obey the RIP for low-rank matrices.

Lecture 17 (03/16): Algorithms for Rank Minimization

Lecture 18 (03/18): Iterative Shrinkage Thresholding for Rank Minimization

Lecture 19 (03/23): Moving beyond Restricted Isometry Properties. Dual Certificates.

Lecture 20 (04/06): Function Fitting. The Bias Variance Tradeoff.

Lecture 21 (04/08): Generalization Bounds

Lecture 22 (04/13): Kernels

Lecture 23 (04/15): Approximation Theory. The curse of dimensionality. The blessing of smoothness.

Upcoming Readings

Project Presentations

(4/20) Jesse Holzer, Benjamin Recht

(4/22) Jingjiang Peng, Yongia Song, Suhail Shergill

(4/27) Laura Balzano, Badri Bhaskar, Yuan Yuan

(4/29) Hyemin Jeon, Chia-Chun Tsai, Alok Deshpande

(5/4) Matt Malloy, Vishnu Katreddy, Nikhil Rao

(5/6) Bo Li, Zhiting Xu, Shih-Hsuan Hsu