CS838 Topics In Optimization

**Instructor:**Ben Recht

**Time: ** Tue & Thu, 1:00-2:15 PM

**Location: **1221 Computer Science

**Description:** This course will address the design of provably efficient algorithms for data processing that leverage prior information. We will focus on the specific areas of compressed sensing, stochastic algorithms for matrix factorization, rank minimization, and non-parametric machine learning. We will emphasize the pivotal roles of convexity and randomness in problem formulation, estimation guarantees, and algorithm design. The course will provide a unified exposition of these growing research areas and is ideal for advanced graduate students who would like to apply these theoretical and algorithmic developments to their own research.

The course will roughly be broken into the following structure:

**Foundations:**(2 weeks) Embeddings, Encodings, Random Projections, Coverings**Sparsity**: (3 weeks) Classical interpolation methods and Prony's method. Compressed Sensing and L1 Minimization. Restricted Isometries.**Rank: (**3 weeks) Krylov Subspaces and Lanczos methods. Stochastic algorithms for factorization. Matrix Completion.**Smoothness**: (3 weeks) Reproducing Kernel Hilbert Spaces, Elements of Approximation Theory, Atomic Decompositions, Random Features.**Project Presentations**(3 weeks)

**Grading: **Each student will be required to attend class regularly and scribe lecture notes for at least one class. A final project will also be required. This project will require a class presentation and a written report. The project can survey literature on a related topic not covered in the course or an application of the course techniques to a novel research problem.

**Prerequisites**: Graduate level courses in probability (like ECE 730) and nonlinear optimization (like CS 726). An advanced level of mathematical maturity is necessary. Familiarity with elementary functional analysis (L2 spaces, Fourier transforms, etc.) will be helpful for the last part of the course. Please consult the instructor if you are unsure about your background.

Lecture notes template

**Lecture 1** (01/19): Introduction. Slides

**Lecture 2** (01/21): Introduction to Random Mappings.
**Related Readings:**
Proof of Whitney's Embedding Theorem pdf.

**Lecture 3** (01/26): Random Projections Preserve
Distances. The Johnson-Lindenstrauss Lemma.

**Related
Readings:** Dasgupta and Gupta.
*An Elementary Proof of a Theorem of Johnson and Lindenstrauss.*
pdf

**Lecture 4** (01/28): Epislon Nets and Embedding
Subspaces.

**Related
Readings:** Rudelson and Vershynin. *The Smallest
Singular Value
of a Random
Rectangular Matrix*. **Only** the first 5 paragraphs
of
Section 2, Proposition 2.1, and its proof.
pdf.
Baraniuk *et al*.
*A Simple Proof of the Restricted Isometry Property for Random
Matrices.* pdf

**Lecture 5** (02/02): Sparsity and its applications.
**Notes:** pdf.

**Lecture 6** (02/04): Prony's method.
**Notes:** pdf.

A proof of the invertibility of the Vandermonde System. pdf

**Lecture 7** (02/09): l1 minimization, Restricted
Isometry Property.
**Notes:** pdf.

**Lecture 8** (02/11): l1 minimization, Robust
Recovery of Sparse Signals

**Lecture 9** (02/16): Matrices
with the Restricted
Isometry Property.

**Lecture 10** (02/18): Algorithms for l1 minimization

**Related
Readings:**
Wolfe. *The Simplex Method for Quadratic Programming*. pdf.
Donoho and Tsaig. *Fast Solution of l1-norm Minimization Problems When
the Solution May be Sparse*. pdf.
Tropp and Wright. *Computational Methods for Sparse Solution of
Linear Inverse Problems.* pdf.

**Lecture 11** (02/23): Matrix Norms and Rank

**Lecture 12** (02/25): Rank Minimization in
Data Analysis, Hardness Results.

**Lecture 13** (03/02): Easily solvable rank minimization
problems. Pass efficient approximations.

**Lecture 14** (03/04): The nuclear norm heuristic.

**Lecture 15** (03/09): RIP for low-rank matrices. The
nuclear norm succeeds.

**Lecture 16** (03/11): Gaussians obey the RIP for
low-rank matrices.

**Lecture 17** (03/16): Algorithms for Rank Minimization

**Lecture 18** (03/18): Iterative Shrinkage Thresholding
for Rank Minimization

**Lecture 19** (03/23): Moving beyond Restricted Isometry
Properties. Dual Certificates.

**Lecture 20** (04/06): Function Fitting. The Bias
Variance Tradeoff.

**Lecture 21** (04/08): Generalization Bounds

**Lecture 22** (04/13): Kernels

**Lecture 23** (04/15): Approximation Theory. The curse
of dimensionality. The blessing of smoothness.

**Upcoming Readings**

- Halko et al.
*Finding structure with randomness: Stochastic algorithms for constructing approximate matrix decompositions.*pdf - Recht et al.
*Guaranteed Minimum Rank Solutions to Linear Matrix Equations via Nuclear Norm Minimization.*pdf - Cucker and Smale
*The Mathematical Foundations of Learning.*pdf - Evgeniou et al.
*Regularization Networks and Support Vector Machines.*pdf - Barron
*Universal Approximation Bounds for Superpositions of a Sigmoidal Function.*pdf

**Project Presentations**

(4/20) Jesse Holzer, Benjamin Recht

(4/22) Jingjiang Peng, Yongia Song, Suhail Shergill

(4/27) Laura Balzano, Badri Bhaskar, Yuan Yuan

(4/29) Hyemin Jeon, Chia-Chun Tsai, Alok Deshpande

(5/4) Matt Malloy, Vishnu Katreddy, Nikhil Rao

(5/6) Bo Li, Zhiting Xu, Shih-Hsuan Hsu