CS 726: Nonlinear Optimization I - Fall 2010

Lecture: MWF 11:00AM - 12:15 PM
Location: 2239 Engineering Hall
Mailing List: TBA

Instructor: Ben Recht
Office: 4387 CS
Office Hours: Mondays 10-11, Thursdays 3-4

Teaching Assistant: Srikrishna Sridhar
Office: 5395 CS
Office Hours: Thursdays 1-2 PM

General Course Infromation

• Introduction:
• Optimization and its applications
• Elements of Matrix Analysis
• Convex sets and functions
• Unconstrained optimization: Theory and algorithms
• Optimality conditions
• Gradient methods and Newton’s method
• Line search methods
• Trust region methods
• Large-scale unconstrained optimization:
• Conjugate gradient methods (linear and nonlinear)
• Limited-memory quasi-Newton methods
• Least-squares problems
• Linear least squares: direct (normal equations and QR) and iterative methods (conjugate gradient applied to normal equations)
• Nonlinear least squares: Gauss-Newton, Levenberg-Marquardt
• Derivative-free optimization
• Subgradient Methods
• Elementary Constrained Optimization
• Gradient Projection Methods
• Stochastic and Incremenetal Gradient Methods
• Applications in data analysis, signal processing, and machine learning.

Required Text:

• Numerical Optimization. J. Nocedal and S. J. Wright, Springer Series in Operations Research, Springer-Verlag, New York, 2006 (2nd edition).

Other Nonlinear Programming References:

• Convex Optimization. S. Boyd and L. Vandenberghe. Cambridge University Press, Cambridge, 2003. Pdf copy available here.
• Efficient Methods in Convex Programming. A. Nemirovski. Lecture Notes available here.
• Introductory Lectures on Convex Optimization: A Basic Course Y. Nesterov. Kluwer Academic Publishers, Boston. 2004.
• Nonlinear Programming D. P. Bertsekas. Athena Scientific, Belmont, Massachusetts. (2nd edition). 1999.

Recommended Linear Algebra References:

• Matrix Computations G. H. Golub and C. F. Van Loan. The Johns Hopkins University Press, Baltimore. (3rd edition). 1996.
• Matrix Analysis R. A. Horn and C. R. Johnson. Cambridge University Press. Cambridge, UK. 1990.

Assessment

Keep track of your grades through the learn@uw system.

• Homework: Approximately 10 homework assignments, 70% of grade.
  • The electronic learn@uw system will be used for some homeworks.
  • Homework is due at the beginning of class on the designated date.
  • No homeworks will be accepted by TAs, in mailbox or in person.
  • No homework or project is accepted in mailbox of instructor.
  • You may discuss homework with classmates, but the submitted version must be worked out, written, and submitted alone.
  • Submitting someone else's work as your own is academic misconduct. Such cheating and plagiarism will be dealt with in accordance with University procedures (see the Academic Misconduct Guide).
• Final, 30% of grade. To be held on Thursday, Dec 23, 2010, 10:05pm - 12:05pm. Location: TBA
  • You may bring into the exam one sheet of paper, handwritten on both sides.

Lectures

Generally, we will meet two days a week. On some weeks we may have three lectures. I will be absent on a number of class days, and the extra lectures will make up for these absences.

The lecture schedule is posted below - it is subject to change but each week's schedule will be finalized by the preceding week.

Lecture 1 (09/03): Introduction.

Lecture 2 (09/08): Linear Algebra Review.

Lecture 3 (09/10): Positive Definite Matrices. Limits and Rates.

Lecture 4 (09/13): Convex Sets

Lecture 5 (09/15): Convex Functions

Lecture 6 (09/20): Optimality Conditions and Convex Quadratics

Lecture 7 (09/22): Gradient Descent

Lecture 8 (09/24): Strong Convexity

Lecture 9 (09/27): Newton's Method

Lecture 10 (10/4): Lyapunov Functions, Heavy Ball Method

Lecture 11 (10/6): Analysis of the Heavy Ball Method

Lecture 12 (10/8): Non-convex problems are hard

NO CLASS OCT 11-15

Lecture 13 (10/18): Conjugate Gradient Method

Lecture 14 (10/20): Nonlinear Conjugate Gradients and Nesterov's Accelerated Method

Lecture 15 (10/25): BFGS

Lecture 16 (10/27): LBFGS and Barzilai-Borwein

Lecture 17 (11/1): Derivative Free Optimization

Lecture 18 (11/3): Coordinate Descent, Expectation Maximization

Lecture 19 (11/10): Subgradients

Lecture 20 (11/12): Subgradient Method

Lecture 21 (11/15): Proximal Point Methods

Lecture 22 (11/17): Projected Gradient Method

Lecture 23 (11/22): Stochastic Gradient Methods

Lecture 24 (11/29): Least-Squares

Lecture 25 (12/10): Compressed Sensing

Lecture 26 (12/13): Loose Ends

Lecture 27 (12/15): Final Exam

Homework Assignments

Problem Set 1. Due 09/20.

Problem Set 2. Due 09/27.

Problem Set 3. Due 10/08. Matlab files: hwk3.m, obja.m, objb.m, objc.m, objd.m.

Problem Set 4. Due 10/27. Matlab files: hwk4_cg.m, hwk4_nonlinearcg.m, Aexplicit.m, xpowsing.m.

Problem Set 5. Due 11/10. Matlab files: hwk5.m, tridia.m. You may use the script WolfeLineSearch.m from the minFunc package for line search.

Problem Set 6. Due 11/22.

Problem Set 7. Due 12/13. Matlab files: hwk7.m, adult.mat.

Handouts and Examples

• Michael Ferris' math review. Eigenvalues, Signular Values, Positive Definitines, Strong Convexity, Lipshitz Continuity.
• Steve Wright's math review. Rank, bases, linear systems, basic topology.
• Notes on the Heavy Ball Method.
• Proof of the hardness of checking matrix copositivity.
• Some papers/expositions on the accelerated method by Paul Tseng and Dimitri Bertsekas. See the references for further sources of intuition.
• Tom Minka's explanation of Expectation Maximization.
• Stephen Boyd's slides and notes on subgradients and their properties
• Stephen Boyd's slides and notes on subgradients methods
• Notes on the proximal point and projected gradient methods. For more details, see Nesterov's paper on Projected Gradient Methods. Read this for details on line search and how to adapt to an unknown strong convexity parameter.
• Notes on the stochastic gradient method. For more details, see Nemirovski, Juditsky, Shapiro, and Lan's paper on Robust Stochastic Approximation.

Computing Information

Use the CS Unix Labs on the first floor of CS: Locations here.

For new users of Unix and the CS Unix facilities, orientation sessions will be held in CS&S 1325 early in the semester. Schedules will be posted in the lobby of the CS Building.

Here are some instructions for setting up your Matlab environment on the linux machines, if you have not done this before.