CS 726: Nonlinear Optimization I - Fall 2012

Lecture: MWF 2:30pm-3:45pm
Location: 1227 Engineering Hall
Mailing List: TBA

Instructor: Ben Recht
Office: 4387 CS
Office Hours: Mondays 1:30pm-2:30pm, Tuesdays 10am-11am

General Course Infromation

This course will explore theory and algorithms for nonlinear optimization with a focus on unconstrained optimization. Topics will include
  • Introduction:
    • Optimization and its applications
    • Elements of Matrix Analysis
    • Convex sets and functions
  • Unconstrained optimization: Theory and algorithms
    • Optimality conditions
    • Gradient methods and Newton’s method
  • Large-scale unconstrained optimization:
    • Momentum and extra-gradient methods
    • Limited-memory quasi-Newton methods
    • Stochastic and Incremenetal Gradient Methods
  • Elementary Constrained Optimization
    • Duality Theory
    • Gradient Projection Methods
    • Alternating Direction Method of Multipliers
  • Applications in data analysis, signal processing, and machine learning.

Required Texts:

  • Numerical Optimization. J. Nocedal and S. J. Wright, Springer Series in Operations Research, Springer-Verlag, New York, 2006 (2nd edition).
  • Convex Optimization. S. Boyd and L. Vandenberghe. Cambridge University Press, Cambridge, 2003. Pdf copy available here.

Other Nonlinear Programming References:

  • Efficient Methods in Convex Programming. A. Nemirovski. Lecture Notes available here.
  • Introductory Lectures on Convex Optimization: A Basic Course Y. Nesterov. Kluwer Academic Publishers, Boston. 2004.
  • Nonlinear Programming D. P. Bertsekas. Athena Scientific, Belmont, Massachusetts. (2nd edition). 1999.

Recommended Linear Algebra References:

  • Matrix Computations G. H. Golub and C. F. Van Loan. The Johns Hopkins University Press, Baltimore. (3rd edition). 1996.
  • Matrix Analysis R. A. Horn and C. R. Johnson. Cambridge University Press. Cambridge, UK. 1990.

Assessment

Keep track of your grades through the learn@uw system.

  • Homework: Approximately 10 homework assignments, 70% of grade.
    • The electronic learn@uw system will be used for some homeworks.
    • Homework is due at the beginning of class on the designated date.
    • No homework is accepted in mailbox of instructor.
    • You may discuss homework with classmates, but the submitted version must be worked out, written, and submitted alone.
    • Submitting someone else's work as your own is academic misconduct. Such cheating and plagiarism will be dealt with in accordance with University procedures (see the Academic Misconduct Guide).
  • Final, 30% of grade. 24 hour take home. Date to be announced.

Lectures

Generally, we will meet two days a week. On some weeks we may have three lectures. I will be absent on a number of class days, and the extra lectures will make up for these absences.

The lecture schedule is posted below - it is subject to change but each week's schedule will be finalized by the preceding week.

Lecture 1 (09/05): Introduction.

Lecture 2 (09/07): Math Review.

Lecture 3 (09/10): Convex Sets.

Lecture 4 (09/12): Convex Functions.

Lecture 5 (09/17): Convex Functions.

Lecture 6 (09/19): Descent Methods and Line Search

Lecture 7 (09/24): Gradient Descent Convergence

Lecture 8 (09/26): Strongly Convex Functions, Newton's Method

Lecture 9 (10/01): Hardness of non-convex optimiztaion

Lecture 10 (10/03): Multi-step methods, linear conjugate gradient

Lecture 11 (10/05): Nonlinear conjugate gradient

Lecture 12 (10/08): Quasi-Newton Methods, BFGS

Lecture 13 (10/10): L-BFGS and Barzilai-Borwein

Lecture 14 (10/15): Constrained Optimization

Lecture 15 (10/17): Duality

Lecture 16 (10/22): Examples and applications of duality

Lecture 17 (10/24): Strong Duality and Slater's Condition

Lecture 18 (10/29): Optimality Conditions

Lecture 19 (10/31): Perturbation Theory, Theorems of the Alternative

Lecture 20 (11/05): Subgradients

Lecture 21 (11/07): Dual ascent and the subgradient method

Lecture 22 (11/09): The proximal point method

Homework Assignments

Problem Set 1. Due 09/19.

Problem Set 2. Due 09/26.

Problem Set 3. Due 10/03.

Problem Set 4. Due 10/10.

Problem Set 5. Due 10/24.

Problem Set 6. Due 10/31.

Problem Set 7. Due 11/7

Problem Set 8. Due 11/19

Problem Set 9. Due 12/14

Handouts and Examples

  • Michael Ferris' math review. Eigenvalues, Signular Values, Positive Definitines, Strong Convexity, Lipshitz Continuity.
  • Steve Wright's math review. Rank, bases, linear systems, basic topology.
  • Proof of the hardness of checking local minimality.
  • Notes on the Heavy Ball Method.
  • Some papers/expositions on the accelerated method by Paul Tseng and Dimitri Bertsekas. See the referen ces for further sources of intuition.
  • James Burke's notes on bisection line search and the Weak Wolfe Conditions.
  • The original paper on the Barzalai-Borwein method.
  • Stephen Boyd's slides and notes on subgradients and their properties
  • Stephen Boyd's slides and notes on subgradients methods
  • Notes on the proximal point and projected gradient methods. For more details, see Nesterov's paper on Projected Gradient Methods. Read this for details on line search and how to adapt to an unknown strong convexity parameter.
  • Monograph on the Alternating Direction Method of Multipliers by Boyd et al.

Computing Information

Use the CS Unix Labs on the first floor of CS: Locations here.

For new users of Unix and the CS Unix facilities, orientation sessions will be held in CS&S 1325 early in the semester. Schedules will be posted in the lobby of the CS Building.

Here are some instructions for setting up your Matlab environment on the linux machines, if you have not done this before.