CS 726: Nonlinear Optimization I  Fall 2012
Lecture: MWF 2:30pm3:45pm
Location: 1227 Engineering Hall
Mailing List: TBA
Instructor: Ben Recht
Office: 4387 CS
Office Hours: Mondays 1:30pm2:30pm, Tuesdays 10am11am
General Course Infromation
This course will explore theory and algorithms for nonlinear optimization
with a focus on unconstrained optimization. Topics will
include
 Introduction:
 Optimization and its applications
 Elements of Matrix Analysis
 Convex sets and functions
 Unconstrained optimization: Theory and algorithms
 Optimality conditions
 Gradient methods and Newton’s method
 Largescale unconstrained optimization:
 Momentum and extragradient methods
 Limitedmemory quasiNewton methods
 Stochastic and Incremenetal Gradient Methods
 Elementary Constrained Optimization
 Duality Theory
 Gradient Projection Methods
 Alternating Direction Method of Multipliers
 Applications in data analysis, signal processing, and machine learning.
Required Texts:
 Numerical Optimization. J.
Nocedal and S. J. Wright, Springer Series in Operations Research,
SpringerVerlag, New York, 2006 (2nd edition).
 Convex Optimization. S. Boyd and L. Vandenberghe. Cambridge
University Press, Cambridge, 2003. Pdf copy available here.
Other Nonlinear Programming References:
 Efficient Methods in Convex Programming. A. Nemirovski.
Lecture Notes available here.
 Introductory Lectures on Convex Optimization: A Basic
Course Y. Nesterov. Kluwer Academic Publishers, Boston. 2004.
 Nonlinear Programming D. P. Bertsekas. Athena Scientific,
Belmont, Massachusetts. (2nd edition). 1999.
Recommended Linear Algebra References:
 Matrix Computations G. H. Golub and C. F. Van Loan. The Johns
Hopkins University Press, Baltimore. (3rd edition). 1996.
 Matrix Analysis R. A. Horn and C. R. Johnson. Cambridge
University Press. Cambridge, UK. 1990.
Assessment
Keep track of your grades through the
learn@uw system.
 Homework: Approximately 10 homework assignments, 70% of grade.
 The electronic learn@uw system will be used for some homeworks.
 Homework is due at the beginning of class on the designated date.
 No homework is accepted in mailbox of instructor.
 You may discuss homework with classmates, but the submitted version must be worked out, written, and submitted alone.
 Submitting someone else's work as your own is academic misconduct. Such cheating and plagiarism will be dealt
with in accordance with University procedures (see the Academic
Misconduct Guide).
 Final, 30% of grade. 24 hour take home. Date to be announced.
Lectures
Generally, we will meet two days a week. On some weeks we may have
three lectures. I will be absent on a number of class days, and the extra
lectures will make up for these absences.
The lecture schedule is posted below  it is subject to change but
each week's schedule will be finalized by the preceding week.
Lecture 1 (09/05): Introduction.
Lecture 2 (09/07): Math Review.
Lecture 3 (09/10): Convex Sets.
Lecture 4 (09/12): Convex Functions.
Lecture 5 (09/17): Convex Functions.
Lecture 6 (09/19): Descent Methods and Line Search
Lecture 7 (09/24): Gradient Descent Convergence
Lecture 8 (09/26): Strongly Convex Functions, Newton's Method
Lecture 9 (10/01): Hardness of nonconvex optimiztaion
Lecture 10 (10/03): Multistep methods, linear conjugate gradient
Lecture 11 (10/05): Nonlinear conjugate gradient
Lecture 12 (10/08): QuasiNewton Methods, BFGS
Lecture 13 (10/10): LBFGS and BarzilaiBorwein
Lecture 14 (10/15): Constrained Optimization
Lecture 15 (10/17): Duality
Lecture 16 (10/22): Examples and applications of duality
Lecture 17 (10/24): Strong Duality and Slater's Condition
Lecture 18 (10/29): Optimality Conditions
Lecture 19 (10/31): Perturbation Theory, Theorems of the Alternative
Lecture 20 (11/05): Subgradients
Lecture 21 (11/07): Dual ascent and the subgradient method
Lecture 22 (11/09): The proximal point method
Homework Assignments
Problem Set 1. Due 09/19.
Problem Set 2. Due 09/26.
Problem Set 3. Due 10/03.
Problem Set 4. Due 10/10.
Problem Set 5. Due 10/24.
Problem Set 6. Due 10/31.
Problem Set 7. Due 11/7
Problem Set 8. Due 11/19
Problem Set 9. Due 12/14
Handouts and Examples
 Michael Ferris' math
review. Eigenvalues,
Signular Values, Positive Definitines, Strong Convexity, Lipshitz
Continuity.
 Steve Wright's math
review. Rank, bases, linear
systems, basic topology.
 Proof of the hardness of
checking local minimality.
 Notes on the Heavy Ball Method.
 Some papers/expositions on the accelerated method by
Paul Tseng and
Dimitri Bertsekas. See the referen
ces for further sources of intuition.
 James Burke's notes on bisection line search and the Weak Wolfe Conditions.
 The original paper on the BarzalaiBorwein method.
 Stephen Boyd's slides
and notes
on subgradients and their properties
 Stephen Boyd's
slides
and notes
on subgradients methods
 Notes on the
proximal point and projected gradient methods. For more details, see Nesterov's
paper on Projected Gradient Methods. Read this for details on line
search and how to adapt to an unknown strong convexity parameter.

Monograph on the Alternating Direction Method of Multipliers by Boyd et al.
Computing Information
Use the CS Unix Labs on the first floor of CS: Locations
here.
For new users of Unix and the CS Unix facilities, orientation sessions will be held in
CS&S 1325 early in the semester. Schedules will be posted in the lobby of the CS
Building.
Here are some instructions for setting
up your Matlab environment on the linux machines, if you have not done this before.
