CS 726: Nonlinear Optimization I - Fall 2012
Lecture: MWF 2:30pm-3:45pm
Location: 1227 Engineering Hall
Mailing List: TBA
Instructor: Ben Recht
Office: 4387 CS
Office Hours: Mondays 1:30pm-2:30pm, Tuesdays 10am-11am
General Course Infromation
This course will explore theory and algorithms for nonlinear optimization
with a focus on unconstrained optimization. Topics will
- Optimization and its applications
- Elements of Matrix Analysis
- Convex sets and functions
- Unconstrained optimization: Theory and algorithms
- Optimality conditions
- Gradient methods and Newton’s method
- Large-scale unconstrained optimization:
- Momentum and extra-gradient methods
- Limited-memory quasi-Newton methods
- Stochastic and Incremenetal Gradient Methods
- Elementary Constrained Optimization
- Duality Theory
- Gradient Projection Methods
- Alternating Direction Method of Multipliers
- Applications in data analysis, signal processing, and machine learning.
- Numerical Optimization. J.
Nocedal and S. J. Wright, Springer Series in Operations Research,
Springer-Verlag, New York, 2006 (2nd edition).
- Convex Optimization. S. Boyd and L. Vandenberghe. Cambridge
University Press, Cambridge, 2003. Pdf copy available here.
Other Nonlinear Programming References:
- Efficient Methods in Convex Programming. A. Nemirovski.
Lecture Notes available here.
- Introductory Lectures on Convex Optimization: A Basic
Course Y. Nesterov. Kluwer Academic Publishers, Boston. 2004.
- Nonlinear Programming D. P. Bertsekas. Athena Scientific,
Belmont, Massachusetts. (2nd edition). 1999.
Recommended Linear Algebra References:
- Matrix Computations G. H. Golub and C. F. Van Loan. The Johns
Hopkins University Press, Baltimore. (3rd edition). 1996.
- Matrix Analysis R. A. Horn and C. R. Johnson. Cambridge
University Press. Cambridge, UK. 1990.
Keep track of your grades through the
- Homework: Approximately 10 homework assignments, 70% of grade.
- The electronic learn@uw system will be used for some homeworks.
- Homework is due at the beginning of class on the designated date.
- No homework is accepted in mailbox of instructor.
- You may discuss homework with classmates, but the submitted version must be worked out, written, and submitted alone.
- Submitting someone else's work as your own is academic misconduct. Such cheating and plagiarism will be dealt
with in accordance with University procedures (see the Academic
- Final, 30% of grade. 24 hour take home. Date to be announced.
Generally, we will meet two days a week. On some weeks we may have
three lectures. I will be absent on a number of class days, and the extra
lectures will make up for these absences.
The lecture schedule is posted below - it is subject to change but
each week's schedule will be finalized by the preceding week.
Lecture 1 (09/05): Introduction.
Lecture 2 (09/07): Math Review.
Lecture 3 (09/10): Convex Sets.
Lecture 4 (09/12): Convex Functions.
Lecture 5 (09/17): Convex Functions.
Lecture 6 (09/19): Descent Methods and Line Search
Lecture 7 (09/24): Gradient Descent Convergence
Lecture 8 (09/26): Strongly Convex Functions, Newton's Method
Lecture 9 (10/01): Hardness of non-convex optimiztaion
Lecture 10 (10/03): Multi-step methods, linear conjugate gradient
Lecture 11 (10/05): Nonlinear conjugate gradient
Lecture 12 (10/08): Quasi-Newton Methods, BFGS
Lecture 13 (10/10): L-BFGS and Barzilai-Borwein
Lecture 14 (10/15): Constrained Optimization
Lecture 15 (10/17): Duality
Lecture 16 (10/22): Examples and applications of duality
Lecture 17 (10/24): Strong Duality and Slater's Condition
Lecture 18 (10/29): Optimality Conditions
Lecture 19 (10/31): Perturbation Theory, Theorems of the Alternative
Lecture 20 (11/05): Subgradients
Lecture 21 (11/07): Dual ascent and the subgradient method
Lecture 22 (11/09): The proximal point method
Problem Set 1. Due 09/19.
Problem Set 2. Due 09/26.
Problem Set 3. Due 10/03.
Problem Set 4. Due 10/10.
Problem Set 5. Due 10/24.
Problem Set 6. Due 10/31.
Problem Set 7. Due 11/7
Problem Set 8. Due 11/19
Problem Set 9. Due 12/14
Handouts and Examples
- Michael Ferris' math
Signular Values, Positive Definitines, Strong Convexity, Lipshitz
- Steve Wright's math
review. Rank, bases, linear
systems, basic topology.
- Proof of the hardness of
checking local minimality.
- Notes on the Heavy Ball Method.
- Some papers/expositions on the accelerated method by
Paul Tseng and
Dimitri Bertsekas. See the referen
ces for further sources of intuition.
- James Burke's notes on bisection line search and the Weak Wolfe Conditions.
- The original paper on the Barzalai-Borwein method.
- Stephen Boyd's slides
on subgradients and their properties
- Stephen Boyd's
on subgradients methods
- Notes on the
proximal point and projected gradient methods. For more details, see Nesterov's
paper on Projected Gradient Methods. Read this for details on line
search and how to adapt to an unknown strong convexity parameter.
Monograph on the Alternating Direction Method of Multipliers by Boyd et al.
Use the CS Unix Labs on the first floor of CS: Locations
For new users of Unix and the CS Unix facilities, orientation sessions will be held in
CS&S 1325 early in the semester. Schedules will be posted in the lobby of the CS
Here are some instructions for setting
up your Matlab environment on the linux machines, if you have not done this before.