CS 726: Nonlinear Optimization I - Fall 2016

Schedule

In general, classes will be held on MWF every week, and most lectures will be 60 minutes, but may take up the full 75-minutes slot on a few occasions. (I will be absent on a number of class days, and the longer lectures will make up for these absences.)

The lecture schedule is posted below - it is subject to change but each week's schedule will be finalized by the preceding week.

• Week 1: Wed 9/7 (60 min), Fri 9/9 (60 min)
• Week 2: Mon 9/12 (60 min), Wed 9/14 (60 min), Fri 9/16 (60 min)
• Week 3: Mon 9/19 (60 min), Wed 9/21 (60 min), Fri 9/23 (60 min)
• Week 4: Mon 9/26 (60 min), Wed 9/28 (60 min), Fri 9/30 (60 min)
• Week 5: Mon 10/3 (NO CLASS), Wed 10/5 (60 min), Fri 10/7 (60 min)
• Week 6: Mon 10/10 (NO CLASS), Wed 10/12 (60 min), Fri 10/14 (60 min)
• Week 7: Mon 10/17 (60 min), Wed 10/19 (60 min), Fri 10/21 (60 min)
• Week 8: Mon 10/24 (60 min), Wed 10/26 (60 min), Fri 10/28 (NO CLASS)
• Week 9: Mon 10/31 (60 min), Wed 11/2 (60 min), Fri 11/4 (60 min)
• Week 10: Mon 11/7 (60 min), Wed 11/9 (60 min), Fri 11/11 (60 min)
• Week 11: Mon 11/14 (NO CLASS), Wed 11/16 (60 min), Fri 11/18 (60 min)
• Week 12: Mon 11/21 (60 min), Wed 11/23 (NO CLASS), Fri 11/25 (THANKSGIVING HOLIDAY)
• Week 13: Mon 11/28 (60 min), Wed 11/30 (60 min), Fri 12/2 (60 min)
• Week 14: Mon 12/5 (60 min), Wed 12/7 (60 min), Fri 12/9 (NO CLASS)
• Week 15: Mon 12/12 (NO CLASS), Wed 12/14 (60 min)

Instructor: Steve Wright

Teaching Assistant: Lin Liu

General Course Information

Prerequisite

• Linear Algebra, some Analysis. See guidebook for specifics.
• We'll use Matlab for some homeworks. You are responsible for teaching yourself, if you don't already know it. (We will not use highly advanced features of the language.) We may also use cvx (a Matlab add-on that allows modeling and solution of some convex optimization problems) for some homeworks.

Text

• J. Nocedal and S. J. Wright, Numerical Optimization, Second Edition, Springer, 2006. (It's essential to get the second edition!) Here is the current list of typos.

References

• D. P. Bertsekas, with A. Nedic and A. Ozdaglar, Convex Analysis and Optimization, Athena Scientific, Belmont, MA, 2003.
• Nesterov, Y., Introductory Lectures on Convex Optimization, Kluwer, 2004.
• S. Boyd and L. Vandenberghe, Convex Optimization, Cambridge University Press. Available here.
• D. P. Bertsekas, Nonlinear Programming, Second Edition, Athena Scientific, Belmont, MA, 1999.
• R. Fletcher, Practical Methods of Optimization, 2nd Edition, Wiley, Chichester & New York, 1987.
• R. T. Rockafellar and R. J.-B. Wets, Variational Analysis, Springer, 1998. (This is a more advanced book and an invaluable reference.)
• A. Ruszczynski, Nonlinear Optimization, Princeton University Press, 2006.
• S. J. Wright, Primal-Dual Interior-Point Methods, SIAM, 1997.

Lecture Notes

These can be found on the resources page of the piazza site and are updated sporadically. They are mostly for my own benefit and hence are terse, and refer heavily to the text in places. I supply them to help you fill in any missing details in the notes that you take in class.

Course Outline

This will likely be adadpted as the semester proceeds, but most of the following topics will be covered.

• Introduction
• Optimization paradigms and applications
• Mathematical background: convex sets and functions, linear algebra, topology, convergence rates
• Smooth unconstrained optimization: Background
• Taylor's theorem
• Optimality conditions
• First-Order Methods
• Steepest descent. Convergence for convex and nonconvex cases.
• Accelerated gradient. Convergence for convex case.
• Line search methods based on descent directions
• Conjugate gradient methods
• Conditional gradient for optimization over closed convex sets
• Higher-order methods
• Newton's method
• Line-search Newton
• Trust-region Newton and cubic regularization
• Conjugate gradient-Newton
• Quasi-Newton methods
• Limited-memory quasi-Newton
• Stochastic optimization
• Basic methods and their convergence properties
• reduced-variance approaches
• Differentiation
• Adjoint calculations
• Automatic differentiation
• Derivative-free optimization
• Model-based methods
• Pattern-search methods
• Least-squares and nonlinear equations
• Linear least squares: direct and iterative methods
• Nonlinear least squares: Gauss-Newton, Levenberg-Marquardt
• Newton’s method for nonlinear equations
• Merit functions for nonlinear equations, and line searches
• Optimization with linear constraints
• Normal cones to convex sets
• Farkas Lemma and first-order optimality conditions (KKT)
• Gradient projection algorithms

Assessment

(The scheme below is provisional and subject to change.)

Keep track of your grades through the learn@uw system. The course home page is here.

• 7-8 homework assignments, 30% of grade. Some of these will be graded by class members, using a key supplied by me. Some may be graded by a grader, if one is assigned. Some will be graded by me. Some will not be graded.
• Homework is due at the beginning of class on the designated date.
• No homework or project is accepted in mailbox of instructor.
• You may discuss homework with classmates. However, you may not share any code, carry out the assignment together, or copy solutions from another person. Discussion should be verbal only. The submitted version must be worked out, written, and submitted by you alone.
• Submitting someone else's work as your own is academic misconduct. Such cheating and plagiarism will be dealt with in accordance with University procedures (see the Academic Misconduct page).
• MIDTERM, approx 25% of grade. 10/27/16, 7:15pm-9:15pm. Location 1240 Computer Sciences.
• FINAL, approx 45% of grade. 21 December 2016, 12:25pm-2:25pm. Room: Psychology 107.

Homeworks

Previous Exams

• 2006: Final (Solutions).
• 2007: Final (Solutions).
• 2008: Final
• 2013: Midterm (Solutions). Final (Solutions).
• 2016: Midterm

Miscellaneous

• Archive of class mailing list
• Computer Systems Laboratory documentation.
• Instructional Facilities.
• Instructional Account Policies
• Account Information
• Optimization at UW-Madison
• Introduction to MATLAB