CS 726: Nonlinear Optimization I - Fall 2013


Lecture: 11:00-12:15, Psychology 103
Class Mailing List: compsci726-1-f13@lists.wisc.edu (here's the archive)
Course URL: http://www.cs.wisc.edu/~swright/cs726-f13.html
Piazza Home Page: https://piazza.com/wisc/fall2013/cs726/home
Learn@UW Course Page: https://uwmad.courses.wisconsin.edu/d2l/home/2169218

In general, classes will be held on MWF every week, and most lectures will be 60 minutes, but may take up the full 75-minutes slot on a few occasions. (I will be absent on a number of class days, and the longer lectures will make up for these absences.)

The lecture schedule is posted below - it is subject to change but each week's schedule will be finalized by the preceding week.

  • Week 1: Wed 9/4 (60 min), Fri 9/6 (60 min)
  • Week 2: Mon 9/9 (60 min), Wed 9/11 (60 min), Fri 9/13 (60 min)
  • Week 3: Mon 9/16 (no class), Wed 9/18 (no class), Fri 9/20 (75 min)
  • Week 4: Mon 9/23 (60 min), Wed 9/25 (65 min), Fri 9/27 (60 min)
  • Week 5: Mon 9/30 (60 min), Wed 10/2 (65 min), Fri 10/4 (60 min)
  • Week 6: Mon 10/7 (60 min), Wed 10/9 (60 min), Fri 10/11 (60 min)
  • Week 7: Mon 10/14 (60 min), Wed 10/16 (60 min), Fri 10/18 (60 min)
  • Week 8: Mon 10/21 (60 min), Wed 10/23 (60 min), Fri 10/25 (60 min)
  • Week 9: Mon 10/28 (60 min), Wed 10/30 (60 min), Fri 11/1 (no class)
  • Week 10: Mon 11/4 (60 min), Wed 11/6 (60 min), Fri 11/8 (60 min)
  • Week 11: Mon 11/11 (60 min), Wed 11/13 (60 min), Fri 11/15 (60 min)
  • Week 12: Mon 11/18 (60 min), Wed 11/20 (60 min), Fri 11/22 (60 min)
  • Week 13: Mon 11/25 (60 min), Wed 11/27 (60 min)
  • Week 14: Mon 12/2 (60 min), Wed 12/4 (60 min), Fri 12/6 (60 min)
  • Week 15: Mon 12/9 (60 min), Wed 12/11 (60 min?), Fri 12/13 (no class).


Instructor: Steve Wright

Office: 4379 CS
Office Hours: Monday 4-5, Thursday 4-5

General Course Information


  • Linear Algebra, some Analysis. See guidebook for specifics.
  • We'll use Matlab for some homeworks. You are responsible for teaching yourself, if you don't already know it. (We will not use highly advanced features of the language.) We may also use cvx (a Matlab add-on that allows modeling and solution of some convex optimization problems) for some homeworks.


  • J. Nocedal and S. J. Wright, Numerical Optimization, Second Edition, Springer, 2006. (It's essential to get the second edition! The version published in China in 2006 is a reprint of the first edition, so is not useful.) Here is the current list of typos.


  • D. P. Bertsekas, with A. Nedic and A. Ozdaglar, Convex Analysis and Optimization, Athena Scientific, Belmont, MA, 2003.
  • Nesterov, Y., Introductory Lectures on Convex Optimization, Kluwer, 2004.
  • S. Boyd and L. Vandenberghe, Convex Optimization, Cambridge University Press. Available here.
  • D. P. Bertsekas, Nonlinear Programming, Second Edition, Athena Scientific, Belmont, MA, 1999.
  • R. Fletcher, Practical Methods of Optimization, 2nd Edition, Wiley, Chichester & New York, 1987.
  • R. T. Rockafellar and R. J.-B. Wets, Variational Analysis, Springer, 1998. (This is a more advanced book and an invaluable reference.)
  • A. Ruszczynski, Nonlinear Optimization, Princeton University Press, 2006.
  • S. J. Wright, Primal-Dual Interior-Point Methods, SIAM, 1997.

Lecture Notes

These can be found on the resources page of the piazza site and are updated sporadically. They are mostly for my own benefit and hence are terse, and refer heavily to the text in places. I supply them to help you fill in any missing details in the notes that you take in class.


Course Outline

This will likely be adadpted as the semester proceeds, but most of the following topics will be covered.

  • Introduction
    • Continuous optimization paradigms
    • Representative applications
    • Mathematical background, including basic linear algebra, Taylor's theorem, convex sets and functions
  • Unconstrained optimization: Theory and algorithms
    • Optimality conditions
    • Line searches, model functions, and trust regions
    • Steepest descent, including short-step methods for convex functions.
    • Accelerated first-order methods for convex functions.
    • Newton’s method
    • Quasi-Newton methods
  • Large-scale unconstrained optimization:
    • Conjugate gradient methods (linear and nonlinear)
    • Limited-memory quasi-Newton methods
    • Approximate Newton methods
  • Derivative-free optimization
    • Model-based methods
    • Pattern-search methods
  • Least-squares and nonlinear equations
    • Linear least squares: direct and iterative methods
    • Nonlinear least squares: Gauss-Newton, Levenberg-Marquardt
    • Newton’s method for nonlinear equations
    • Merit functions for nonlinear equations, and line searches
  • Optimization with linear constraints
    • First-order optimality conditions (KKT)
    • Gradient projection algorithms for bound constraints


(The scheme below is provisional and subject to change.)

Keep track of your grades through the learn@uw system. The course home page is here.

  • Approximately 10 homework assignments, 30% of grade. Some of these will be graded by class members, using a key supplied by me. Some may be graded by a grader, if one is assigned. Some will be graded by me. Some will not be graded.
    • The electronic handin system may be used for some homeworks - see here for details
    • Homework is due at the beginning of class on the designated date.
    • No homework or project is accepted in mailbox of instructor.
    • You may discuss homework with classmates. However, you may not share any code, carry out the assignment together, or copy solutions from another person. Discussion should be verbal only. The submitted version must be worked out, written, and submitted by you alone.
    • Submitting someone else's work as your own is academic misconduct. Such cheating and plagiarism will be dealt with in accordance with University procedures (see the Academic Misconduct page).
  • MIDTERM, approx 25% of grade. Monday, 28 October 2013, 7:00pm-9:00pm, Room 1240 CS.
  • FINAL, approx 45% of grade. Friday 20 December 2013, 12:25pm-2:25pm. Room 1221 CS.


Previous Exams

Note that the curriculum has been changed significantly since 2010, so some questions on the earlierexams are not relevant to the current version of the course.

Handouts and Examples