CS726: Nonlinear Optimization I

Fall 2008


Lecture: 11:00-12:15 MWF, 1207 CS&S
Class Mailing List: compsci726-1-f08@lists.wisc.edu (here's the archive)
Mail Instructor/TAs
Course URL: http://www.cs.wisc.edu/~swright/cs726.html


In general, lectures will be 60 minutes, but may take up the full 75-minutes slot on a few occasions. I will be absent on a number of class days, and the longer lectures will make up for these absences.

The lecture schedule is posted below - it is subject to change but each week's schedule will be finalized by the preceding week.

  • Week 1: Wed 9/3 (60 mins), Fri 9/5 (60 mins)
  • Week 2: Mon 9/8 (60 mins), Wed 9/10 (60 mins), Fri 9/12 (60 mins)
  • Week 3: Mon 9/15 (60 mins), Wed 9/17 (no class), Fri 9/19 (60 mins)
  • Week 4: Mon 9/22 (60 mins), Wed 9/24 (60 mins), Fri 9/26 (60 mins)
  • Week 5: Mon 9/29 (60 mins), Wed 10/1 (60 mins), Fri 10/3 (60 mins)
  • Week 6: Mon 10/6 (60 mins), Wed 10/8 (60 mins), Fri 10/10 (60 mins)
  • Week 7: Mon 10/13 (60 mins), Wed 10/15 (60 mins), Fri 10/17 (60 mins)
  • Week 8: Mon 10/20 (60 mins), Wed 10/22 (60 mins), Fri 10/24 (50 mins)
  • Week 9: Mon 10/27 (50 mins), Wed 10/29 (60 mins), Fri 10/31 (60 mins)
  • Week 10: Mon 11/3 (60 mins), Wed 11/5 (60 mins), Fri 11/7 (60 mins)
  • Week 11: Mon 11/10 (60 mins), Wed 11/12 (60 mins), Fri 11/14 (50 mins)
  • Week 12: Mon 11/17 (60 mins), Wed 11/19 (60 mins), Fri 11/21 (55 mins)
  • Week 13: Mon 11/24 (50 mins), Wed 11/26 (no class), Fri 11/28 (no class-Thanksgiving)
  • Week 14: Mon 12/1 (60 mins), Wed 12/3 (60 mins), Fri 12/5 (55 mins)
  • Week 15: Mon 12/8 (60 mins), Wed 12/10 (no class), Fri 12/12 (no class)


Instructor: Stephen Wright

Office: 4379 CS
Phone: 262-4838
Office Hours: Monday 10-11, Thursday 3-4

Teaching Assistant: Siddharth Barman

Office: 3367 CS
Office Hours: Wednesday 2-3, Thursday 4-5

General Course Information


  • J. Nocedal and S. J. Wright, Numerical Optimization, Second Edition, Springer, 2006. (It's essential to get the second edition! The version published in China in 2006 is a reprint of the first edition.) Here is the current list of typos.


  • D. P. Bertsekas, Nonlinear Programming, Second Edition, Athena Scientific, Belmont, MA, 1999.
  • R. Fletcher, Practical Methods of Optimization, 2nd Edition, Wiley, Chichester & New York, 1987.
  • A. Ruszczynski, Nonlinear Optimization, Princeton University Press, 2006.

Course Outline

We will follow the same curriculum that has been used for the past two years, except that I may add a two-week module in the latter part of the course on compressed sensing, a research topic of current interest.

  • Introduction:
    • Continuous optimization paradigms
    • Representative applications
    • Mathematical background, including convex sets and functions
  • Unconstrained optimization: Theory and algorithms
    • Optimality conditions
    • Gradient methods and Newton’s method
    • Line search methods
    • Trust region methods
    • Quasi-Newton methods
  • Derivative-free optimization
    • Model-based methods
    • Pattern-search methods
  • Large-scale unconstrained optimization:
    • Conjugate gradient methods (linear and nonlinear)
    • Limited-memory quasi-Newton methods
  • Least-squares problems
    • Linear least squares: direct (normal equations and QR) and iterative methods (conjugate gradient applied to normal equations)
    • Nonlinear least squares: Gauss-Newton, Levenberg-Marquardt
  • Nonlinear equations
    • Newton’s method
    • Merit functions and line searches
  • Optimization with bound constraints:
    • Projections and gradient projection algorithms
    • Enhancements of gradient projection using second-order information and quasi-Newton techniques.
  • Nonlinear programming algorithms
    • Constraint elimination
    • Penalty methods


Keep track of your grades through the learn@uw system.

  • Approximately 11 homework assignments, 50% of grade.
    • The electronic handin system will be used for some homeworks - see here for details
    • Homework is due at the beginning of class on the designated date.
    • No homeworks will be accepted by TAs, in mailbox or in person.
    • No homework or project is accepted in mailbox of instructor.
    • You may discuss homework with classmates, but the submitted version must be worked out, written, and submitted alone.
    • Submitting someone else's work as your own is academic misconduct. Such cheating and plagiarism will be dealt with in accordance with University procedures (see the Academic Misconduct Guide).
  • PROJECT, which consists of writing a report on a recently published research paper in optimization, 10% of grade. See further information below.
  • FINAL, 40% of grade. To be held on Monday, Dec 15, 2008, 12:25pm-2:25pm. Location: 1325 CS. You may bring into the exam one sheet of paper, handwritten on both sides.

Homework Assignments

  • Homework 1 (due at start of class, Friday 9/12/08)
  • Homework 2 (due at start of class, Friday, 9/19/08)
  • Homework 3 (due by electronic submission at 5:00pm Friday, 9/26/08; hard copy answer to first two questions to be submitted in class on 9/26/08). Here are the files you need for the programming part: hwk3.m, obja.m, objb.m, objc.m
  • Homework 4 (due electronically by 5:00pm Friday 10/3/08)
  • Homework 5 (due by electronic submission at 5:00pm Friday, 10/10/08; hard copy answer to the questions from the text to be submitted in class on 10/10/08). Here are the files you need for the programming part: hwk5.m, objc.m, objd.m
  • Homework 6 (Questions 3 and 4 due at start of class Monday 10/20/08, Matlab codes and comments handed in electronically by 5:00pm Monday 10/20/08). Additional files for this assignment: hwk6_cg.m, hwk6_nonlinearcg.m, Aexplicit.m, xpowsing.m
  • Homework 7 (due at start of class, Monday 10/27/08)
  • Homework 8 (Written questions due at start of class Monday 11/3/08, Matlab codes and comments handed in electronically by 5:00pm Monday 11/3/08). Additional files for this assignment: hwk8.m, nls_resida.m, nls_residb.m, tridia.m, xpowsing.m
  • Homework 9 (due at start of class, Friday 11/14/08)
  • Homework 10 (due at start of class, Friday 11/21/08)
  • Homework 11 (for practice only)

Class Project

The class project for 726 to read a recent research paper in optimization and write a report. The report should summarize the content of the paper - its novelty, major results, applications (if applicable), how it relates to material we covered in class.

It should be about four pages long in standard (typeset) format.

The paper should have appeared in the literature within the past 6 years. Look at the web sites of the major journals - SIAM Journal on Optimization, Mathematical Programming, Computational Optimization and Applications, Journal of Optimization Theory and Applications. You can also consider recent preprints on the Optimization Online web site.

You should choose a paper whose content is related to the material covered in class - either what we have done to date, or what we will cover by the end of semester, according to the curriculum posted on the web site.

You are also free to suggest a paper that is related to your own research interests, provided it has strong optimization content (and is not just an application of a known optimization approach).

Please send me with the paper you propose to use for the project by Monday November 17. I will either approve it or suggest something else.

Hand in your project at the final exam on Monday December 15, at 12:25pm. Please submit hard copies of your report AND the paper you reviewed.


Computing Information

Use the CS Unix Labs on the first floor of CS: Locations here.

For new users of Unix and the CS Unix facilities, orientation sessions will be held in CS&S 1325 early in the semester. Schedules will be posted in the lobby of the CS Building.

Here are some instructions for setting up your Matlab environment on the linux machines, if you have not done this before.


Handouts and Examples