CS 726: Nonlinear Optimization I  Fall 2013

Schedule
In general, classes will be held on MWF every week, and most lectures will be 60 minutes, but may take up the full 75minutes slot on a few occasions. (I will be absent on a number of class days, and the longer lectures will make up for these absences.)
The lecture schedule is posted below  it is subject to change but each week's schedule will be finalized by the preceding week.
 Week 1: Wed 9/4 (60 min), Fri 9/6 (60 min)
 Week 2: Mon 9/9 (60 min), Wed 9/11 (60 min), Fri 9/13 (60 min)
 Week 3: Mon 9/16 (no class), Wed 9/18 (no class), Fri 9/20 (75 min)
 Week 4: Mon 9/23 (60 min), Wed 9/25 (65 min), Fri 9/27 (60 min)
 Week 5: Mon 9/30 (60 min), Wed 10/2 (65 min), Fri 10/4 (60 min)
 Week 6: Mon 10/7 (60 min), Wed 10/9 (60 min), Fri 10/11 (60 min)
 Week 7: Mon 10/14 (60 min), Wed 10/16 (60 min), Fri 10/18 (60 min)
 Week 8: Mon 10/21 (60 min), Wed 10/23 (60 min), Fri 10/25 (60 min)
 Week 9: Mon 10/28 (60 min), Wed 10/30 (60 min), Fri 11/1 (no class)
 Week 10: Mon 11/4 (60 min), Wed 11/6 (60 min), Fri 11/8 (60 min)
 Week 11: Mon 11/11 (60 min), Wed 11/13 (60 min), Fri 11/15 (60 min)
 Week 12: Mon 11/18 (60 min), Wed 11/20 (60 min), Fri 11/22 (60 min)
 Week 13: Mon 11/25 (60 min), Wed 11/27 (60 min)
 Week 14: Mon 12/2 (60 min), Wed 12/4 (60 min), Fri 12/6 (60 min)
 Week 15: Mon 12/9 (60 min), Wed 12/11 (60 min?), Fri 12/13 (no class).


Office: 
4379 CS 
Office Hours: 
Monday 45, Thursday 45 


General Course Information
Prerequisite
 Linear Algebra, some Analysis. See guidebook for specifics.
 We'll use Matlab for some homeworks. You are responsible for teaching yourself, if you don't already know it. (We will not use highly advanced features of the language.)
We may also use cvx (a Matlab addon that allows modeling and solution of some convex optimization problems) for some homeworks.
Text
 J. Nocedal and S. J. Wright, Numerical Optimization, Second Edition, Springer, 2006. (It's essential to get the second edition! The version published in China in 2006 is a reprint of the first edition, so is not useful.) Here is the current list of typos.
References
 D. P. Bertsekas, with A. Nedic and A. Ozdaglar, Convex Analysis and Optimization, Athena Scientific, Belmont, MA, 2003.
 Nesterov, Y., Introductory Lectures on Convex Optimization, Kluwer, 2004.
 S. Boyd and L. Vandenberghe, Convex Optimization, Cambridge University Press. Available here.
 D. P. Bertsekas, Nonlinear Programming, Second Edition, Athena Scientific, Belmont, MA, 1999.
 R. Fletcher, Practical Methods of Optimization, 2nd Edition, Wiley, Chichester & New York, 1987.
 R. T. Rockafellar and R. J.B. Wets, Variational Analysis, Springer, 1998. (This is a more advanced book and an invaluable reference.)
 A. Ruszczynski, Nonlinear Optimization, Princeton University Press, 2006.
 S. J. Wright, PrimalDual InteriorPoint Methods, SIAM, 1997.
Lecture Notes
These can be found on the resources page of the piazza site and are updated sporadically. They are mostly for my own benefit and hence are terse, and refer heavily to the text in places. I supply them to help you fill in any missing details in the notes that you take in class.


Course Outline
This will likely be adadpted as the semester proceeds, but most of the following topics will be covered.
 Introduction
 Continuous optimization paradigms
 Representative applications
 Mathematical background, including basic linear algebra, Taylor's theorem, convex sets and functions
 Unconstrained optimization: Theory and algorithms
 Optimality conditions
 Line searches, model functions, and trust regions
 Steepest descent, including shortstep methods for convex functions.
 Accelerated firstorder methods for convex functions.
 Newton’s method
 QuasiNewton methods
 Largescale unconstrained optimization:
 Conjugate gradient methods (linear and nonlinear)
 Limitedmemory quasiNewton methods
 Approximate Newton methods
 Derivativefree optimization
 Modelbased methods
 Patternsearch methods
 Leastsquares and nonlinear equations
 Linear least squares: direct and iterative methods
 Nonlinear least squares: GaussNewton, LevenbergMarquardt
 Newton’s method for nonlinear equations
 Merit functions for nonlinear equations, and line searches
 Optimization with linear constraints
 Firstorder optimality conditions (KKT)
 Gradient projection algorithms for bound constraints


Assessment
(The scheme below is provisional and subject to change.)
Keep track of your grades through the learn@uw system. The course home page is here.
 Approximately 10 homework assignments, 30% of grade. Some of these will be graded by class members, using a key supplied by me. Some may be graded by a grader, if one is assigned. Some will be graded by me. Some will not be graded.
 The electronic handin system may be used for some homeworks  see here for details
 Homework is due at the beginning of class on the designated date.
 No homework or project is accepted in mailbox of instructor.
 You may discuss homework with classmates. However, you may not share any code, carry out the assignment together, or copy solutions from another person. Discussion should be verbal only. The submitted version must be worked out, written, and submitted by you alone.
 Submitting someone else's work as your own is academic misconduct. Such cheating and plagiarism will be dealt with in accordance with University procedures (see the Academic Misconduct page).
 MIDTERM, approx 25% of grade. Monday, 28 October 2013, 7:00pm9:00pm, Room 1240 CS.
 FINAL, approx 45% of grade. Friday 20 December 2013, 12:25pm2:25pm. Room 1221 CS.


Homeworks


Previous Exams
Note that the curriculum has been changed significantly since 2010, so some questions on the earlierexams are not relevant to the current version of the course.


Handouts and Examples


Miscellaneous

