CS 726: Nonlinear Optimization I - Fall 2011 (Also ISyE, Stat, Math)

Schedule

  Lecture:     8:50 - 9:40 MWF, 1221 CS
               Start at 8:40 from Sep 7 to Dec 2
  Mailing list: compsci726-1-f11@lists.wisc.edu 
  Course URL: http://www.cs.wisc.edu/~cs726-1 

Instructor: Michael C. Ferris

  Office:       4381 CS
  Telephone:    262-4281
  E-mail:       ferris@cs.wisc.edu
                I will respond to the class mailing list, including
		your original message in most cases.
  Office Hours: 10:00 - 11:00 Mondays, 11:00 - 12:00 Wednesdays
 
  Class cancelled: September 14, 2011; October 17-19, 2011; October 31 - November 4, 2011
  Class meet in WID (Forum): September 30 at 8:40am
 

General Course Information (http://www.cs.wisc.edu/~ferris/cs726.html)

Course Outline

Theory and algorithms for nonlinear optimization, focusing on unconstrained optimization. Line-search and trust-region methods; quasi-Newton methods; conjugate-gradient and limited-memory methods for large-scale problems; derivative-free optimization; algorithms for least-squares problems and nonlinear equations; gradient projection algorithms for bound-constrained problems; and simple penalty methods for nonlinearly constrained optimization.

• Continuous optimization paradigms
• Representative applications
• Mathematical background, including convex sets and functions
• Unconstrained optimization: Theory and algorithms
  • Optimality conditions
  • Gradient methods and Newton's method
  • Line search methods
  • Quasi-Newton methods
• Derivative-Free optimization
  • Model-based methods
  • Pattern-search methods
• Large-scale unconstrained optimization:
  • Conjugate gradient methods (linear and nonlinear)
  • Limited-memory quasi-Newton methods
• Duality
• Nonlinear equations and least-squares problems
• Optimization with bound constraints:
  • Projections and gradient projection algorithms
  • Enhancements of gradient projection using second-order information and quasi-Newton techniques.
• Constrained nonlinear programming algorithms
  • Constraint elimination
  • Penalty methods

  Required Text

  • I will use a set of notes specially prepared for this course. They will cover the first part of the Nocedal and Wright book.
  • Numerical Optimization, J. Nocedal and S.J. Wright, Springer Series in Operations Research, Springer-Verlag, New York, 2006 (2nd edition).

  Other References:

  • Convex Optimization, S. Boyd and L. Vandenberghe, Cambridge University Press, UK 2004.
  • Nonlinear Optimization, Andrzej Ruszczynski, Princeton University Press, NJ 2006.
  • Nonlinear Programming, 2nd Edition, Dimitri Bertsekas, Athena Scientific, Belmont, MA 1999.
  • Practical Methods of Optimization, 2nd Edition, R. Fletcher, Wiley, 1987.
  • Practical Optimization, P. Gill, W. Murray and M. Wright, Academic Press, 1981.
  • Nonlinear Programming Theory and Algorithms , M. S. Bazaraa, H. D. Sherali and C. M. Shetty, Second Edition, Wiley, New York 2006.
  • The MATLAB PRIMER (Third Edition): An introduction to the basic commands and utilities that you may need in Matlab.

  Assignments and examinations

  • 1 Assignment per week approximately. Homework due at beginning of class one week after assigned unless otherwise noted.
  • Examinations are closed book, with the exception that 1 handwritten sheet (standard size paper) can be brought in to the examination.
  • No Midterm Examination
  • Final Examination - Friday, December 23 at 7:45 am - 9:45 am in 1221 CS. Previous exams (from 730) are given below, relevant questions are: 2003 (Q1), 2004 (Q2), 2005 (Q1)
  • Final Exam 2003
  • Final Exam 2004
  • Final Exam 2005
  • Final Exam 2009
  • Prereq: Familiarity with basic analysis (e.g. Math 521) and either Math 443 or 320, or consent of instructor

  Grading

  • Grades for the class will be available at Learn@UW. You will need to log-on, move to the course page, and use the "Grades" tab at the top of the page.
  • Approximately: 50% Homework, 50% Final
  • You may discuss the assignments with your classmates. However, you may not share any code, copy solution from another person, or carry out an assignment together. Discussion should only involve verbal communication. All assignments need to be written up entirely separately.

    Submitting someone else's work as your own is academic misconduct. Such cheating and plagiarism will be dealt with in accordance with University procedures (see the Academic Misconduct Guide for Students) .

  Handouts:

  • Eigenvalues, Singular Values, PSD, Strong Convexity, Lipschitz Continuity class handout 1.
  • anneal.m (Basic Simulated Annealing Code).
  • direct.m (DIRECT: divided rectangle code for optimization on a box).
  • Duality Theorem Proof class handout 2.
  • Penalty Function Theorem Proof class handout 3.

  Programming Assignments and Homeworks

  All assignments need to be written up entirely separately. You may discuss the problems informally with others in the class, but the discussions must not include code or explicit solutions to any of the problems.
  • Most homeworks will be handed in either in hard copy or using the drop box facility of Learn@UW. The drop box menu is found in the top menu bar once you have logged into the system.
  • Homework 1 (due September 21); hard copy ( Homework 1 solutions)
  • Homework 2 (due September 30); drop box and hard copy Files mentioned in assignment are obja.m
  • Homework 3 (due October 14) . Files mentioned in assignment are hwk3a.m, hwk3b.m, StepSize.m, obja.m, objb.m, objc.m, objd.m
  • Homework 4 (due October 28) . Files mentioned in assignment are hwk4.m, objg.m objampl.m, rosenbr.nl, brownal.nl, watson.nl, geodesic.nl, cragglvy.nl, woods.nl, allhwk4.zip Note that the zip archive contains all the files above and their corresponding "mod" files. cvsrch.m, a more sophisticated version of the StepSize routine. hwk4-sol.pdf
  • Homework 5 (due November 11) . Files mentioned in assignment are: hwk5.m, LRGV_Cost.m, LRGV_driver.m, LRGV_datafile.m, anneal.m
  • Homework 6 (due November 28) . Files mentioned in assignment are: hwk6.m, objnls.m, resida.m, residb.m, residc.m, residd.m, penalty1.mod, penalty1.nl, penalty2.mod, penalty2.nl hwk6-sol.pdf
  • Homework 7 (due December 7) Files needed in assignment are: hwk7.m, objqp.m, hwk9.mat. hwk7-sol.pdf
  • Homework 8 (due December 14, at class time) Files mentioned in assignment are: hwk8.mat. hwk8.m. hwk8-sol.pdf

  ---

  CS Department Computing Information

  • Instructional Facilities
  • Hours
  • Locations
  • CS 1000 contains unix startup information and is available online.

  Miscellaneous

  • Archive of Class Mailing List. You will need to enter your cs account login and password to access this archive.
  • Computer Systems Laboratory FAQ
  • Instructional Facilities
  • Account Activation
  • Instructional Account Policies
  • Forwarding Email
  • Optimization at the University of Wisconsin
  • Courses related to 726
  • Nonlinear Programming FAQ
  • NEOS Optimization Guide

  ---

  This page was updated September 1, 2011.

  ---