In general, classes will be held on MWF every week, and most lectures will be 60 minutes, but may take 70 minutes on a few occasions. (I will be absent on a number of class days, and the longer lectures will make up for these absences.)
General Course Information
- Linear Algebra, some Analysis. See guidebook for specifics.
The course will involve some programming to test algorithms. One useful option is to use Matlab with the free add-on cvx. A second option (particularly appealing if you took 524 recently) is to use Julia with the JuMP optimization toolbox. A third option is to use Python, but I can provide less support for this.
- J. Nocedal and S. J. Wright, Numerical Optimization, Second Edition, Springer, 2006. (It's essential to get the second edition!) Here is the current list of typos.
- D. P. Bertsekas, with A. Nedic and A. Ozdaglar, Convex Analysis and Optimization, Athena Scientific, Belmont, MA, 2003.
- Nesterov, Y., Introductory Lectures on Convex Optimization, Kluwer, 2004.
- S. Boyd and L. Vandenberghe, Convex Optimization, Cambridge University Press. Available here.
- D. P. Bertsekas, Nonlinear Programming, Second Edition, Athena Scientific, Belmont, MA, 1999.
- R. Fletcher, Practical Methods of Optimization, 2nd Edition, Wiley, Chichester & New York, 1987.
- R. T. Rockafellar and R. J.-B. Wets, Variational Analysis, Springer, 1998. (This is a more advanced book and an invaluable reference.)
- A. Ruszczynski, Nonlinear Optimization, Princeton University Press, 2006.
- S. J. Wright, Primal-Dual Interior-Point Methods, SIAM, 1997.
This will likely be adadpted as the semester proceeds, but most of the following topics will be covered.
- Optimization paradigms and applications
- Mathematical background: convex sets and functions, linear algebra, topology, convergence rates
- Smooth unconstrained optimization: Background
- Taylor's theorem
- Optimality conditions
- First-Order Methods
- Steepest descent. Convergence for convex and nonconvex cases.
- Accelerated gradient. Convergence for convex case.
- Line search methods based on descent directions
- Conjugate gradient methods
- Conditional gradient for optimization over closed convex sets
- Higher-order methods
- Newton's method
- Line-search Newton
- Trust-region Newton and cubic regularization
- Conjugate gradient-Newton
- Quasi-Newton methods
- Limited-memory quasi-Newton
- Stochastic optimization
- Basic methods and their convergence properties
- reduced-variance approaches
- Adjoint calculations
- Automatic differentiation
- Least-squares and nonlinear equations
- Linear least squares: direct and iterative methods
- Nonlinear least squares: Gauss-Newton, Levenberg-Marquardt
- Newton’s method for nonlinear equations
- Merit functions for nonlinear equations, and line searches
- Optimization with linear constraints
- Normal cones to convex sets
- Farkas Lemma and first-order optimality conditions (KKT)
- Gradient projection algorithms