Books on Optimization for Machine Learning-2

Numerical Optimization

This book was written by Jorge Nocedal and Stephen Wright and was published in 2006.

Numerical Optimization

Numerical Optimization

This book is focused on the math and theory of the optimization algorithms presented and does cover many of the foundational techniques used by common machine learning algorithms. It may be a little too heavy for the average practitioner.

The book is intended as a textbook for graduate students in mathematical subjects.

We intend that this book will be used in graduate-level courses in optimization, as offered in engineering, operations research, computer science, and mathematics departments.

— Page xviii, Numerical Optimization, 2006.

Even though it is highly mathematical, the descriptions of the algorithms are precise and may provide a useful alternative description to complement the other books listed.

The complete table of contents for the book is listed below.

  • Chapter 01: Introduction
  • Chapter 02: Fundamentals of Unconstrained Optimization
  • Chapter 03: Line Search Methods
  • Chapter 04: Trust-Region Methods
  • Chapter 05: Conjugate Gradient Methods
  • Chapter 06: Quasi-Newton Methods
  • Chapter 07: Large-Scale Unconstrained Optimization
  • Chapter 08: Calculating Derivatives
  • Chapter 09: Derivative-Free Optimization
  • Chapter 10: Least-Squares Problems
  • Chapter 11: Nonlinear Equations
  • Chapter 12: Theory of Constrained Optimization
  • Chapter 13: Linear Programming: The Simplex Method
  • Chapter 14: Linear Programming: Interior-Point Methods
  • Chapter 15: Fundamentals of Algorithms for Nonlinear Constrained Optimization
  • Chapter 16: Quadratic Programming
  • Chapter 17: Penalty and Augmented Lagrangian Methods
  • Chapter 18: Sequential Quadratic Programming
  • Chapter 19: Interior-Point Methods for Nonlinear Programming

It’s a solid textbook on optimization.