Books on Optimization for Machine Learning-1

swapneel-panda-419bc751 · 30 June 2021 15:34

Algorithms for Optimization

This book was written by Mykel Kochenderfer and Tim Wheeler and was published in 2019.

Algorithms for Optimization

This book might be one of the very few textbooks that I’ve seen that broadly covers the field of optimization techniques relevant to modern machine learning.

This book provides a broad introduction to optimization with a focus on practical algorithms for the design of engineering systems. We cover a wide variety of optimization topics, introducing the underlying mathematical problem formulations and the algorithms for solving them. Figures, examples, and exercises are provided to convey the intuition behind the various approaches.

— Page xiiix, Algorithms for Optimization, 2019.

Importantly the algorithms range from univariate methods (bisection, line search, etc.) to first-order methods (gradient descent), second-order methods (Newton’s method), direct methods (pattern search), stochastic methods (simulated annealing), and population methods (genetic algorithms, particle swarm), and so much more.

It includes both technical descriptions of algorithms with references and worked examples of algorithms in Julia. It’s a shame the examples are not in Python as this would make the book near perfect in my eyes.

The complete table of contents for the book is listed below.

Chapter 01: Introduction
Chapter 02: Derivatives and Gradients
Chapter 03: Bracketing
Chapter 04: Local Descent
Chapter 05: First-Order Methods
Chapter 06: Second-Order Methods
Chapter 07: Direct Methods
Chapter 08: Stochastic Methods
Chapter 09: Population Methods
Chapter 10: Constraints
Chapter 11: Linear Constrained Optimization
Chapter 12: Multiobjective Optimization
Chapter 13: Sampling Plans
Chapter 14: Surrogate Models
Chapter 15: Probabilistic Surrogate Models
Chapter 16: Surrogate Optimization
Chapter 17: Optimization under Uncertainty
Chapter 18: Uncertainty Propagation
Chapter 19: Discrete Optimization
Chapter 20: Expression Optimization
Chapter 21: Multidisciplinary Optimization