Optimization involves finding the inputs to an objective function that result in the minimum or maximum output of the function.
The open-source Python library for scientific computing called SciPy provides a suite of optimization algorithms. Many of the algorithms are used as a building block in other algorithms, most notably machine learning algorithms in the scikit-learn library.
These optimization algorithms can be used directly in a standalone manner to optimize a function. Most notably, algorithms for local search and algorithms for global search, the two main types of optimization you may encounter on a machine learning project.
The Python SciPy open-source library for scientific computing provides a suite of optimization techniques.
Many of the algorithms are used as building blocks for other algorithms within the SciPy library, as well as machine learning libraries such as scikit-learn.
Before we review specific techniques, let’s look at the types of algorithms provided by the library.
They are:
- Scalar Optimization: Optimization of a convex single variable function.
- Local Search: Optimization of a unimodal multiple variable function.
- Global Search: Optimization of a multimodal multiple variable function.
- Least Squares: Solve linear and non-linear least squares problems.
- Curve Fitting: Fit a curve to a data sample.
- Root Finding: Find the root (input that gives an output of zero) of a function.
- Linear Programming: Linear optimization subject to constraints.
All algorithms assume the objective function that is being optimized is a minimization function. If your function is maximizing, it can be converted to minimizing by adding a negative sign to values returned from your objective function.
In addition to the above list, the library also provides utility functions used by some of the algorithms, as well as the Rosenbrock test problem.