Machine Learning and Optimization

Data Science, AI and ML
#1

Functional optimization is the problem of finding the set of inputs to a target objective function that result in the minimum or maximum of the function.

It can be a challenging problem as the function may have tens, hundreds, thousands, or even millions of inputs, and the structure of the function is unknown, and often non-differentiable and noisy.

  • Function Optimization: Find the set of inputs that results in the minimum or maximum of an objective function.

Machine learning can be described as function approximation. That is, approximating the unknown underlying function that maps examples of inputs to outputs in order to make predictions on new data.

It can be challenging as there is often a limited number of examples from which we can approximate the function, and the structure of the function that is being approximated is often nonlinear, noisy, and may even contain contradictions.

  • Function Approximation: Generalize from specific examples to a reusable mapping function for making predictions on new examples.

Function optimization is often simpler than function approximation.

Importantly, in machine learning, we often solve the problem of function approximation using function optimization.

At the core of nearly all machine learning algorithms is an optimization algorithm.

In addition, the process of working through a predictive modeling problem involves optimization at multiple steps in addition to learning a model, including:

  • Choosing the hyperparameters of a model.
  • Choosing the transforms to apply to the data prior to modeling
  • Choosing the modeling pipeline to use as the final model.