Function Optimization

Data Science, AI and ML
#1

Function optimization is a subfield of mathematics, and in modern times is addressed using numerical computing methods.

Continuous function optimization (“function optimization” here for short) belongs to a broader field of study called mathematical optimization.

It is distinct from other types of optimization as it involves finding optimal candidate solutions composed of numeric input variables, as opposed to candidate solutions composed of sequences or combinations (e.g. combinatorial optimization).

Function optimization is a widely used tool bag of techniques employed in practically all scientific and engineering disciplines.

People optimize. Investors seek to create portfolios that avoid excessive risk while achieving a high rate of return. […] Optimization is an important tool in decision science and in the analysis of physical systems.

— Page 2, Numerical Optimization, 2006.

It plays a central role in machine learning, as almost all machine learning algorithms use function optimization to fit a model to a training dataset.

For example, fitting a line to a collection of points requires solving an optimization problem. As does fitting a linear regression or a neural network model on a training dataset.

In this way, optimization provides a tool to adapt a general model to a specific situation. Learning is treated as an optimization or search problem.

Practically, function optimization describes a class of problems for finding the input to a given function that results in the minimum or maximum output from the function.

The objective depends on certain characteristics of the system, called variables or unknowns. Our goal is to find values of the variables that optimize the objective.

— Page 2, Numerical Optimization, 2006.

Function Optimization involves three elements: the input to the function (e.g. x), the objective function itself (e.g. f()) and the output from the function (e.g. cost).

  • Input (x): The input to the function to be evaluated, e.g. a candidate solution.
  • Function (f()): The objective function or target function that evaluates inputs.
  • Cost: The result of evaluating a candidate solution with the objective function, minimized or maximized.