Stochastic Optimization Algorithms

Data Science, AI and ML
#1

The use of randomness in the algorithms often means that the techniques are referred to as “heuristic search” as they use a rough rule-of-thumb procedure that may or may not work to find the optima instead of a precise procedure.

Many stochastic algorithms are inspired by a biological or natural process and may be referred to as “metaheuristics” as a higher-order procedure providing the conditions for a specific search of the objective function. They are also referred to as “black box” optimization algorithms.

Metaheuristics is a rather unfortunate term often used to describe a major subfield, indeed the primary subfield, of stochastic optimization.

— Page 7, Essentials of Metaheuristics, 2011.

There are many stochastic optimization algorithms.

Some examples of stochastic optimization algorithms include:

  • Iterated Local Search
  • Stochastic Hill Climbing
  • Stochastic Gradient Descent
  • Tabu Search
  • Greedy Randomized Adaptive Search Procedure

Some examples of stochastic optimization algorithms that are inspired by biological or physical processes include:

  • Simulated Annealing
  • Evolution Strategies
  • Genetic Algorithm
  • Differential Evolution
  • Particle Swarm Optimization