The Nelder-Mead optimization algorithm works well for a range of challenging nonlinear and non-differentiable objective functions.
Nevertheless, it can get stuck on multimodal optimization problems and noisy problems.
To make this concrete, let’s look at an example of each.
Noisy Optimization Problem
A noisy objective function is a function that gives different answers each time the same input is evaluated.
We can make an objective function artificially noisy by adding small Gaussian random numbers to the inputs prior to their evaluation.
For example, we can define a one-dimensional version of the x^2 function and use the randn() function to add small Gaussian random numbers to the input with a mean of 0.0 and a standard deviation of 0.3.
objective function
def objective(x):
return (x + randn(len(x))*0.3)**2.0
objective function
def objective(x):
return (x + randn(len(x))*0.3)**2.0
The noise will make the function challenging to optimize for the algorithm and it will very likely not locate the optima at x=0.0.
The complete example of using Nelder-Mead to optimize the noisy objective function is listed below.
nelder-mead optimization of noisy one-dimensional convex function
from scipy.optimize import minimize
from numpy.random import rand
from numpy.random import randn
objective function
def objective(x):
return (x + randn(len(x))*0.3)**2.0
define range for input
r_min, r_max = -5.0, 5.0
define the starting point as a random sample from the domain
pt = r_min + rand(1) * (r_max - r_min)
perform the search
result = minimize(objective, pt, method=‘nelder-mead’)
summarize the result
print(‘Status : %s’ % result[‘message’])
print(‘Total Evaluations: %d’ % result[‘nfev’])
evaluate solution
solution = result[‘x’]
evaluation = objective(solution)
print(‘Solution: f(%s) = %.5f’ % (solution, evaluation))
nelder-mead optimization of noisy one-dimensional convex function
from scipy.optimize import minimize
from numpy.random import rand
from numpy.random import randn
objective function
def objective(x):
return (x + randn(len(x))*0.3)**2.0
define range for input
r_min, r_max = -5.0, 5.0
define the starting point as a random sample from the domain
pt = r_min + rand(1) * (r_max - r_min)
perform the search
result = minimize(objective, pt, method=‘nelder-mead’)
summarize the result
print(‘Status : %s’ % result[‘message’])
print(‘Total Evaluations: %d’ % result[‘nfev’])
evaluate solution
solution = result[‘x’]
evaluation = objective(solution)
print(‘Solution: f(%s) = %.5f’ % (solution, evaluation))
Running the example executes the optimization, then reports the results.
Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.
In this case, the algorithm does not converge and instead uses the maximum number of function evaluations, which is 200.
Status: Maximum number of function evaluations has been exceeded.
Total Evaluations: 200
Solution: f([-0.6918238]) = 0.79431
The algorithm may converge on some runs of the code but will arrive on a point away from the optima.