There are many different algorithms you can use for optimization, but how do you know whether the results you get are any good?
One approach to solving this problem is to establish a baseline in performance using a naive optimization algorithm.
A naive optimization algorithm is an algorithm that assumes nothing about the objective function that is being optimized.
It can be applied with very little effort and the best result achieved by the algorithm can be used as a point of reference to compare more sophisticated algorithms. If a more sophisticated algorithm cannot achieve a better result than a naive algorithm on average, then it does not have skill on your problem and should be abandoned.
There are two naive algorithms that can be used for function optimization; they are:
- Random Search
- Grid Search
These algorithms are referred to as “search” algorithms because, at base, optimization can be framed as a search problem. E.g. find the inputs that minimize or maximize the output of the objective function.
There is another algorithm that can be used called “exhaustive search” that enumerates all possible inputs. This is rarely used in practice as enumerating all possible inputs is not feasible, e.g. would require too much time to run.
Nevertheless, if you find yourself working on an optimization problem for which all inputs can be enumerated and evaluated in reasonable time, this should be the default strategy you should use.