Can you list all direct search methods to estimate parameters? Direct search methods (also called derivative-free) are function minimization techniques that do not rely on a function's gradient to find the minimum. More importantly, they can be used to solve non-smooth optimization problems, which can be then used in parameter estimation (e.g., xxx). 
As nearly every statistician will recognize, the most commonly used method is the Nelder-Mead, 1965 (see here for a description). 
The optim function in R implements the Nelder-Mead method.
Here, I'd like to create a complete list (with strengths and defects) of other approaches and how to use them. Can you help?
 A: I'll recommend the survey paper 
Tamara G. Kolda, Robert Michael Lewis, and Virginia Torczon.  Optimization by direct search: new perspectives on some classical and modern methods
and this book:
Andrew R. Conn, Katya Scheinberg, and Luis N. Vicente.  Introduction to Derivative-Free Optimization
A: Hyperparameter search uses a very wide variety of optimization methods. In my opinion, Nelder-Mead is often a poor choice because hyperparameter response surfaces are usually not smooth with tons of local optima. In my experience, Nelder-Mead almost always gets stuck in poor local optima (in the context of optimizing hyperparameters for machine learning methods, so ymmv).
Common methods include:


*

*Nelder-Mead

*particle swarm optimization

*genetic algorithms

*harmonic search

*racing algorithms

*EGO

*Bayesian optimization

*...


Many of the metaheuristic methods mentioned above are offered in Optunity, while several other packages offer Bayesian optimization (e.g. Hyperopt, SMAC and BayesOpt).
