Direct search methods (also called derivative-free) are function minimization techniques that do not rely on a function's gradient to find the minimum. More importantly, they can be used to solve non-smooth optimization problems, which can be then used in parameter estimation (e.g., xxx).

As nearly every statistician will recognize, the most commonly used method is the Nelder-Mead, 1965 (see here for a description). The optim function in R implements the Nelder-Mead method.

Here, I'd like to create a complete list (with strengths and defects) of other approaches and how to use them. Can you help?

  • 1
    $\begingroup$ Typically, shopping-list questions are frowned upon, so I wouldn't be surprised if this gets closed. $\endgroup$ Oct 4, 2015 at 14:59

2 Answers 2


Hyperparameter search uses a very wide variety of optimization methods. In my opinion, Nelder-Mead is often a poor choice because hyperparameter response surfaces are usually not smooth with tons of local optima. In my experience, Nelder-Mead almost always gets stuck in poor local optima (in the context of optimizing hyperparameters for machine learning methods, so ymmv).

Common methods include:

  • Nelder-Mead
  • particle swarm optimization
  • genetic algorithms
  • harmonic search
  • racing algorithms
  • EGO
  • Bayesian optimization
  • ...

Many of the metaheuristic methods mentioned above are offered in Optunity, while several other packages offer Bayesian optimization (e.g. Hyperopt, SMAC and BayesOpt).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.