# Can you list all direct search methods to estimate parameters?

Direct search methods (also called derivative-free) are function minimization techniques that do not rely on a function's gradient to find the minimum. More importantly, they can be used to solve non-smooth optimization problems, which can be then used in parameter estimation (e.g., xxx).

As nearly every statistician will recognize, the most commonly used method is the Nelder-Mead, 1965 (see here for a description). The optim function in R implements the Nelder-Mead method.

Here, I'd like to create a complete list (with strengths and defects) of other approaches and how to use them. Can you help?

• Typically, shopping-list questions are frowned upon, so I wouldn't be surprised if this gets closed. Oct 4, 2015 at 14:59

Hyperparameter search uses a very wide variety of optimization methods. In my opinion, Nelder-Mead is often a poor choice because hyperparameter response surfaces are usually not smooth with tons of local optima. In my experience, Nelder-Mead almost always gets stuck in poor local optima (in the context of optimizing hyperparameters for machine learning methods, so ymmv).

Common methods include: