# Selecting optimization algorithms

I am in the process of developing a multi-state model (using the msm package in R), and have been encountering the error message

initial value in 'vmmin' is not finite

This error occurs regardless of whether or not I scale the parameters or not, and for almost every optimization algorithm except the bobyqa algorithm (from the minqa package). Basically I went through a process of experimentation until something worked, but that does not necessarily mean that the choices I made are methodologically sound. I've tried to read up on this topic, but every resource I find immediately launches into heavy math jargon.

My question is - how does one go about choosing an optimization algorithm? What do I need to take into consideration when making the selection, and to what extent does the selection of a optimization algorithm affect how I interpret the results of the function using the algorithm? Also, I would be grateful if someone could recommend a resource explaining optimization that scaffolds well.

• Nocedal and Wright comes to mind. Commented Feb 1, 2015 at 18:16
• Thank you. This resource was helpful. Still a little heavy on the math, but the most accessible I've come across so far. Commented Feb 2, 2015 at 16:57

There are a number of things to consider:

• Global vs. Local
• Do you have analytic gradients or Hessians
• Is the function even differentiable
• The size of the problem/data set
• How much time do you have
• Are there special features of the problem for which a particular algorithm may be better

While there exist algorithms that are able to find global extrema (simulated annealing, various genetic algorithms) they can take an inordinately long time for very little return.

If you have analytic gradients or Hessians, it makes quasi-Newton methods much more appealing. If you have Hessians you can use trust methods, although with some exceptions (e.g. R trustOptim) these tend to bog down with large problems as the Hessian grows rather quickly. If you have gradients, you can use BFGS-type methods. If you don't, you can either use gradient-free methods like Nelder-Mead, Subplex (my favorite), COBYLA, BOBYQA, etc. Or you can use the quasi-Newton methods with a finite-differecer to estimate the gradient (as is default in R optim), but only if your objective function is differentiable. If you are looking at a least-squares issue specifically, Levernberg-Marquardt is a possibility.

I believe it does boil down to the specifics of your problem, although I've personally found most success with the Subplex-type algorithms (multi-start Nelder-Mead).

• I would also add to your list: is your function differentiable? Commented Feb 1, 2015 at 20:59
• Wouldn't that be covered in having gradients or hessians? Or do you mean to prevent use of finite differencers in those cases; good point. Commented Feb 1, 2015 at 21:00
• +1, though I can't help but miss the word convex in this answer. Maybe a note on heuristic methods would also be useful (tabu search, PSO, racing, Bayesian optimization, CMA-ES, ...). Commented Feb 1, 2015 at 23:14
• @Avraham, hessian of a function(x) requires that we take partial derivative. Let's suppose we are trying to optimize a discontinuous function signum(x), traditional optimization will not work since this function is discontinuous. In that case you have to rely on non traditional optimization like evolutionary algorithms. Commented Feb 2, 2015 at 2:08
• @forecaster Exactly, which is why I added the point after your last comment. Even numerical methods won't work if the function is truly non-differentiable. Commented Feb 2, 2015 at 2:09