Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Both processes seem to be used to estimate the maximum value of an unknown function, and both obviously have different ways of doing so.
But in practice is either method essentially interchangeable? Where would I want to use one over the other?
https://en.wikipedia.org/wiki/Simulated_annealing
Similar Question
Bayesian optimization or gradient descent?
Simulated Annealing (SA) is a very simple algorithm in comparison with Bayesian Optimization (BO). Neither method assumes convexity of the cost function and neither method relays heavily on gradient information.
SA is in a way a slightly educated random walk. The candidate solution jumps around over the solution space having a particular jump schedule (the cooling parameter). You do not care where you landed before, you don't know where you will land next. It is a typical Markov Chain approach. You do not model any strong assumptions about the underlaying solution surface. MCMC optimization has gone a long way from SA (see for example Hamiltonian Monte Carlo) but we will not expand further. One of the key issues with SA is that you need to evaluate a lot of times "fast". And it makes sense, you need as many samples as possible to explore as many states (ie. candidate solutions) as possible. You use only a tiny bit of gradient information (that you almost always accept "better" solutions).
Look now at BO. BO (or simplistically Gaussian Process (GP) regression over your cost function evaluations) tries to do exactly the opposite in terms of function evaluation. It tries to minimize the number of evaluation you do. It builds a particular non-parametric model (usually a GP) for your cost function that often assumes noise. It does not use gradient information at all. BO allows you to build an informative model of your cost function with a small number of function evaluations. Afterwards you "query" this fitted function for its extrema. Again the devil is in the details; you need to sample intelligently (and assume that your prior is half-reasonable too). There is work on where to evaluate your function next especially when you know that your function actually evolves slightly over time (eg. here).
An obvious advantage of SA over BO is that within SA is very straightforward to put constraints on your solution space. For example if you want non-negative solutions you just confine your sample distribution in non negative solutions. The same is not so direct in BO because even you evaluate your functions according your constraints (say non-negativity) you will need to actually constraint your process too; this taske while not impossible is more involved.
In general, one would prefer SA in cases that the cost function is cheap to evaluate and BO in cases that the cost function is expensive to evaluate. I think SA is slowly but steadily falling out of favour; especially the work of gradient-free optimization (eg. NEWQUA, BOBYQA) takes away one of its major advantages in comparsion with the standard gradient descent methods which is not having to evaluate a derivative. Similarly the work on adaptive MCMC (eg. see reference above) renders it wasteful in terms of MCMC optimization for almost all cases.
