Have there been large scale studies of MCMC methods that compare the performance of several different algorithms on a suite of test densities? I am thinking of something equivalent to Rios and Sahinidis' paper (2013), which is a thorough comparison of a large number of derivative-free black-box optimizers on several classes of test functions.
For MCMC, performance can be estimated in, e.g., effective number of samples (ESS) per density evaluation, or some other appropriate metric.
A few comments:
I appreciate that performance will strongly depend on details of the target pdf, but a similar (possibly not identical) argument holds for optimization, and nonetheless there is a plethora of benchmark functions, suites, competitions, papers, etc. that deals with benchmarking optimization algorithms.
Also, it is true that MCMC differs from optimization in that comparatevely much more care and tuning is needed from the user. Nonetheless, there are now several MCMC methods that require little or no tuning: methods that adapt in the burn-in phase, during sampling, or multi-state (also called ensemble) methods (such as Emcee) that evolve multiple interacting chains and use information from other chains to guide the sampling.
I am particularly interested in the comparison between standard and multi-state (aka ensemble) methods. For the definition of multi-state, see Section 30.6 of MacKay's book:
In a multi-state method, multiple parameter vectors $\textbf{x}$ are maintained; they evolve individually under moves such as Metropolis and Gibbs; there are also interactions among the vectors.
- This question originated from here.
Update
- For an interesting take on multi-state aka ensemble methods, see this blog post by Bob Carpenter on Gelman's blog, and my comment referring to this CV post.