Have there been large scale studies of MCMC methods that compare the performance of several different algorithms on a suite of test densities? I am thinking of something equivalent to Rios and Sahinidis' paper (2013), which is a thorough comparison of a large number of derivative-free black-box optimizers on several classes of test functions.

For MCMC, performance can be estimated in, e.g., effective number of samples (ESS) per density evaluation, or some other appropriate metric.

A few comments:

  • I appreciate that performance will strongly depend on details of the target pdf, but a similar (possibly not identical) argument holds for optimization, and nonetheless there is a plethora of benchmark functions, suites, competitions, papers, etc. that deals with benchmarking optimization algorithms.

  • Also, it is true that MCMC differs from optimization in that comparatevely much more care and tuning is needed from the user. Nonetheless, there are now several MCMC methods that require little or no tuning: methods that adapt in the burn-in phase, during sampling, or multi-state (also called ensemble) methods (such as Emcee) that evolve multiple interacting chains and use information from other chains to guide the sampling.

  • I am particularly interested in the comparison between standard and multi-state (aka ensemble) methods. For the definition of multi-state, see Section 30.6 of MacKay's book:

In a multi-state method, multiple parameter vectors $\textbf{x}$ are maintained; they evolve individually under moves such as Metropolis and Gibbs; there are also interactions among the vectors.

  • This question originated from here.

Update

  • For an interesting take on multi-state aka ensemble methods, see this blog post by Bob Carpenter on Gelman's blog, and my comment referring to this CV post.
up vote 5 down vote accepted

After some online searching, I have come under the impression that a comprehensive benchmark of established MCMC methods, analogous to what one can find in the optimization literature, does not exist. (I'd be happy to be wrong here.)

It is easy to find comparisons of a few MCMC methods on specific problems within an applied domain. This would be okay if we could pool this information -- however, the quality of such benchmarks is often insufficient (e.g., due to lack in the reported metrics or poor design choices).

In the following I will post what I believe are valuable contributions as I find them:

  • Nishihara, Murray and Adams, Parallel MCMC with Generalized Elliptical Slice Sampling, JMLR (2014). The authors propose a novel multi-state method, GESS, and perform a comparison with 6 other single-state and multi-state methods on 7 test functions. They evaluate performance as ESS (Effective Sample Size) per second and per function evaluation.

  • SamplerCompare is a R package with the goal of benchmarking MCMC algorithms -- exactly what was I was asking about in my original question. Unfortunately, the package contains only a few test functions; the accompanying paper reports no actual benchmarks (just a small example); and it seems there have been no follow-ups.

Thompson, Madeleine B. "Introduction to SamplerCompare." Journal of Statistical Software 43.12 (2011): 1-10 (link).

  • For an interesting take on multi-state aka ensemble methods, see this blog post by Bob Carpenter on Gelman's blog, and my comment referring to this CV post.
  • Your second link is dead -- could you change it to a working one? – Tim Jun 11 '16 at 16:09
  • @Tim: Fixed, thanks. – lacerbi Jun 11 '16 at 16:34

I would agree with your assessment that there are no comprehensive benchmarks established for MCMC methods. This is because every MCMC sampler has pros and cons, and are extremely problem specific.

In a typical Bayesian modeling setting, you can run the same sampler with diverse mixing rates when the data is different. I would go to the extent of saying that if in the future there comes out a comprehensive benchmark study of various MCMC samplers, I would not trust the results to be applicable outside of the examples shown.

Regarding the usage of ESS to assess sampling quality, it is worth mentioning that ESS depends on the quantity that is to be estimated from the sample. If you want to find the mean of the sample, the ESS obtained will be different from if you want to estimate the 25th quantile. Having said that, if the quantity of interest is fixed, ESS is a reasonable way of comparing samplers. Maybe a better idea is ESS per unit time.

One flaw with ESS is that for multivariate estimation problems, ESS returns an effective sample size for each component separately, ignoring all cross-correlations in the estimation process. In this paper recently, a multivariate ESS has been proposed, and implemented in R package mcmcsevia the function multiESS. It is unclear how this method compares to the ESS of the coda package, but at the very outset seems more reasonable than univariate ESS methods.

  • 2
    (+1) Thanks for the answer. I agree with some of your points, but I still think that some information could be gained from such a benchmark. How one uses the results of such benchmarks to guide future choices it's up to them -- but some evidence is better than no evidence. Good points about ESS. By multi-state I mean multi-state (or multi-chain, if you prefer), not simply multivariate -- see the quote from MacKay's book in my original question. – lacerbi Apr 5 '16 at 1:44
  • 2
    In general, some samplers are known to perform poorly for multimodal distributions(MH, Gibbs), and some are bad for non-convex support (Hamiltonian MC). On the other hand, for high dimensional problems Hamiltonian MC works well and for multimodal distributions simulated tempering etc are good. To do any benchmarking, one might need to define different broad classes of target distributions (sub-exponential, log concave etc) for the results to be interpretable generally. – Greenparker Apr 9 '16 at 17:46
  • 1
    Well, yes, that's the whole point of building a benchmark for a class of algorithms. See for example this for global optimization. Clearly a benchmark for MCMC cannot just borrow the ones existing for optimization; there is a need to focus on features of the target densities that are specific, common and of interest to MCMC problems, as those you mentioned. – lacerbi Apr 9 '16 at 19:24

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.