6
$\begingroup$

I have seen many people claiming Stan is "much better" than JAGS, meaning roughly this: "although Jags is much faster, the quality of the samples is worse. So it's worth waiting for the much longer time that Stan takes, because you "better" compute the uncertainty."

Ok, so far so good. But... I have not seen any good examples proving that practical claim. By that I mean examples in which the posterior distribution outputted by both samplers would be so different to the point of making a substantive difference (for instance, a different decision), and thus the burden of using Stan would be worth the time and pay off.

Could you provide such an example (with model and code)?

$\endgroup$
11
  • 2
    $\begingroup$ This isn't an answer, but I often find that methods I construct using JAGS have good Frequentist characteristics for the things I care about, and that the marginal distributions of the things I care about match what I get in STAN but are much faster to obtain. The mixing on things I don't care about might be very bad in these situations. I know from personal experience that (some on) the STAN team would tell you to use STAN anyways in these situations. $\endgroup$
    – guy
    Mar 13, 2021 at 1:46
  • 1
    $\begingroup$ But surely there must be some use case in which the wait is worth it, that's what I'm looking for. The developers just make generic claims like "you will see this difference in complex models." $\endgroup$
    – user314217
    Mar 13, 2021 at 1:50
  • 2
    $\begingroup$ I think your experience is fairly common among academic statisticians to be frank. I could say a lot more about this, but I'll leave it at that. The most convincing example I'm aware of is Neal's implementation of neural networks (which won a NeurIPS competition back in the day), which could never work via Gibbs sampling. Although, amusingly, the last time I tried using STAN to fit a neural network it completely choked. Neal also used some tricks that are not possible to use in STAN (nor will they ever be possible). $\endgroup$
    – guy
    Mar 13, 2021 at 1:56
  • 1
    $\begingroup$ Just another bit of food-for-thought on this, I know at least one STAN team member's take on tree-based Bayesian methods is basically "don't use them until we know how to do HMC on trees." BART, which is fit by Gibbs sampling the trees, is lightning-fast compared to STAN and wins the ACIC causal inference competition every year it seems. $\endgroup$
    – guy
    Mar 13, 2021 at 2:00
  • 3
    $\begingroup$ To be fair to them, it's pretty easy to write down models in JAGS that have no prayer of working, but have no problems in STAN. If you use some tricks you might be able to get JAGS to go through, or you might roll some custom Gibbs sampler to get it to work, but the value of STAN from an MCMC perspective is that you don't need to be as careful. $\endgroup$
    – guy
    Mar 13, 2021 at 2:09

2 Answers 2

4
$\begingroup$

Whenever I want to get started with understanding a new statistical topic, I start by reading articles about it. In this case, I'd start with Carpenter et al. "Stan: A Probabilistic Programming Language," in the Journal of Statistical Software which introduces Stan. The first paragraph is enough to get us started.

The goal of the Stan project is to provide a flexible probabilistic programming language for statistical modeling along with a suite of inference tools for fitting models that are robust, scalable, and efficient.

Stan differs from BUGS (Lunn, Thomas, and Spiegelhalter 2000; Lunn, Spiegelhalter, Thomas, and Best 2009; Lunn, Jackson, Best, Thomas, and Spiegelhalter 2012) and JAGS (Plummer 2003) in two primary ways. First, Stan is based on a new imperative probabilistic programming language that is more flexible and expressive than the declarative graphical modeling languages underlying BUGS or JAGS, in ways such as declaring variables with types and supporting local variables and conditional statements. Second, Stan’s Markov chain Monte Carlo (MCMC) techniques are based on Hamiltonian Monte Carlo (HMC), a more efficient and robust sampler than Gibbs sampling or Metropolis Hastings for models with complex posteriors.1

The number at the end is a footnote. It has citations which support the claim made in that sentence. In this case, the footnote reads "Neal (2011) analyzes the scaling benfit of HMC with dimensionality. Hoffman and Gelman (2014) provide practical comparisons of Stan’s adaptive HMC algorithm with Gibbs, Metropolis, and standard HMC samplers," and the citations are

  • Neal R (2011). “MCMC Using Hamiltonian Dynamics.” In S Brooks, A Gelman, GL Jones, XL Meng (eds.), Handbook of Markov Chain Monte Carlo, pp. 116–162. Chapman and Hall/CRC.

  • Hoffman MD, Gelman A (2014). “The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo.” Journal of Machine Learning Research, 15(Apr), 1593–1623.

which will elaborate on the differences in more detail.

It's important to point out that the advantages of stan cited in this passage are not related to the posteriors being different, but are facts about efficiency.

Indeed, one can show that, under certain conditions, Gibbs, MCMC and Metropolis-Hastings will converge to the posterior (albeit it might take far too long for the chains to mix compared to HMC/NUTS), so it would be surprising that HMC/NUTS would differ when these conditions are met.


Bob Carpenter, one of the developers of Stan, provides a concrete example of a case where Stan can solve a problem that Gibbs sampling cannot in this thread on the Stan forums.

[T]here’s an example of how to code exactly this model in the latent discrete parameters chapter of the users guide. You can find this example and others in my latest paper ["Comparing Bayesian Models of Annotation" by Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, Massimo Poesio. Transactions of the Association for Computational Linguistics (2018)], all coded in Stan.

Gibbs is actually a very bad way to fit these models—it’s super slow to converge. These models used to take 24 hours to fit in WinBUGS with very poor mixing and they now fit in like 30 minutes in Stan. Just be careful to use reasonable inits because there’s a non-identifiability. Duco Veen’s visiting us at Columbia from Utrecht and working on a case study that should be out soon.

In other words, if you're trying to estimate this model and you run WinBUGS for 30 minutes, the chains that you get from the WinBUGS model will exhibit poor mixing, the model will not converge, and the samples will not be representative of the posterior density. At that point, you have a choice. You can wait another 23 hours and 30 minutes for the chains to mix, or you can code the model in Stan.


Not all parameterizations, or even all models, are going to be fast to estimate in Stan. There are problematic parameterizations, also discussed in the Stan User Guide, which have a geometry that's very hard for HMC/NUTS to navigate. The User Guide also contains suggested reparameterizations which can ameliorate these problems. This does not imply that all models can be estimated in Stan, or even that Stanwill be more efficient for any particular model; some models are simply challenging, either generically or for Stan specifically.

That said, Stan is a tool that solves some specific problems more quickly compared to popular alternatives. Part of obtaining expertise is knowing how to differentiate among the various alternative tools and methods for solving problems and choosing the tool that is best for the job.

$\endgroup$
0
0
$\begingroup$

Currently there are no obvious real world examples in which using Stan vs JAGS matters (that is, the posterior would be so different that it would induce a substantively different decisions or conclusions). This is backed by personal experience and it was further evidenced by the lack of answers to this very simple question, even after a bounty was offered.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.