5
$\begingroup$

There is a kind of simulation study that is commonly used to validate an implementation of a Bayesian model:

  • For independent replication $i = 1, ..., n$:
    1. Draw a set of "true" parameters parameters from the joint prior.
    2. Draw a dataset from the likelihood given the parameter draws from (1).
    3. Approximate the full joint posterior distribution, e.g. with MCMC or variational inference.
    4. For each parameter (index $p$) let $c_{ip}$ = 1 if the $100(1 - \alpha)$% posterior interval covers the prior predictive draw from (1). Otherwise, $c_{ip}$ = 0.
  • For each parameter $p$, calculate coverage: $C_p = \frac{1}{n} \sum_{i = i}^n c_{ip}$. If $C_p < 1 - \alpha$, then there are problems in the model or the software.

This technique is super useful in my team's work, and it has caught a lot of errors. Does anyone know if it has an official name? I have been searching but have been unable to find it. At first I thought it was called "simulation-based calibration", but what I am describing does (4) above instead of the calibration part.

References

  • Andrew Gelman, Aki Vehtari, Daniel Simpson, Charles C. Margossian, Bob Carpenter, Yuling Yao, Lauren Kennedy, Jonah Gabry, Paul-Christian Bürkner, & Martin Modrák. (2020). Bayesian Workflow. https://arxiv.org/abs/2011.01808

  • Cook, Samantha R., Andrew Gelman, and Donald B. Rubin. 2006. “Validation of Software for Bayesian Models Using Posterior Quantiles.” Journal of Computational and Graphical Statistics 15 (3): 675–92. http://www.jstor.org/stable/27594203.

  • Talts, Sean, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. 2020. “Validating Bayesian Inference Algorithms with Simulation-Based Calibration.” http://arxiv.org/abs/1804.06788.

$\endgroup$
13
  • 1
    $\begingroup$ Have you considered "posterior predictive checks"? $\endgroup$
    – svendvn
    Mar 9 at 23:54
  • $\begingroup$ I often do when feasible, but this particular simulation does not use the posterior predictive distribution (only the marginal posterior of each parameter). “Posterior predictive checks” and “posterior checks” sound a bit too general for this. $\endgroup$
    – landau
    Mar 9 at 23:57
  • $\begingroup$ Also, I would like to find the name that is already widely used in the community, rather than try to invent a name myself $\endgroup$
    – landau
    Mar 9 at 23:59
  • $\begingroup$ Never heard of it. $\endgroup$
    – Xi'an
    Mar 10 at 7:12
  • $\begingroup$ Whilst I have heard of people investigating the frequentist properties of credible regions, CRs offer no coverage guarantees. And so I can’t see why you’d conclude a bug in software or model if it didn’t have a particular coverage. $\endgroup$
    – innisfree
    Mar 10 at 15:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.