12
$\begingroup$

There is a kind of simulation study that is commonly used to validate an implementation of a Bayesian model:

  • For independent replication $i = 1, ..., n$:
    1. Draw a set of "true" parameters parameters from the joint prior.
    2. Draw a dataset from the likelihood given the parameter draws from (1).
    3. Approximate the full joint posterior distribution, e.g. with MCMC or variational inference.
    4. For each parameter (index $p$) let $c_{ip}$ = 1 if the $100(1 - \alpha)$% posterior interval covers the prior predictive draw from (1). Otherwise, $c_{ip}$ = 0.
  • For each parameter $p$, calculate coverage: $C_p = \frac{1}{n} \sum_{i = i}^n c_{ip}$. If $C_p < 1 - \alpha$, then there are problems in the model or the software.

This technique is super useful in my team's work, and it has caught a lot of errors. Does anyone know if it has an official name? I have been searching but have been unable to find it. At first I thought it was called "simulation-based calibration", but what I am describing does (4) above instead of the calibration part.

References

  • Andrew Gelman, Aki Vehtari, Daniel Simpson, Charles C. Margossian, Bob Carpenter, Yuling Yao, Lauren Kennedy, Jonah Gabry, Paul-Christian Bürkner, & Martin Modrák. (2020). Bayesian Workflow. https://arxiv.org/abs/2011.01808

  • Cook, Samantha R., Andrew Gelman, and Donald B. Rubin. 2006. “Validation of Software for Bayesian Models Using Posterior Quantiles.” Journal of Computational and Graphical Statistics 15 (3): 675–92. http://www.jstor.org/stable/27594203.

  • Talts, Sean, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. 2020. “Validating Bayesian Inference Algorithms with Simulation-Based Calibration.” http://arxiv.org/abs/1804.06788.

Improve this question
$\endgroup$
13
  • 1
    $\begingroup$ Have you considered "posterior predictive checks"? $\endgroup$
    – svendvn
    Commented Mar 9, 2021 at 23:54
  • $\begingroup$ I often do when feasible, but this particular simulation does not use the posterior predictive distribution (only the marginal posterior of each parameter). “Posterior predictive checks” and “posterior checks” sound a bit too general for this. $\endgroup$
    – landau
    Commented Mar 9, 2021 at 23:57
  • $\begingroup$ Also, I would like to find the name that is already widely used in the community, rather than try to invent a name myself $\endgroup$
    – landau
    Commented Mar 9, 2021 at 23:59
  • $\begingroup$ Never heard of it. $\endgroup$
    – Xi'an
    Commented Mar 10, 2021 at 7:12
  • 1
    $\begingroup$ Proof of average property: just write the joint as $p(x,D|M) = p(x|D,M) p(D|M)$. This is distribution from which we’re sampling true parameters and data. Then clear that in every simulation, for whatever $D$ you draw, since $p(x|D,M)$ is the posterior, there’s a X% chance you get a draw that lies the X% CR $\endgroup$
    – innisfree
    Commented Mar 11, 2021 at 15:34

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.