# Official name of a common type of Bayesian simulation study

There is a kind of simulation study that is commonly used to validate an implementation of a Bayesian model:

• For independent replication $$i = 1, ..., n$$:
1. Draw a set of "true" parameters parameters from the joint prior.
2. Draw a dataset from the likelihood given the parameter draws from (1).
3. Approximate the full joint posterior distribution, e.g. with MCMC or variational inference.
4. For each parameter (index $$p$$) let $$c_{ip}$$ = 1 if the $$100(1 - \alpha)$$% posterior interval covers the prior predictive draw from (1). Otherwise, $$c_{ip}$$ = 0.
• For each parameter $$p$$, calculate coverage: $$C_p = \frac{1}{n} \sum_{i = i}^n c_{ip}$$. If $$C_p < 1 - \alpha$$, then there are problems in the model or the software.

This technique is super useful in my team's work, and it has caught a lot of errors. Does anyone know if it has an official name? I have been searching but have been unable to find it. At first I thought it was called "simulation-based calibration", but what I am describing does (4) above instead of the calibration part.

### References

• Andrew Gelman, Aki Vehtari, Daniel Simpson, Charles C. Margossian, Bob Carpenter, Yuling Yao, Lauren Kennedy, Jonah Gabry, Paul-Christian Bürkner, & Martin Modrák. (2020). Bayesian Workflow. https://arxiv.org/abs/2011.01808

• Cook, Samantha R., Andrew Gelman, and Donald B. Rubin. 2006. “Validation of Software for Bayesian Models Using Posterior Quantiles.” Journal of Computational and Graphical Statistics 15 (3): 675–92. http://www.jstor.org/stable/27594203.

• Talts, Sean, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. 2020. “Validating Bayesian Inference Algorithms with Simulation-Based Calibration.” http://arxiv.org/abs/1804.06788.

• Have you considered "posterior predictive checks"? Mar 9 at 23:54
• I often do when feasible, but this particular simulation does not use the posterior predictive distribution (only the marginal posterior of each parameter). “Posterior predictive checks” and “posterior checks” sound a bit too general for this. Mar 9 at 23:57
• Also, I would like to find the name that is already widely used in the community, rather than try to invent a name myself Mar 9 at 23:59
• Never heard of it. Mar 10 at 7:12
• Whilst I have heard of people investigating the frequentist properties of credible regions, CRs offer no coverage guarantees. And so I can’t see why you’d conclude a bug in software or model if it didn’t have a particular coverage. Mar 10 at 15:43