# Coverage probability of credible intervals if we take Bayesian model literally

Let's say I have a Bayesian model with a proper prior $\pi$, likelihood $L$, data distribution $p(x|\theta)$ (assume $\theta$ is a scalar) and the vector of sample values $x$:

$$p(\theta|x) = \frac{\pi(\theta)L(\theta|x)}{p(x)}$$

If I took my model seriously as a hierarchical description of the data generating process,

$$\theta \sim \pi$$ $$x\sim p(\theta)$$

then how often would a 95% credibility interval (constructed from the posterior distribution) contain the generated $\theta$?

Also, would this coverage statement be conditional on the collected data, or would it be unconditional and hence vulnerable to the usual relevant subset arguments directed at confidence intervals?

I asked a similar question: Methods for testing a Bayesian method's software implementation and got this answer from @jaradniemi:

Bayesians don't lose the relative frequency-based interpretation of probability. In particular, if you define this procedure:

1. simulate from the prior,
2. then simulate from the model using those values from the prior, and
3. estimate the parameters using the same prior.

Then your credible intervals should have the appropriate frequentist coverage, i.e. 95% intervals should include the true parameter in 95% of your analyses, over repeated replicates of the procedure.

I think he's right. You're allowed to stray from the relative-frequency concept of probability as a bayesian, but you don't automatically lose it. And if you're generating data, you definitely haven't lost it.

Regarding your question about conditional vs unconditional, while each individual credible interval would be conditional on the data, the relative-frequency based coverage would be unconditional since you're averaging over draws of the data. You can also search the web for "frequentist properties of bayesian methods" and get quite a few hits.

• Great answer. As far as conditional vs unconditional...does this mean that the frequentist assessments advocated by Little, Rubin, etc under the name "Calibrated Bayes" are also unconditional, even though the intervals provided by Bayes are free of relevant subsets? I guess one really can't get away from the reference class problem, eh? ime.usp.br/~abe/lista/pdffHigH29tEm.pdf – user145807 Jun 12 '17 at 13:18
• Just glancing through the paper, Box's factorization in Section 5 seems to completely average over the data, whereas the second factorization (Rubin, Gelman, etc.) is still conditional on the "observed data." The latter method does average over future unseen data, however, and is thus unconditional in that regard. About the relevant subsets, I would guess that there would again be pathological subsets that, when conditioned upon, would lead to the same nonsense results, but I offer no proof! – Ben Ogorek Jun 13 '17 at 0:03

As there is no generally accepted / unique way to specify (uninformative) priors, and as different priors will lead to a different credible intervals, it seems obvious that the coverage of Bayesian CIs is not fixed, but will depend on the prior that you choose, in relation to the "true" parameter values.

It will generally also depend on the data, in particular on the size of the data - in a low data situation, the prior dominates the CI, with the mentioned consequences of a strong prior influence on the coverage. When moving to large data / asymptotics, the Bayesian CI and the frequentist CI will usually become increasingly similar, including their coverage properties.