8
$\begingroup$

Let's say I have a Bayesian model with a proper prior $\pi$, likelihood $L$, data distribution $p(x|\theta)$ (assume $\theta$ is a scalar) and the vector of sample values $x$:

$$p(\theta|x) = \frac{\pi(\theta)L(\theta|x)}{p(x)}$$

If I took my model seriously as a hierarchical description of the data generating process,

$$ \theta \sim \pi$$ $$ x\sim p(\theta)$$

then how often would a 95% credibility interval (constructed from the posterior distribution) contain the generated $\theta$?

Also, would this coverage statement be conditional on the collected data, or would it be unconditional and hence vulnerable to the usual relevant subset arguments directed at confidence intervals?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

I asked a similar question: Methods for testing a Bayesian method's software implementation and got this answer from @jaradniemi:

Bayesians don't lose the relative frequency-based interpretation of probability. In particular, if you define this procedure:

  1. simulate from the prior,
  2. then simulate from the model using those values from the prior, and
  3. estimate the parameters using the same prior.

Then your credible intervals should have the appropriate frequentist coverage, i.e. 95% intervals should include the true parameter in 95% of your analyses, over repeated replicates of the procedure.

I think he's right. You're allowed to stray from the relative-frequency concept of probability as a bayesian, but you don't automatically lose it. And if you're generating data, you definitely haven't lost it.

Regarding your question about conditional vs unconditional, while each individual credible interval would be conditional on the data, the relative-frequency based coverage would be unconditional since you're averaging over draws of the data. You can also search the web for "frequentist properties of bayesian methods" and get quite a few hits.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Great answer. As far as conditional vs unconditional...does this mean that the frequentist assessments advocated by Little, Rubin, etc under the name "Calibrated Bayes" are also unconditional, even though the intervals provided by Bayes are free of relevant subsets? I guess one really can't get away from the reference class problem, eh? ime.usp.br/~abe/lista/pdffHigH29tEm.pdf $\endgroup$
    – user145807
    Jun 12, 2017 at 13:18
  • 1
    $\begingroup$ Just glancing through the paper, Box's factorization in Section 5 seems to completely average over the data, whereas the second factorization (Rubin, Gelman, etc.) is still conditional on the "observed data." The latter method does average over future unseen data, however, and is thus unconditional in that regard. About the relevant subsets, I would guess that there would again be pathological subsets that, when conditioned upon, would lead to the same nonsense results, but I offer no proof! $\endgroup$
    – Ben Ogorek
    Jun 13, 2017 at 0:03
3
$\begingroup$

As there is no generally accepted / unique way to specify (uninformative) priors, and as different priors will lead to a different credible intervals, it seems obvious that the coverage of Bayesian CIs is not fixed, but will depend on the prior that you choose, in relation to the "true" parameter values.

It will generally also depend on the data, in particular on the size of the data - in a low data situation, the prior dominates the CI, with the mentioned consequences of a strong prior influence on the coverage. When moving to large data / asymptotics, the Bayesian CI and the frequentist CI will usually become increasingly similar, including their coverage properties.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.