7
$\begingroup$

I've been doing some reading on the topic of credible vs confidence intervals but unfortunately it feels like the more I read the more I'm confused. There seems to be a general sense or consensus that confidence intervals are unintuitive. In particular, a $(1-\alpha)$ confidence interval you've calculated based on your data cannot be interpreted as having probability $(1-\alpha)$ of containing the true value $\theta_0$ of the parameter $\theta$ you're trying to estimate. Instead the correct interpretation of confidence intervals seem to be upon repeated samples of the data from the likelihood ('repeated experiments'), $(1-\alpha)$ of the confidence intervals generated (which will differ every experiment) will contain the true value $\theta_0$.

The Bayesian credible interval is usually provided as a way to resolve this problem, where a $(1-\alpha)$ credible interval can indeed be interpreted as the true value $\theta_0$ having probability $(1-\alpha)$ of lying in the credible interval.

However, among theoretical statisticians there seems to be significant effort and interest in trying to prove mathematically whether a $(1-\alpha)$ posterior credible interval (or region) also have $(1-\alpha)$ frequentist coverage. The most famous theorem on this is perhaps the Bernstein-von Mises theorem. In this report of open problems in Bayesian statistics published in 2011 by the International Society for Bayesian Analysis (https://www.stat.berkeley.edu/~aldous/157/Papers/Bayesian_open_problems.pdf), trying to establish the frequentist coverage of Bayesian credible regions in nonparametric models is repeatedly mentioned.

I also get the gist from talking to some researchers that confidence intervals is the 'correct' way to quantify uncertainty somehow in many if not most scientific studies, so it's important that any credible intervals from Bayesian models in statistical studies should have the same frequentist coverage, but I don't fully understand why.

Any comments on helping me understand the following would be much appreciated:

  1. Why is it important for Bayesian credible intervals to have the same level of frequentist coverage from both a theoretical/mathematical and practical/scientific point of view?

  2. Is the crux that if the parameter $\theta$ you're trying to estimate has a 'true value' in the real world (e.g. average height of a population), that frequentist coverage is the property you're looking for in uncertainty quantification, and if so why? If not, what's the deficiency credible intervals have that confidence intervals do not have, and why is this deficiency important in statistical studies?

$\endgroup$

1 Answer 1

1
$\begingroup$

Instead the correct interpretation of confidence intervals seem to be upon repeated samples of the data from the likelihood ('repeated experiments'), (1−α) of the confidence intervals generated (which will differ every experiment) will contain the true value θ

Your understanding of the confidence interval is complicated by your thinking of it as a Bayesian interval. For a frequentist, an interval does not contain a true value (or not), rather it summarizes an interval or range into which an estimated frequency of results would occur if the study were replicated. Consider the estimation of normal variance without a degree of freedom correction: this is the maximum likelihood estimator, and it is biased. If I construct a 95% CI for this value using the appropriate $\chi^2$ statistics, and I replicate the study an infinite number of times, my biased estimators will fall in the interval 95% of the time as stated. These replicates will not, however, contain the true variance 95% of the time... the coverage will be slightly less than that. It is standard pedagogy to emphasize that the confidence interval does not reflect a probability of containing a true value - the only reason I can imagine for the persistence of this misunderstanding seems to be a lazy approach to pedagogy.

Results about frequentist properties of Bayesian estimators are tremendously interesting for a number of reasons. In particular, it may not be desirable to appeal to the Bayesian interpretation of probability, but rather to develop new ways of estimating values that were previously intractable. Mixed models and models for missing or truncated data come to mind. It's somewhat well known that the EM algorithm can maximize a likelihood, and that there are semiparametric regression techniques for some of these cases, but what about a Bayesian estimator? Could an informative prior be useful? Could the Bayes estimators in these cases improve the MSE? Can they overcome issues with singularities or unstable likelihood functions on the boundary? Bayesian statistics are, among many things, an interesting estimation technique.

I think the best way to understand this is that our preference for frequentist versus Bayesian interpretations of probability are, at times, subjective, and certainly do not apply for every type of problem. Given that, it should be easy to conceptualize frequentist problems (like a decision rule with well understood error rates, or applying standard NHST) where a Bayesian estimation technique could be useful.

$\endgroup$
3
  • $\begingroup$ For example, the statement "For a frequentist, an interval does not contain a true value or not, rather .......". What are you meaning? $\endgroup$ Feb 1 at 7:16
  • $\begingroup$ @GrahamBornholt my point is: the frequentist paradigm works (and Bayesians fail) when the estimand is defined by the study, so to speak. I gave you the example of the biased estimate of variance with $n$ fixed. Another example to consider is the case-control (CC) design. Miettenen points out (in many ways), that you could estimate the risk ratio (RR) if the goal is performing inference. The RR changes with the sampling fraction of cases when the alternative is true, but it is always 1 when the null is true.,, 1/2 $\endgroup$
    – AdamO
    Feb 1 at 17:40
  • $\begingroup$ @GrahamBornholt A Bayesian credible interval for the RR in a CC design is not well motiviated, but a frequentist confidence interval is. $\endgroup$
    – AdamO
    Feb 1 at 17:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.