4
$\begingroup$

I know that a bayesian credible interval and a frequentist confidence interval measure very different things, and have different interpretations. However, is it ever appropriate to treat a bayesian credible interval as a frequentist confidence interval (for example, for the purposes of adjustments for multiple comparisons) or to treat the bayesian quantity P(effect < 0) as a frequentist (one-sided) p-value?

Since there are many questions on Stats.SE asking about comparing credible intervals and confidence intervals, I want to clarify that I am specifically asking about taking a credible interval (or probability statement) estimated in a bayesian framework and treating it as if it were a confidence interval (or p-value) for numerical adjustment, NOT about whether I can interpret this credible interval as a confidence interval.

Improve this question
$\endgroup$
4
  • $\begingroup$ You should review stats.stackexchange.com/questions/2272/… $\endgroup$
    – Jon
    Dec 19, 2016 at 23:47
  • $\begingroup$ @Jon, thanks for pointing me toward that question. Top answer is a good read, and points out that there are cases when the two are not numerically equivalent. What I'm wondering is more along the lines of whether there are cases when it is appropriate to treat them as numerically equivalent. $\endgroup$
    – Empiromancer
    Dec 20, 2016 at 0:51
  • $\begingroup$ Sounds like you're asking if you can treat results from two opposing (loosely) philosophies as the same thing. Numerically, they can look the same but interpretation won't be. $\endgroup$
    – Jon
    Dec 20, 2016 at 1:24
  • $\begingroup$ @user164385 Maybe this answer stats.stackexchange.com/a/250148/14346 may provide some clues. $\endgroup$
    – beuhbbb
    Dec 20, 2016 at 9:11

1 Answer 1

Reset to default
4
$\begingroup$

"Is it ever appropriate to treat a Bayesian credible interval as a frequentist confidence interval (for example, for the purposes of adjustments for multiple comparisons) or to treat the Bayesian quantity P(effect < 0) as a frequentist (one-sided) p-value?"

The short answer is no. There may be some numerical correspondences (not necessarily identities) in special cases, but not in general. And even when the numerical values happen to correspond, their meaning is still quite different.

In particular, a Bayesian credible interval does not make adjustments for multiple comparisons anything like a frequentist confidence interval. Frequentist p values and confidence intervals are adjusted for the intended set of tests because the goal is to quantify and control the error rate across the set of tests. There is no such quantification of error rates in a Bayesian derivation of a posterior distribution. (Bayesian models can indirectly attenuate false alarms [Type I errors], for example via shrinkage in hierarchical models.)

The posterior probability p(effect<0|Data) is sometimes used in Bayesian decision rules, but it is not a frequentist p value. Again, there may be special cases in which their numerical values correspond, but even then they mean very different things. The frequentist p value changes when the set of tests changes or when the stopping intention changes, while the Bayesian posterior distribution does not. The Bayesian posterior distribution changes when the prior changes, while the frequentist p value does not.

For a non-technical overview organized around the table below, see https://osf.io/dktc5/ enter image description here

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.