A researcher computes both a frequentist confidence interval, and a Bayesian credible interval. After the computation, the researcher realizes that the credible interval is much more narrow than the confidence interval. What could be a reason why this happened?

An analyst conducts both a Bayesian hypothesis test (using Bayes Factors) and classic NHST (using p-values). The Bayesian results indicate weak support for the null hypothesis, but the p-value is small, indicating that the null should be rejected. Consider a reason why this could happen.

We have discussed four ways of statistical inference in class (NHST, Confidence Intervals, Bayes Factors, Credible Intervals). For each of these four methods, describe BRIEFLY (one or two sentences) two advantages and two disadvantages. You can list reasons that we have covered in class, or you can be creative and anticipate certain problems or benefits of the methods.

  • $\begingroup$ The Bayesian credible can be shorter than the frequentist CI if the Bayesian analysis uses an informative prior--especially an informative prior that is somewhat in agreement with the population. $\endgroup$ – BruceET Oct 16 '20 at 0:45
  • 1
    $\begingroup$ @BruceET thank you that makes great sense. So if the credible interval is in the same area but only narrower than the confidence interval, then there must be some information we have included in the prior that gives us a more specific range of values that we are sure are the true values? $\endgroup$ – whitsonBeau Oct 16 '20 at 1:01
  • $\begingroup$ That's the idea. See examples in my Answer. $\endgroup$ – BruceET Oct 16 '20 at 1:18

Example: Consider $n = 100$ observations from a population with success probability $\theta = 1/4.$ Suppose the results are as sampled in R below, with $X = 29$ Successes.

Frequentist confidence interval. Then a frequentist Agresti-Coull 95% CI for $\theta$ is $(0.210, 0.386).$

x = rbinom(1, 100, 1/4);  x
[1] 29
th.est = (x+2)/(100+4);  pm = c(-1,1)
th.est + pm*1.96*sqrt(th.est*(1-th.est)/104)
[1] 0.2101649 0.3859890

Bayestian credible interval from informative prior. Suppose the Bayesian prior distribution is $\theta \sim \mathsf{Beta}(10, 30),$ which puts 95% of its probability in $(0.130,0.393).$

[1] 0.1303768 0.3932615

Then with the given prior distribution and a binomial likelihood function based on $x = 29$ successes in $n = 100,$ the Bayesian posterior distribution is $\mathsf{Beta}(10+x, 30+n-x) = \mathsf{Beta}(49, 101)$ and the Bayesian 95% posterior interval estimate is $(0.254, 0.404),$ which is somewhat shorter than the frequentist Agresti-Coull interval above.

qbeta(c(.025,.975), 49, 101)
[1] 0.2541815 0.4035301

Bayes credible interval from Jeffreys prior. By contrast, if the Bayesian analysis had begun with the non-informative Jeffreys prior distribution $\mathsf{Beta}(0.5,0.5),$ then the Bayesian credible interval would have been $(0.208, 0.384),$ much the same as the frequentist Agresti interval.

qbeta(c(.025, .975), 29.5, 71.5)
[1] 0.2079839 0.3839729
  • 1
    $\begingroup$ thank you so much. This example is illuminating. $\endgroup$ – whitsonBeau Oct 16 '20 at 1:29
  • $\begingroup$ What do you suppose happens if I'm doing a Bayesian analysis and pretend to have prior knowledge which I don't really have, giving an "informative" prior that is pretty much wrong: say $\mathsf{Beta}(30, 10).?$ $\endgroup$ – BruceET Oct 16 '20 at 1:36
  • $\begingroup$ Then we fail to reject the null? Or we draw an opposite conclusion? $\endgroup$ – whitsonBeau Oct 16 '20 at 1:52
  • $\begingroup$ While your question deals with both interval estimation and hypothesis testing, please notice that my answer deals explicitly only with interval estimation. So the question in my last comment has to do with credible intervals. Can you say what Bayesian interval estimate results from the data of my example and the prior $\mathsf{Beta}(30,10).?$ How long in that interval? $\endgroup$ – BruceET Oct 16 '20 at 6:10
  • $\begingroup$ Just to make it clear, you are probably making the homework for OP $\endgroup$ – Firebug Oct 16 '20 at 17:39

Not the answer you're looking for? Browse other questions tagged or ask your own question.