4
$\begingroup$

I learned that the credible interval does not have the frequentist property, but recently I read the following statements that derived from the credible interval/region:

Point (0,0) is on the edge line of the 98% credible region for the joint posterior density. The test for overall treatment effects is significant with p-value 0.02.

in this thesis (Page 88) and

We also depict the upper and lower 2.5% posterior quantiles in the figure. From these posterior inferences, we can further identify differentially expressed proteins. For example, if we require the 2.5% quantile above zero or the 97.5% quantile below zero, there are 19 up-regulated and 7 down-regulated proteins.

in this paper (Section 4).

Are they proper? Or what is the proper way to conclude from a credible interval? Any input will be greatly appreciated.

$\endgroup$
  • 1
    $\begingroup$ A related and interesting discussion can be found here, about the connection between credible regions and Bayesian hypothesis tests, though "it does not seem to solve the point-null hypothesis Bayesian testing problem." $\endgroup$ – Randel Nov 30 '15 at 17:37
6
$\begingroup$

Confidence intervals can be used equivalently to hypothesis tests, but highest density intervals are not the same as confidence intervals. Let's start with what $p$-value is by quoting Cohen (1994)

What we want to know is "Given this data what is the probability that $H_0$ is true?" But as most of us know, what it $p$-value tells us is "Given that $H_0$ is true, what is the probability of this (or more extreme) data?" These are not the same (...)

So $p$-value tells us what is the $P(D|H_0)$. In Bayesian approach we want to learn directly (rather than indirectly) about probability of some parameter given the data that we have $P(\theta|D)$ by employing the Bayes theorem and using priors for $\theta$

$$ \underbrace{P(\theta|D)}_\text{posterior} \propto \underbrace{P(D|\theta)}_\text{likelihood} \times \underbrace{P(\theta)}_\text{prior} $$

So if 95% confidence interval does not include the null value(s), than you can reject your null hypothesis: your data is more extreme than you would expect given your hypothesis. On the other hand, if in Bayesian setting your 95% highest density interval does not include null value(s), than you can conclude that probability of observing such value(s) is less than 95%.

Kruschke (2010) can be quoted for comparison of both approaches

The primary goal of NHST [Null Hypothesis Significance Testing] is determining whether a particular "null" value of a parameter can be rejected. One can also ask what range of parameter values would not be rejected. This range of non-rejectable parameter values is called the confidence interval. (...) The confidence interval tells us something about the probability of extreme unobserved data values that we might have gotten if we repeated the experiment (...)

A concept in Bayesian inference, that is somewhat analogous to the NHST confidence interval, is the highest density interval (HDI), (...) The 95% HDI consists of those values of $\theta$ that have at least some minimal level of posterior believability, such that the total probability of all such $\theta$ values is 95%. (...) The NHST confidence interval, on the other hand, has no direct relationship with what we want to know; there's no clear relationship between the probability of rejecting the value $\theta$ and the believability of $\theta$.

Posterior probability can be used and is used for testing hypothesis, but you have to remember that it provides answer for a different question than $p$-values.

See also: What is the connection between credible regions and Bayesian hypothesis tests? and Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?


Cohen, J. (1994). The earth is round (p<.05). American Psychologist, 49, 997-1003.

Kruschke, J.K. (2010). Doing Bayesian Data Analysis: A Tutorial with R and BUGS. Academic Press / Elsevier.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ +1. So the two quotes in the question are improper? $\endgroup$ – Randel Dec 1 '15 at 14:10
  • $\begingroup$ @Randel As about #1, as said, this is not exactly p-value (while often it is used in similar manner); as about #2 it seems to describe a different scenario. $\endgroup$ – Tim Dec 1 '15 at 14:22
  • 1
    $\begingroup$ @Randel what I'm saying is that even if you want to use it in similar fashion as p-values, you have to remember that is is different and concerns with different probabilities than p-values. $\endgroup$ – Tim Dec 1 '15 at 14:27
  • $\begingroup$ Thanks! Can I think in this way? For #1, "while often it is used in similar manner", but we cannot call it "p-value"; for #2, it directly uses credible interval for hypothesis testing but it's not in the frequentist favor? It seems not rigorous, e.g., not consider multiple testing. $\endgroup$ – Randel Dec 1 '15 at 14:36
  • 1
    $\begingroup$ @Randel with #1 you are correct. But #2 is taken out of context and it is hard to comment like this. $\endgroup$ – Tim Dec 1 '15 at 14:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.