# Is rejecting the hypothesis using p-value equivalent to hypothesis not belonging to the confidence interval?

While formally deriving the confidence interval of an estimate, I ended up with a formula that resembles very closely the way $p$-value is computed.

Thus the question: are they formally equivalent? I.e. is rejecting an hypotheses $H_0 = 0$ with a critical value $\alpha$ equivalent to $0$ not belonging to the confidence interval with critical value $\alpha$?

• @f coppens: yes, if two tests are used, with different statistics, you end up with two different confidence intervals. But I think the OP discovered a basic fact: both the confidence interval and the p-value are obtained from the distribution of the same statistic, so both of them can be used to decide on rejecting the null hypothesis or not. – StijnDeVuyst Aug 28 '15 at 7:59
• @StijnDeVuyst: The Clopper/Pearon interval for a proportion and the Sterne interval for a proportion are both derived from the Binomial distribution with the same size (the p is unknown because they find a confidence interval for p). The difference between Clopper/Pearson and Sterne is due to the asymmetrie of the Binomial density. The Sterne interval tries to minimize the width of the interval and Clopper_pearson tries to keep symmetry (but due to the skewness of the Binomial this can only be found approximately). – user83346 Aug 28 '15 at 8:13
• Not in general, no. Consider cases where the width of the interval is a function of the estimated parameter value, while for the test the width of the interval is a function of the hypothesized one. An obvious example would be testing a binomial p. Let's use the normal approx. for simplicity (though the form of argument doesn't rely on it). Consider n=10, and a null of p=0.5. Imagine observing 2 heads; the null is not rejected (because "2" is inside a 95% interval about 0.5) but the CI for p doesn't include 0.5 (because the CI is narrower than the interval width under the null. – Glen_b Aug 28 '15 at 10:03
• Or if you need it to be large enough that the normal approx is good, try 469 heads in 1000 tosses, for H0 p=0.5; again the 95% CI for p doesn't include 0.5 but the 5% test doesn't reject, because the corresponding interval width under H0 is wider than under the alternative (which is what you do the CI from). – Glen_b Aug 28 '15 at 10:46
• @Glen_b: It seems that this newer question stats.stackexchange.com/questions/173005 provides an example of exactly the situation you were describing here. – amoeba Sep 18 '15 at 10:12

Yes and no.

## First the "yes"

What you've observed is that when a test and a confidence interval is based on the same statistic, there is an equivalence between them: we can interpret the $p$-value as the smallest value of $\alpha$ for which the null value of the parameter would be included in the $1-\alpha$ confidence interval.

Let $\theta$ be an unknown parameter in the parameter space $\Theta\subseteq\mathbb{R}$, and let the sample $\mathbf{x}=(x_1,\ldots,x_n)\in\mathcal{X}^ n\subseteq\mathbb{R}^n$ be a realization of the random variable $\mathbf{X}=(X_1,\ldots,X_n)$. For simplicity, define a confidence interval $I_\alpha(\mathbf{X})$ as a random interval such that its coverage probability $$P_\theta(\theta\in I_\alpha(\mathbf{X}))= 1-\alpha\qquad\mbox{for all }\alpha\in(0,1).$$ (You could similarly consider more general intervals, where the coverage probability either is bounded by or approximately equal to $1-\alpha$. The reasoning is analogous.)

Consider a two-sided test of the point-null hypothesis $H_0(\theta_0): \theta=\theta_0$ against the alternative $H_1(\theta_0): \theta\neq \theta_0$. Let $\lambda(\theta_0,\mathbf{x})$ denote the p-value of the test. For any $\alpha\in(0,1)$, $H_0(\theta_0)$ is rejected at the level $\alpha$ if $\lambda(\theta_0,x)\leq\alpha$. The level $\alpha$ rejection region is the set of $\mathbf{x}$ which lead to the rejection of $H_0(\theta_0)$: $$R_\alpha(\theta_0)=\{\mathbf{x}\in\mathbb{R}^n: \lambda(\theta_0,\mathbf{x})\leq\alpha\}.$$

Now, consider a family of two-sided tests with p-values $\lambda(\theta,\mathbf{x})$, for $\theta\in\Theta$. For such a family we can define an inverted rejection region $$Q_\alpha(\mathbf{x})=\{\theta\in\Theta: \lambda(\theta,\mathbf{x})\leq\alpha\}.$$

For any fixed $\theta_0$, $H_0(\theta_0)$ is rejected if $\mathbf{x}\in R_\alpha(\theta_0)$, which happens if and only if $\theta_0\in Q_\alpha(\mathbf{x})$, that is, $$\mathbf{x}\in R_\alpha(\theta_0) \Leftrightarrow \theta_0\in Q_\alpha(\mathbf{x}).$$ If the test is based on a test statistic with a completely specified absolutely continuous null distribution, then $\lambda(\theta_0,\mathbf{X})\sim \mbox{U}(0,1)$ under $H_0(\theta_0)$. Then $$P_{\theta_0}(\mathbf{X}\in R_\alpha(\theta_0))=P_{\theta_0}(\lambda(\theta_0,\mathbf{X})\leq\alpha)=\alpha.$$ Since this equation holds for any $\theta_0\in\Theta$ and since the equation above it implies that $$P_{\theta_0}(\mathbf{X}\in R_\alpha(\theta_0))=P_{\theta_0}(\theta_0\in Q_\alpha(\mathbf{X})),$$ it follows that the random set $Q_\alpha(\mathbf{x})$ always covers the true parameter $\theta_0$ with probability $\alpha$. Consequently, letting $Q_\alpha^C(\mathbf{x})$ denote the complement of $Q_\alpha(\mathbf{x})$, for all $\theta_0\in\Theta$ we have $$P_{\theta_0}(\theta_0\in Q_\alpha^C(\mathbf{X}))=1-\alpha,$$ meaning that the complement of the inverted rejection region is a $1-\alpha$ confidence interval for $\theta$.

An illustration is given below, showing rejection regions and confidence intervals corresponding to the the $z$-test for a normal mean, for different null means $\theta$ and different sample means $\bar{x}$, with $\sigma=1$. $H_0(\theta)$ is rejected if $(\bar{x},\theta)$ is in the shaded light grey region. Shown in dark grey is the rejection region $R_{0.05}(-0.9)=(-\infty,-1.52)\cup(-0.281,\infty)$ and the confidence interval $I_{0.05}(1/2)=Q_{0.05}^C(1/2)=(-0.120,1.120)$.

(Much of this is taken from my PhD thesis.)

## Now for the "no"

Above I described the standard way of constructing confidence intervals. In this approach, we use some statistic related to the unknown parameter $\theta$ to construct the interval. There are also intervals based on minimization algorithms, which seek to minimize the length of the interval condition on the value of $X$. Usually, such intervals do not correspond to a test.

This phenomenon has to do with problems related to such intervals not being nested, meaning that the 94 % interval can be shorter than the 95 % interval. For more on this, see Section 2.5 of this recent paper of mine (to appear in Bernoulli).

## And a second "no"

In some problems, the standard confidence interval is not based on the same statistic as the standard test (as discussed by Michael Fay in this paper). In those cases, confidence intervals and tests may not give the same results. For instance, $\theta_0=0$ may be rejected by the test even though 0 is included in the confidence interval. This does not contradict the "yes" above, as different statistics are used.

## And sometimes "yes" is not a good thing

As pointed out by f coppens in a comment, sometimes intervals and tests have somewhat conflicting goals. We want short intervals and tests with high power, but the shortest interval does not always correspond to the test with the highest power. For some examples of this, see this paper (multivariate normal distribution), or this (exponential distribution), or Section 4 of my thesis.

## Bayesians can also say both yes and no

Some years ago, I posted a question here about whether a test-interval-equivalence exists also in Bayesian statistics. The short answer is that using standard Bayesian hypothesis testing, the answer is "no". By reformulating the testing problem a little bit, the answer can however be "yes". (My attempts at answering my own question eventually turned into a paper!)

• Nice answer (+1) and (you partially do that) it may be good to point to the fact that sometimes confidence intervals and hypothesis tests have (potentially) conflicting goals: one tries to find a confidence interval 'as small as possible' while for hypothesis testing one tries to find a critical region 'as powerful as possible'. – user83346 Aug 28 '15 at 8:27
• @amoeba: Actually, I think that his "no" is my second "no". As far as I can tell, he bases the confidence interval on the statistic $T_1=(\hat{p}-p)/\sqrt{\hat{p}(1-\hat{p})/n}$ and the test on the statistic $T_2=(\hat{p}-p)/\sqrt{p(1-p)/n}$. Note the difference in the denominator. You can construct tests and intervals using either statistic, and as long as you use the same statistic for both, there will be no discrepancy. – MånsT Aug 28 '15 at 19:44
When looking at a single parameter, it is possible that a test about the value of the parameter and the confidence interval "mismatch" depending on how they are constructed. In particular, a hypothesis test is a level $\alpha$-test, if it rejects the null hypothesis a proportion $\leq \alpha$ of the time when the null hypothesis is true. For that reason one can e.g. use estimates of model parameters (e.g. the variance) that are only valid under the null hypothesis. If one then tried to construct a CI by inverting this test, the coverage may be not quite right under the alternative hypothesis. For that reason one would usually construct a confidence interval differently so that the coverage is also right under the alternative, which can then lead to a (usually very small) mismatch.