Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
While formally deriving the confidence interval of an estimate, I ended up with a formula that resembles very closely the way $p$-value is computed.
Thus the question: are they formally equivalent? I.e. is rejecting an hypotheses $H_0 = 0$ with a critical value $\alpha$ equivalent to $0$ not belonging to the confidence interval with critical value $\alpha$?
Yes and no.
What you've observed is that when a test and a confidence interval is based on the same statistic, there is an equivalence between them: we can interpret the $p$-value as the smallest value of $\alpha$ for which the null value of the parameter would be included in the $1-\alpha$ confidence interval.
Let $\theta$ be an unknown parameter in the parameter space $\Theta\subseteq\mathbb{R}$, and let the sample $\mathbf{x}=(x_1,\ldots,x_n)\in\mathcal{X}^ n\subseteq\mathbb{R}^n$ be a realization of the random variable $\mathbf{X}=(X_1,\ldots,X_n)$. For simplicity, define a confidence interval $I_\alpha(\mathbf{X})$ as a random interval such that its coverage probability $$ P_\theta(\theta\in I_\alpha(\mathbf{X}))= 1-\alpha\qquad\mbox{for all }\alpha\in(0,1). $$ (You could similarly consider more general intervals, where the coverage probability either is bounded by or approximately equal to $1-\alpha$. The reasoning is analogous.)
Consider a two-sided test of the point-null hypothesis $H_0(\theta_0): \theta=\theta_0$ against the alternative $H_1(\theta_0): \theta\neq \theta_0$. Let $\lambda(\theta_0,\mathbf{x})$ denote the p-value of the test. For any $\alpha\in(0,1)$, $H_0(\theta_0)$ is rejected at the level $\alpha$ if $\lambda(\theta_0,x)\leq\alpha$. The level $\alpha$ rejection region is the set of $\mathbf{x}$ which lead to the rejection of $H_0(\theta_0)$: $$ R_\alpha(\theta_0)=\{\mathbf{x}\in\mathbb{R}^n: \lambda(\theta_0,\mathbf{x})\leq\alpha\}.$$
Now, consider a family of two-sided tests with p-values $\lambda(\theta,\mathbf{x})$, for $\theta\in\Theta$. For such a family we can define an inverted rejection region $$ Q_\alpha(\mathbf{x})=\{\theta\in\Theta: \lambda(\theta,\mathbf{x})\leq\alpha\}.$$
For any fixed $\theta_0$, $H_0(\theta_0)$ is rejected if $\mathbf{x}\in R_\alpha(\theta_0)$, which happens if and only if $\theta_0\in Q_\alpha(\mathbf{x})$, that is, $$ \mathbf{x}\in R_\alpha(\theta_0) \Leftrightarrow \theta_0\in Q_\alpha(\mathbf{x}). $$ If the test is based on a test statistic with a completely specified absolutely continuous null distribution, then $\lambda(\theta_0,\mathbf{X})\sim \mbox{U}(0,1)$ under $H_0(\theta_0)$. Then $$ P_{\theta_0}(\mathbf{X}\in R_\alpha(\theta_0))=P_{\theta_0}(\lambda(\theta_0,\mathbf{X})\leq\alpha)=\alpha. $$ Since this equation holds for any $\theta_0\in\Theta$ and since the equation above it implies that $$P_{\theta_0}(\mathbf{X}\in R_\alpha(\theta_0))=P_{\theta_0}(\theta_0\in Q_\alpha(\mathbf{X})),$$ it follows that the random set $Q_\alpha(\mathbf{x})$ always covers the true parameter $\theta_0$ with probability $\alpha$. Consequently, letting $Q_\alpha^C(\mathbf{x})$ denote the complement of $Q_\alpha(\mathbf{x})$, for all $\theta_0\in\Theta$ we have $$P_{\theta_0}(\theta_0\in Q_\alpha^C(\mathbf{X}))=1-\alpha,$$ meaning that the complement of the inverted rejection region is a $1-\alpha$ confidence interval for $\theta$.
An illustration is given below, showing rejection regions and confidence intervals corresponding to the the $z$-test for a normal mean, for different null means $\theta$ and different sample means $\bar{x}$, with $\sigma=1$. $H_0(\theta)$ is rejected if $(\bar{x},\theta)$ is in the shaded light grey region. Shown in dark grey is the rejection region $R_{0.05}(-0.9)=(-\infty,-1.52)\cup(-0.281,\infty)$ and the confidence interval $I_{0.05}(1/2)=Q_{0.05}^C(1/2)=(-0.120,1.120)$.
(Much of this is taken from my PhD thesis.)
Above I described the standard way of constructing confidence intervals. In this approach, we use some statistic related to the unknown parameter $\theta$ to construct the interval. There are also intervals based on minimization algorithms, which seek to minimize the length of the interval condition on the value of $X$. Usually, such intervals do not correspond to a test.
This phenomenon has to do with problems related to such intervals not being nested, meaning that the 94 % interval can be shorter than the 95 % interval. For more on this, see Section 2.5 of this recent paper of mine (to appear in Bernoulli).
In some problems, the standard confidence interval is not based on the same statistic as the standard test (as discussed by Michael Fay in this paper). In those cases, confidence intervals and tests may not give the same results. For instance, $\theta_0=0$ may be rejected by the test even though 0 is included in the confidence interval. This does not contradict the "yes" above, as different statistics are used.
As pointed out by f coppens in a comment, sometimes intervals and tests have somewhat conflicting goals. We want short intervals and tests with high power, but the shortest interval does not always correspond to the test with the highest power. For some examples of this, see this paper (multivariate normal distribution), or this (exponential distribution), or Section 4 of my thesis.
Some years ago, I posted a question here about whether a test-interval-equivalence exists also in Bayesian statistics. The short answer is that using standard Bayesian hypothesis testing, the answer is "no". By reformulating the testing problem a little bit, the answer can however be "yes". (My attempts at answering my own question eventually turned into a paper!)
When looking at a single parameter, it is possible that a test about the value of the parameter and the confidence interval "mismatch" depending on how they are constructed. In particular, a hypothesis test is a level $\alpha$-test, if it rejects the null hypothesis a proportion $\leq \alpha$ of the time when the null hypothesis is true. For that reason one can e.g. use estimates of model parameters (e.g. the variance) that are only valid under the null hypothesis. If one then tried to construct a CI by inverting this test, the coverage may be not quite right under the alternative hypothesis. For that reason one would usually construct a confidence interval differently so that the coverage is also right under the alternative, which can then lead to a (usually very small) mismatch.
