We observe an observation of a random variable $X$ assumed to be distributed $X \sim F(\cdot | \theta)$ where $\theta \in \Theta$ is unknown and the parameter of interest. We want to define a confidence set $C(X) \subset \Theta$ with coverage probability $1 - \alpha$ or formally: $$ \inf_{\theta \in \Theta} \text{Pr}_\theta (\theta \in C(X)) \ge 1 - \alpha $$ One way to construct such a confidence interval is through test inversion. Specifically, we can construct a confidence interval as all $\theta_0 \in \Theta$ such that we cannot reject the null hypothesis that $\theta = \theta_0$ at level $\alpha$ or: $$ C(X) = \{\theta_0 \in \Theta: H_0 : \theta = \theta_0 \text{ is not rejected}\} $$ Assuming that the hypothesis tests are valid, this confidence interval has a coverage probability of greater than or equal to $ 1 - \alpha$ because for any $\theta \in \Theta$: $$ \text{Pr}_\theta(\theta \in C(X)) = \text{Pr}_\theta(\text{we don't reject } \theta = \theta_0) \ge 1 - \alpha $$ where the last inequality follows because the hypothesis test is valid.


While the above definitions make sense, I have always been confused how to reconcile this with my understanding of hypothesis tests. Specifically, my understanding of hypothesis tests is that their results and corresponding p-values tell you nothing about the probability that the null-hypothesis is true (or $\text{Pr}(\theta = \theta_0 | X)$). Yet, the confidence interval constructed via test inversion gives you a subset of $\Theta$ where we know the probability that $\theta$ is in this subset. This seems contradictory.

Does anyone have any intuition (or can correct a mistake somewhere) for how to reconcile this understanding of hypothesis tests with valid confidence intervals from test inversion?


1 Answer 1


First, note that there are various ways to try to test a dataset, and there are various ways to try to construct a confidence interval. As a result, it is possible for a test to be inconsistent with a confidence interval (see: Is rejecting the hypothesis using p-value equivalent to hypothesis not belonging to the confidence interval?). That said, you can create a confidence interval using the method you suggest, and they would obviously thus correspond to each other.

Your interpretation of hypothesis testing ("their results and corresponding p-values tell you nothing about the probability that the null-hypothesis is true") is correct. Your confusion lies in your description of the nature of a (frequentist) confidence interval. The description is reasonable, as far as it goes, but there is a subtle ambiguity that is leading you astray. Namely, the probability in the definition of the confidence interval pertains to the long-run frequency of confidence intervals that cover the true value of $\theta$. It is not the probability that any given confidence interval does. A formed confidence interval is a realization (a set of realized values). It either includes the true value of $\theta$ (with probability $1$) or it doesn't (in which case the probability is $0$), but you don't know which. It may help to read, Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.