# Intuition for why confidence intervals can be constructed by inverting tests

Setup

We observe an observation of a random variable $X$ assumed to be distributed $X \sim F(\cdot | \theta)$ where $\theta \in \Theta$ is unknown and the parameter of interest. We want to define a confidence set $C(X) \subset \Theta$ with coverage probability $1 - \alpha$ or formally: $$\inf_{\theta \in \Theta} \text{Pr}_\theta (\theta \in C(X)) \ge 1 - \alpha$$ One way to construct such a confidence interval is through test inversion. Specifically, we can construct a confidence interval as all $\theta_0 \in \Theta$ such that we cannot reject the null hypothesis that $\theta = \theta_0$ at level $\alpha$ or: $$C(X) = \{\theta_0 \in \Theta: H_0 : \theta = \theta_0 \text{ is not rejected}\}$$ Assuming that the hypothesis tests are valid, this confidence interval has a coverage probability of greater than or equal to $1 - \alpha$ because for any $\theta \in \Theta$: $$\text{Pr}_\theta(\theta \in C(X)) = \text{Pr}_\theta(\text{we don't reject } \theta = \theta_0) \ge 1 - \alpha$$ where the last inequality follows because the hypothesis test is valid.

Question

While the above definitions make sense, I have always been confused how to reconcile this with my understanding of hypothesis tests. Specifically, my understanding of hypothesis tests is that their results and corresponding p-values tell you nothing about the probability that the null-hypothesis is true (or $\text{Pr}(\theta = \theta_0 | X)$). Yet, the confidence interval constructed via test inversion gives you a subset of $\Theta$ where we know the probability that $\theta$ is in this subset. This seems contradictory.

Does anyone have any intuition (or can correct a mistake somewhere) for how to reconcile this understanding of hypothesis tests with valid confidence intervals from test inversion?

Your interpretation of hypothesis testing ("their results and corresponding p-values tell you nothing about the probability that the null-hypothesis is true") is correct. Your confusion lies in your description of the nature of a (frequentist) confidence interval. The description is reasonable, as far as it goes, but there is a subtle ambiguity that is leading you astray. Namely, the probability in the definition of the confidence interval pertains to the long-run frequency of confidence intervals that cover the true value of $\theta$. It is not the probability that any given confidence interval does. A formed confidence interval is a realization (a set of realized values). It either includes the true value of $\theta$ (with probability $1$) or it doesn't (in which case the probability is $0$), but you don't know which. It may help to read, Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?