Many sources suggest that there is a duality between confidence intervals and hypothesis testing.(*) But I'm having trouble making sense of this philosophically. The frequentist interpretation of a confidence interval is something like (per wikipedia):

Were this procedure to be repeated on multiple samples, the calculated confidence interval (which would differ for each sample) would encompass the true population parameter 90% of the time.

Yet the p-value is defined in terms of values the sample mean might take on if the null hypothesis is true. (I.e. in the one-tailed case: $p = P(\bar x\ge \bar x_{observed}\mid\mu = H_0)$).

How is it possible to manipulate a statement about a procedure that is likely to correctly bound the true population mean into a statement about the probability of the observed sample mean?

If we understand the confidence interval as characterizing the distribution of the means of samples from a population (a view the bootstrap procedure invites), then there's no problem. There's an obvious symmetry between the case in which there is a < 5% chance of the sample mean being more extreme than $H_0$, given the actual population (i.e. $H_0$ outside of the 95% CI) and the case in which there is a < 5% chance of getting a sample mean as extreme as was observed, given that the population is really centered at $H_0$ (i.e. $p<0.05$).

But this interpretation of CIs seems to be disfavored! In particular, the wikipedia article admonishes: "A confidence interval is not a range of plausible values for the sample mean, though it may be understood as an estimate of plausible values for the population parameter."

Even if the CI is in fact a range of plausible values of the sample mean, a question remains. How precisely is such a definition equivalent to the frequentist procedure definition above?

(*) A good example is this Minitab blog post:

The confidence level is equivalent to 1 – the alpha level. So, if your significance level is 0.05, the corresponding confidence level is 95%.

  • If the P value is less than your significance (alpha) level, the hypothesis test is statistically significant.
  • If the confidence interval does not contain the null hypothesis value, the results are statistically significant.
  • If the P value is less than alpha, the confidence interval will not contain the null hypothesis value.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.