Can we reject a null hypothesis with confidence intervals produced via sampling rather than the null hypothesis?

Question

I have been taught that we can produce a parameter estimate in the form of a confidence interval after sampling from a population. For example, 95% confidence intervals, with no violated assumptions, should have a 95% success rate of containing whatever the true parameter we are estimating is in the population.

I.e,

1. Produce a point estimate from a sample.
2. Produce a range of values that theoretically has a 95% chance of containing the true value we are trying to estimate.

However, when the topic has turned to hypothesis testing, the steps were described as the following:

1. Assume some parameter as the null hypothesis.
2. Produce a probability distribution of the likelihood of getting various point estimates given this null hypothesis is true.
3. Reject the null hypothesis if the point estimate we get would be produced less than 5% of the time if the null hypothesis is true.

My question is this:

Is it necessary to produce our confidence intervals using the null hypothesis in order to reject the null? Why not just do the first procedure and get our estimate for the true parameter (not explicitly using our hypothesized value in calculating the confidence interval) then rejecting the null hypothesis if it does not fall within the this interval?

This seems logically equivalent to me intuitively, but I fear that I am missing something very fundamental since there is probably a reason it is taught this way.

Answer 1

A simple problem, by way of example, is given by testing for the mean of a normal population with known variance $\sigma^2=1$. Then, a pivot - a quantity whose distribution does not depend on the parameter, is given by $\bar{Y}-\mu\sim N(0,1/n)$. Critical values $z_{\alpha/2}$ satisfy, in this symmetric case, $\Phi(-z_{\alpha/2})=\alpha/2$ and $\Phi(z_{\alpha/2})=1-\alpha/2$.

Hence, \begin{eqnarray*} 1-\alpha&=&\Pr\{(\bar{X}-\mu)/(1/\sqrt{n})\in(-z_{\alpha/2},z_{\alpha/2})\}\\ &=&\Pr\{-z_{\alpha/2}\leqslant(\bar{X}-\mu)\sqrt{n}\leqslant z_{\alpha/2}\}\\ &=&\Pr\{z_{\alpha/2}\geqslant(\mu-\bar{X})\sqrt{n}\geqslant -z_{\alpha/2}\}\\ &=&\Pr\{-z_{\alpha/2}/\sqrt{n}\leqslant\mu-\bar{X}\leqslant z_{\alpha/2}/\sqrt{n}\}\\ &=&\Pr\{\bar{X}-z_{\alpha/2}/\sqrt{n}\leqslant\mu\leqslant \bar{X}+z_{\alpha/2}/\sqrt{n}\}\\ &=&\Pr\{(\bar{X}-z_{\alpha/2}/\sqrt{n},\bar{X}+z_{\alpha/2}/\sqrt{n})\ni\mu\} \end{eqnarray*} so that $$ (\bar{X}-z_{\alpha/2}/\sqrt{n},\bar{X}+z_{\alpha/2}/\sqrt{n})$$ is a confidence interval of level $1-\alpha$.

At the same time, the event in first line of the display is precisely also the event that the null hypothesis is not rejected for this $\mu$. Since the rest just contains equivalent reformulations, the c.i. indeed contains all $\mu$ for which the null is not rejected, and no reference to "under the null" is needed.

Here is a plot analogous to Martijn's +1 visualization aiming to show what is known as duality between confidence intervals and tests. $C$ denotes the confidence interval belonging to some $\bar{x}^*$ and $A(\mu_0)$ the acceptance region belonging to some hypothesis $\mu=\mu_0$.

Answer 2

Yes you can replace a hypothesis test (comparing sample with a hypothetical distribution of test outcomes) by a comparison with a confidence interval calculated from the sample. But indirectly a confidence interval is already a sort of hypothesis test, namely:

• You might see the confidence intervals as being constructed as a range of values for which an $\alpha$ level hypothesis test would succeed and outside the range an $\alpha$ level hypothesis test would fail.

The consequence of making such range is that the range only fails a fraction $\alpha$ of the time.

Example

I am using an image from an answer to the below question: Confidence Intervals: how to formally deal with $P(L(\textbf{X}) \leq \theta, U(\textbf{X})\geq\theta) = 1-\alpha$

It is a variation of a graph from Clopper-Pearson. Imagine the case of 100 Bernoulli trials where the probability of success is $\theta$ and we observe the total number of successes $X$.

Note that:

• In the vertical direction you see hypothesis testing. E.g. for a given hypothesized value $\theta$ you reject the hypothesis if the measured $X$ is above or below the red or green dotted lines.

• In the horizontal direction you see Clopper-Pearson confidence intervals. If for any given observation X you use these confidence intervals then you will be wrong only 5% of the time

  (because you will only observe such X, on which you base a 'wrong' interval, 5% of the time)