I have been taught that we can produce a parameter estimate in the form of a confidence interval after sampling from a population. For example, 95% confidence intervals, with no violated assumptions, should have a 95% success rate of containing whatever the true parameter we are estimating is in the population.

I.e,

  1. Produce a point estimate from a sample.
  2. Produce a range of values that theoretically has a 95% chance of containing the true value we are trying to estimate.

However, when the topic has turned to hypothesis testing, the steps were described as the following:

  1. Assume some parameter as the null hypothesis.
  2. Produce a probability distribution of the likelihood of getting various point estimates given this null hypothesis is true.
  3. Reject the null hypothesis if the point estimate we get would be produced less than 5% of the time if the null hypothesis is true.

My question is this:

Is it necessary to produce our confidence intervals using the null hypothesis in order to reject the null? Why not just do the first procedure and get our estimate for the true parameter (not explicitly using our hypothesized value in calculating the confidence interval) then rejecting the null hypothesis if it does not fall within the this interval?

This seems logically equivalent to me intuitively, but I fear that I am missing something very fundamental since there is probably a reason it is taught this way.

  • My apologies for being unclear, Martijn. I'll edit my post shortly so that it's clearer to people looking up the same questions in the future. What I meant is we can calculate a parameter estimate from a sample, or we can calculate a range of estimates that we would deem to support the null hypothesis using the null hypothesis. I did not understand why it was necessary to use the null to see if our point estimate was in this interval, rather than simply using our parameter estimate and checking to see if the null was within the bounds of the parameter estimate. I hope that makes sense! – Nikli Jun 14 at 9:29
  • An interesting thought experiment is if someone tries to sell you weighted dice. They roll them, then state that they are weighted in the direction you observe (e.g. 6 comes up 20% of the time). Are they weighted (were enough sample throws done), by how much, and what is it worth to do your own (extra) dice throw tests? The seller and buyer have different goals... – Philip Oakley Jun 14 at 17:05
up vote 4 down vote accepted

A simple problem, by way of example, is given by testing for the mean of a normal population with known variance $\sigma^2=1$. Then, a pivot - a quantity whose distribution does not depend on the parameter, is given by $\bar{Y}-\mu\sim N(0,1/n)$. Critical values $z_{\alpha/2}$ satisfy, in this symmetric case, $\Phi(-z_{\alpha/2})=\alpha/2$ and $\Phi(z_{\alpha/2})=1-\alpha/2$.

Hence, \begin{eqnarray*} 1-\alpha&=&\Pr\{(\bar{X}-\mu)/(1/\sqrt{n})\in(-z_{\alpha/2},z_{\alpha/2})\}\\ &=&\Pr\{-z_{\alpha/2}\leqslant(\bar{X}-\mu)\sqrt{n}\leqslant z_{\alpha/2}\}\\ &=&\Pr\{z_{\alpha/2}\geqslant(\mu-\bar{X})\sqrt{n}\geqslant -z_{\alpha/2}\}\\ &=&\Pr\{-z_{\alpha/2}/\sqrt{n}\leqslant\mu-\bar{X}\leqslant z_{\alpha/2}/\sqrt{n}\}\\ &=&\Pr\{\bar{X}-z_{\alpha/2}/\sqrt{n}\leqslant\mu\leqslant \bar{X}+z_{\alpha/2}/\sqrt{n}\}\\ &=&\Pr\{(\bar{X}-z_{\alpha/2}/\sqrt{n},\bar{X}+z_{\alpha/2}/\sqrt{n})\ni\mu\} \end{eqnarray*} so that $$ (\bar{X}-z_{\alpha/2}/\sqrt{n},\bar{X}+z_{\alpha/2}/\sqrt{n})$$ is a confidence interval of level $1-\alpha$.

At the same time, the event in first line of the display is precisely also the event that the null hypothesis is not rejected for this $\mu$. Since the rest just contains equivalent reformulations, the c.i. indeed contains all $\mu$ for which the null is not rejected, and no reference to "under the null" is needed.

Here is a plot analogous to Martijn's +1 visualization aiming to show what is known as duality between confidence intervals and tests. $C$ denotes the confidence interval belonging to some $\bar{x}^*$ and $A(\mu_0)$ the acceptance region belonging to some hypothesis $\mu=\mu_0$.

enter image description here

Yes you can replace a hypothesis test (comparing sample with a hypothetical distribution of test outcomes) by a comparison with a confidence interval calculated from the sample. But indirectly a confidence interval is already a sort of hypothesis test, namely:

  • You might see the confidence intervals as being constructed as a range of values for which an $\alpha$ level hypothesis test would succeed and outside the range an $\alpha$ level hypothesis test would fail.

The consequence of making such range is that the range only fails a fraction $\alpha$ of the time.

Example

I am using an image from an answer to the below question: Confidence Intervals: how to formally deal with $P(L(\textbf{X}) \leq \theta, U(\textbf{X})\geq\theta) = 1-\alpha$

It is a variation of a graph from Clopper-Pearson. Imagine the case of 100 Bernoulli trials where the probability of success is $\theta$ and we observe the total number of successes $X$.

fiducial probability

Note that:

  • In the vertical direction you see hypothesis testing. E.g. for a given hypothesized value $\theta$ you reject the hypothesis if the measured $X$ is above or below the red or green dotted lines.

  • In the horizontal direction you see Clopper-Pearson confidence intervals. If for any given observation X you use these confidence intervals then you will be wrong only 5% of the time

    (because you will only observe such X, on which you base a 'wrong' interval, 5% of the time)

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.