34
$\begingroup$

I have read about controversies regarding hypothesis testing with some commentators suggesting that hypothesis testing should not be used. Some commentators suggest that confidence intervals should be used instead.

  • What is the difference between confidence intervals and hypothesis testing? Explanation with reference and examples would be appreciated.
$\endgroup$
3
  • 5
    $\begingroup$ I think you wanted to ask why reporting the hypothesis testing results by showing confidence interval is better than just saying something is confirmed or rejected on some p-value level. $\endgroup$
    – user88
    Oct 1, 2011 at 9:53
  • 3
    $\begingroup$ You should consider checking some of your other questions as answered. $\endgroup$
    – Andy W
    Oct 1, 2011 at 13:13
  • $\begingroup$ In simple words, when we are calculating confidence interval we use the observed proportion while when we are calculating the hypothesis test we use the expected mean. $\endgroup$ Aug 24, 2020 at 16:11

3 Answers 3

24
$\begingroup$

You can use a confidence interval (CI) for hypothesis testing. In the typical case, if the CI for an effect does not span 0 then you can reject the null hypothesis. But a CI can be used for more, whereas reporting whether it has been passed is the limit of the usefulness of a test.

The reason you're recommended to use CI instead of just a t-test, for example, is because then you can do more than just test hypotheses. You can make a statement about the range of effects you believe to be likely (the ones in the CI). You can't do that with just a t-test. You can also use it to make statements about the null, which you can't do with a t-test. If the t-test doesn't reject the null then you just say that you can't reject the null, which isn't saying much. But if you have a narrow confidence interval around the null then you can suggest that the null, or a value close to it, is likely the true value and suggest the effect of the treatment, or independent variable, is too small to be meaningful (or that your experiment doesn't have enough power and precision to detect an effect important to you because the CI includes both that effect and 0).

Added Later: I really should have said that, while you can use a CI like a test it isn't one. It's an estimate of a range where you think the parameter values lies. You can make test like inferences but you're just so much better off never talking about it that way.

Which is better?

A) The effect is 0.6, t(29) = 2.8, p < 0.05. This statistically significant effect is... (some discussion ensues about this statistical significance without any mention of or even strong ability to discuss the practical implication of the magnitude of the finding... under a Neyman-Pearson framework the magnitude of the t and p values is pretty much meaningless and all you can discuss is whether the effect is present or isn't found to be present. You can never really talk about there not actually being an effect based on the test.)

or

B) Using a 95% confidence interval I estimate the effect to be between 0.2 and 1.0. (some discussion ensues talking about the actual effect of interest, whether it's plausible values are ones that have any particular meaning and any use of the word significant for exactly what it's supposed to mean. In addition, the width of the CI can go directly to a discussion of whether this is a strong finding or whether you can only reach a more tentative conclusion)

If you took a basic statistics class you might initially gravitate toward A. And there may be some cases where it is a better way to report a result. But for most work B is by far and away superior. A range estimate is not a test.

$\endgroup$
7
  • $\begingroup$ One addition to @john 's comments: First, sometimes the key question is whether the CI spans 1, not 0 (e.g. logistic regression). $\endgroup$
    – Peter Flom
    Oct 1, 2011 at 9:23
  • $\begingroup$ Guys, it is 1 or is it 0? (This looks very illuminating to me, so I guess, I need to learn the correct value to look out for!) @John $\endgroup$ Oct 1, 2011 at 12:06
  • $\begingroup$ What is the relation between 95% CI and two-tailed testing hypothesis with alfa=0.05? are they same?If not then how? $\endgroup$
    – love-stats
    Oct 1, 2011 at 13:49
  • $\begingroup$ love-stats, when used the same they are the same. $\endgroup$
    – John
    Oct 1, 2011 at 14:51
  • $\begingroup$ Adhesh Josh, the null hypothesis can be any fixed value specified beforehand. That's another feature of the CI over straight NHST. It's very easy to use when you want to test against a hypothetical value other than 0. $\endgroup$
    – John
    Oct 1, 2011 at 14:53
8
$\begingroup$

There is an equivalence between hypothesis tests and confidence intervals. (see e.g. http://en.wikipedia.org/wiki/Confidence_interval#Statistical_hypothesis_testing) I'll give a very specific example. Suppose we have sample $x_1, x_2, \ldots, x_n$ from a normal distribution with mean $\mu$ and variance 1, which we'll write as $\mathcal N(\mu,1)$. Suppose we think that $\mu = m$, and we want to test the null-hypothesis $H_0: \mu = m$, at level $0.05.$ So we make a test statistic, which in this case we will take to be the sample average: $v = (x_1 + x_2 + \cdots + x_n ) / n$. Now suppose $A(m)$ is the "acceptance region" for $v$ for this test. That means that $A(m)$ is the set of possible values of $v$ for which the null-hypothesis $\mu=m$ is accepted at level 0.05 (I use "accepted" as a shorthand for "not rejected" -- I am not suggesting that you would conclude the null hypothesis is true.). For this example, we can look at the $\mathcal N(m,1)$ normal distribution and choose any set that has probability at least 0.95 under this distribution. Now, a 95% confidence region for $\mu$ is the set of all $m$ for which $v$ is in $A(m)$. In other words, it is the set of all $m$ for which the null-hypothesis would be accepted for the observed $v$. That's why John says "If the CI for an effect does not span $0$ then you can reject the null hypothesis." (John is referring to the case of testing $\mu = 0$.)

A related topic is the p-value. The p-value is the smallest level for a test at which we would reject the null-hypothesis. To tie it in with the discussion of confidence intervals, suppose we get a particular sample average $v$, from which we construct confidence intervals of different sizes. Suppose a 95% confidence interval for $\mu$ does not contain $m$. Then we can reject the null-hypothesis $\mu=m$ at level $0.05.$ Then suppose we grow the confidence interval until it just touches (but doesn't include) the value $m$, and suppose this is a 98% confidence interval. Then the p-value for the hypothesis $\mu=m$ is $0.02$ (which we get from $1-0.98$).

$\endgroup$
2
  • $\begingroup$ Please read this as p-value can't be interpreted as smallest level of test to reject null. "It has already been shown that interpreting p values in single (or ongoing) experiments is not permissible in a Neyman–Pearson hypothesis testing context. The calculation of a p value depends only on the truth of the null hypothesis. The p value does not measure the amount of evidence supporting HA; it is a measure of inductive evidence against H0." 'Source : ftp.stat.duke.edu/WorkingPapers/03-26.pdf $\endgroup$
    – sree22
    Apr 13, 2018 at 6:51
  • $\begingroup$ @sree22 can you expand on this, or suggest a rewording? I was trying to give a definition of p-value in this context, not an interpretation. $\endgroup$
    – DavidR
    Jun 17, 2019 at 15:54
3
$\begingroup$

'Student' argued for confidence intervals on the grounds that they could show which effects were more important as well as which were more significant.

For example, if you found two effects where the first had a confidence interval for its financial impact from £5 to £6, while the second had a confidence interval from £200 to £2800. The first is more statistically significant but the second is probably more important.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.