Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I have read about controversies regarding hypothesis testing with some commentators suggesting that hypothesis testing should not be used. Some commentators suggest that confidence intervals should be used instead.
You can use a confidence interval (CI) for hypothesis testing. In the typical case, if the CI for an effect does not span 0 then you can reject the null hypothesis. But a CI can be used for more, whereas reporting whether it has been passed is the limit of the usefulness of a test.
The reason you're recommended to use CI instead of just a t-test, for example, is because then you can do more than just test hypotheses. You can make a statement about the range of effects you believe to be likely (the ones in the CI). You can't do that with just a t-test. You can also use it to make statements about the null, which you can't do with a t-test. If the t-test doesn't reject the null then you just say that you can't reject the null, which isn't saying much. But if you have a narrow confidence interval around the null then you can suggest that the null, or a value close to it, is likely the true value and suggest the effect of the treatment, or independent variable, is too small to be meaningful (or that your experiment doesn't have enough power and precision to detect an effect important to you because the CI includes both that effect and 0).
Added Later: I really should have said that, while you can use a CI like a test it isn't one. It's an estimate of a range where you think the parameter values lies. You can make test like inferences but you're just so much better off never talking about it that way.
Which is better?
A) The effect is 0.6, t(29) = 2.8, p < 0.05. This statistically significant effect is... (some discussion ensues about this statistical significance without any mention of or even strong ability to discuss the practical implication of the magnitude of the finding... under a Neyman-Pearson framework the magnitude of the t and p values is pretty much meaningless and all you can discuss is whether the effect is present or isn't found to be present. You can never really talk about there not actually being an effect based on the test.)
or
B) Using a 95% confidence interval I estimate the effect to be between 0.2 and 1.0. (some discussion ensues talking about the actual effect of interest, whether it's plausible values are ones that have any particular meaning and any use of the word significant for exactly what it's supposed to mean. In addition, the width of the CI can go directly to a discussion of whether this is a strong finding or whether you can only reach a more tentative conclusion)
If you took a basic statistics class you might initially gravitate toward A. And there may be some cases where it is a better way to report a result. But for most work B is by far and away superior. A range estimate is not a test.
There is an equivalence between hypothesis tests and confidence intervals. (see e.g. http://en.wikipedia.org/wiki/Confidence_interval#Statistical_hypothesis_testing) I'll give a very specific example. Suppose we have sample $x_1, x_2, \ldots, x_n$ from a normal distribution with mean $\mu$ and variance 1, which we'll write as $\mathcal N(\mu,1)$. Suppose we think that $\mu = m$, and we want to test the null-hypothesis $H_0: \mu = m$, at level $0.05.$ So we make a test statistic, which in this case we will take to be the sample average: $v = (x_1 + x_2 + \cdots + x_n ) / n$. Now suppose $A(m)$ is the "acceptance region" for $v$ for this test. That means that $A(m)$ is the set of possible values of $v$ for which the null-hypothesis $\mu=m$ is accepted at level 0.05 (I use "accepted" as a shorthand for "not rejected" -- I am not suggesting that you would conclude the null hypothesis is true.). For this example, we can look at the $\mathcal N(m,1)$ normal distribution and choose any set that has probability at least 0.95 under this distribution. Now, a 95% confidence region for $\mu$ is the set of all $m$ for which $v$ is in $A(m)$. In other words, it is the set of all $m$ for which the null-hypothesis would be accepted for the observed $v$. That's why John says "If the CI for an effect does not span $0$ then you can reject the null hypothesis." (John is referring to the case of testing $\mu = 0$.)
A related topic is the p-value. The p-value is the smallest level for a test at which we would reject the null-hypothesis. To tie it in with the discussion of confidence intervals, suppose we get a particular sample average $v$, from which we construct confidence intervals of different sizes. Suppose a 95% confidence interval for $\mu$ does not contain $m$. Then we can reject the null-hypothesis $\mu=m$ at level $0.05.$ Then suppose we grow the confidence interval until it just touches (but doesn't include) the value $m$, and suppose this is a 98% confidence interval. Then the p-value for the hypothesis $\mu=m$ is $0.02$ (which we get from $1-0.98$).
'Student' argued for confidence intervals on the grounds that they could show which effects were more important as well as which were more significant.
For example, if you found two effects where the first had a confidence interval for its financial impact from £5 to £6, while the second had a confidence interval from £200 to £2800. The first is more statistically significant but the second is probably more important.
