Hypothesis testing terminology surrounding null

Question

Which is more correct?

• "Accept the null hypothesis"; or
• "Do not reject the null hypothesis"

Is there truly a separate meaning that I'm missing here?

Answer 1

With most tests, it is impossible to control both types of errors. The way (nearly) all tests are set up, is like this: We check the probability of a certain outcome (test statistic) or worse, under the assumption that the null hypothesis is true. Now, we will reject this null hypothesis if that probability is very low (typically < 5%), because we do observe that outcome, and the null hypothesis makes that outcome unlikely (the underlying idea is that there is probably some other 'hypothesis' (e.g. some other value of the mean) that makes our outcome more likely).

So, if this probability is low, due to the construction, we can truly say: there is less than 5 % chance that this would happen if the null hypothesis were true, so it makes sense to reject the null hypothesis.

The problem is: what happens if that probability is (for example) 6 %? Now, we don't reject the null hypothesis, because it doesn't fall below the threshold. But on the other hand, we might test 10 similar hypotheses (10 different values of a mean), and they might all return a probability above 5 %. Clearly, it is impossible to say, then, that we 'accept' all of those 10 hypotheses, right? Otherwise, we would be stating that we accept 10 different values for the mean. The fact that we only do 1 test instead of 10, doesn't change this: just because we have an outcome that could very well happen given the null hypothesis, that value could just as well happen with other hypotheses, so we can never 'accept' the null hypothesis. Best we can do, is say that we see no reason to reject it.

Answer 2

A useful article to read is:

Murdock, D, Tsai, Y-L, and Adcock, J (2008) P-Values are Random Variables. The American Statistician. vol. 62, no. 3, 242-245.

This talks about how p-values are random variables which if the null hypothesis is true follow a uniform distribution. This means that you have the exact same chance of getting data that will result in a p-value of 0.9 as you do of a p-value of 0.1 (when the null is true). When the null is not true then the p-values near 0 are more likely than those near 1 (but in some cases not by much).

You can see this by simulation of cases under the null, cases where the null is false but there is very low power (the truth is very close to the null), and cases where the null is very far from the truth (one tool to help with this is the Pvalue.norm.sim function in the TeachingDemos package for R).

We prefer the "Do not reject" term because it better demonstrates that while the null could be true, it is also possible that it is false, we just don't have enough data to prove it.

Answer 3

In many circumstances neither is really the best approach. Consider using the P value as an index of the evidence against the null and make conclusions in the realm of the real experimental hypothesis rather than in the realm of the statistical hypothesis. Usually the former is much more important.