# Why does rejecting the null in goodness-of-fit tests not imply accepting the null?

From All of Statistics by Wasserman:

Goodness-of-fit testing has some serious limitations. If reject $H_0$ then we conclude we should not use the model. But if we do not reject $H_0$ we cannot conclude that the model is correct. We may have failed to reject simply because the test did not have enough power. This is why it is better to use nonparametric methods whenever possible rather than relying on parametric assumptions.

My questions are:

1. why "if we do not reject $H_0$ we cannot conclude that the model is correct"?

If it is because "We may have failed to reject simply because the test did not have enough power", why "it is better to use nonparametric methods whenever possible rather than relying on parametric assumptions"? Doesn't the same reason and conclusion apply to nonparametric methods?

2. Is it correct that "Goodness-of-fit testing" here mean testing if the distribution of a sample is a specific distribution?

Does the same conclusion "if we do not reject $H_0$ we cannot conclude that the model is correct" also apply to testing if the distribution of a sample and the distribution of another sample are the same, such as z test for two normally distributed groups of samples, and Kolmogorov-Smirnov two-sample test?

Thanks and regards!

• "Are goodness-of-fit tests parametric or nonparametric?" --- you already asked it... why ask it again? – Glen_b May 28 '13 at 3:52
• I wondered if you'd maybe miscopied the text, but that "If reject $H_0$" error is in the original. – Glen_b May 28 '13 at 4:03
• @Glen_b: Thanks! (1) To figure the second part, I think answering the first part will help. But I knew that I had asked it, so I linked to it. (2) The quote is what it is in the book, on p169. Is there something wrong? – Tim May 28 '13 at 10:49
• "The quote is what it is in the book" - yes, I know, I checked; that was what I was saying. Someone reading it is likely - as I did - to think the sentence is mistyped, since it appears not to be correct English. I was confirming that it is exactly as in the original. – Glen_b May 28 '13 at 10:58
• @Glen_b: Thanks for your confirmation! I was wondering What you think about "it is better to use nonparametric methods whenever possible rather than relying on parametric assumptions"? Doesn't the same reason and conclusion applied to goodness-of-fitness tests also apply to nonparametric methods? – Tim May 28 '13 at 11:07

Question 3: that depends on the goodness of fit test. So it is always a good idea to read up on the specific goodness of fit test you want to apply to figure out what exactly the null hypothesis is that is being tested.

Question 2: To understand this you need to see that a goodness of fit test is just like any other statistical test, and understand exactly what the logic is behind statistical tests. The outcome of a statistical test is a $p$-value, which is the probability of finding data that deviates from $H_0$ at least as much as the data you have observed when $H_0$ is true. So it is a thought experiment with the following steps:

1. Assume a population in which $H_0$ is true, that is, your model is correct in some specific sense depending on the goodness of fit test.
2. We draw many samples at random from this population, fit the model, and compute the goodness of fit test in each of these samples.
3. Since you have drawn samples at randome, some of these samples will be "weird", i.e. deviate from $H_0$.
4. The $p$-value is the expected proportion of samples that are "as weird or weirder" than the data you have observed.

If you find data with a small $p$-value then that data is unlikely to have come from a population in which the $H_0$ is true, and the fact that you have observed that data is considered evidence against $H_0$. If the $p$-value is below some pre-defined but arbitrary cut off point $\alpha$ (common values are 5% or 1%), then we call it "siginificant" and reject the $H_0$.

Notice what the opposite, not-significant, means: we have not found enough information to reject $H_0$. This is a case of "absence of evidence", which is not the same thing as "evidence of absence". So, "not rejecting $H_0$" is not the same thing as "accepting $H_0$".

Another way to answer your question would be to ask: "could it be that the $H_0$ is true?" the answer is simply no. In a goodness of fit test, the $H_0$ is that the model is in some sense true. The definition of a model is that it is a simplification of reality and simplification is just an other word for "wrong in some useful way". So models are by definition wrong, and thus the $H_0$ cannot be true.

This has consequences for the statement you quoted: "If reject $H_0$ then we conclude we should not use the model." This is incorrect, all that the significance of a goodness of fit test tells you that your model is likely to be wrong, but you already knew that. The interesting question is whether it is so wrong that it is no longer useful. This is a judgement call. Statistical tests can help you in differentiating between patterns that could just be the result of the randomness that is the result of sampling and "real" patterns. A significant result tells you that the latter is likely to be true, but that is not enough to conclude that the model is not a useful simplification of reality. You now need to investigate what exactly the deviation is, how large that deviation is, and what the concequences are for the performance of your model.

• Thanks! (1) For question 3, I just added two specific tests, such as Z test for two groups of samples, and Kolmogorov-Smirnov two-sample test. What about them? (2) For question 2, why does the quote say that "it is better to use nonparametric methods whenever possible rather than relying on parametric assumptions"? Doesn't the same reason and conclusion applied to goodness-of-fitness tests also apply to nonparametric methods? – Tim May 28 '13 at 11:06
• We never accept $H_0$. The two possible outcomes of a statistical test are accept or fail to accept $H_0$. – Maarten Buis May 28 '13 at 11:18
• As to point (2): That would not be my quote. This quote is more a presentation of a preference of the author for a particular style of doing research. For your concrete application you would have to take into account that many so-called non-parametric test have very low power, so that would represent a real trade-off. – Maarten Buis May 28 '13 at 11:27
• I was wondering if "We never accept H0" and "The two possible outcomes of a statistical test are accept or fail to accept H0" are consistent with each other? Do they mean that we only fail to accept the null? – Tim May 28 '13 at 11:30
• Sorry, the last part should be: we either reject or fail to reject $H_0$. – Maarten Buis May 28 '13 at 11:32