Given that what a p-value tells us is the probability of observing the data assuming the null is true, should we ever use null hypothesis significance testing (NHST) when we would rather accept the null hypothesis?

Take the Shapiro-Wilk test through which we might want to establish the normality of our data. Given a p-value of .36 and a conventional $\alpha$ of .05, we would fail to reject the null hypothesis. And the typical follow up is the suggestion that our data are normal.

However, the specific interpretation of $p=.36$ is: assuming our data are normal, the probability of observing these data is 36%. Considering this, it does not seem like good practice to act like failure to reject the null is equivalent to accepting the null. Am I correct?

Absence of evidence is not evidence of absence. Why then is the NHST framework often used when we would want to accept the null?

  • $\begingroup$ You are right, they are not equivalent and "fail to reject the null hypothesis" is preferable (as noted in other related questions -- see the right pane). If a researcher uses "accept the null hypothesis" in NHST framework, I think it is an unfortunate misunderstanding or bad habit. $\endgroup$ – T.E.G. Jul 12 '17 at 4:40
  • $\begingroup$ But there are times when we want to accept the null hypothesis such as in tests for multivariate normality, not just fail to reject. Because absence of evidence - fail to reject - is not evidence of absence - accept the null. $\endgroup$ – Heteroskedastic Jim Jul 13 '17 at 1:15
  • $\begingroup$ I don't think in tests like the one you gave as an example (Shapiro-Wilk test) the null hypothesis can be accepted (see this answer). Maybe the answers to this question would provide some different perspectives. $\endgroup$ – T.E.G. Jul 13 '17 at 2:01
  • $\begingroup$ I believe I know this already. The Shapiro-Wilk cannot in any way be used to accept a null hypothesis. But I guess my question is why do prominent statisticians create such tests, when what we really want to do is accept the null hypothesis? $\endgroup$ – Heteroskedastic Jim Jul 13 '17 at 2:03

You are right that a lack of (sufficient) evidence of a deviation from the null hypothesis is not necessarily evidence for a lack of a deviation. As already noted one fails to reject the null hypothesis instead of accepting it. In fact the procedure you describe (use of test with normality assumption or rank test depending on a test of normality of regression residuals) is known to lead to tests that do not have the nominal significance level.

The standard solution is to make the null hypothesis the alternative - although that can be difficult. E.g. for the point null treatment A compared to treatment B has no effect on blood pressure (versus the alternative that there is a difference), an "inverted" alternative hypothesis for showing that the treatments are equivalent might be "the absolute difference in blood pressure between treatments A and B is at most 5 mmHg" (with the null being that the difference is either smaller than -5 or greater than 5). To come back yo you example, an irrelevant deviation from normality within some broad general class of distributions is a lot harder to define.

  • $\begingroup$ I guess my question is why would a great mathematician like Kolmogorov, or any other great statistician like Mardia, lead us to see a p-value greater than some alpha level as evidence of normality? $\endgroup$ – Heteroskedastic Jim Jul 13 '17 at 1:21
  • 1
    $\begingroup$ @user162986 Is there evidence that they actually did so? If so, I might speculate about a "normality is an assumption, if it's not fulfilled, then we can't use it"-mindset. That might feel very natural when you come from a assumptions-theorem -proof background. And I am not sure when exactly applied statisticians first realized that mild deviations don't matter (while really severe ones do). $\endgroup$ – Björn Jul 19 '18 at 5:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.