4
$\begingroup$

Given that what a p-value tells us is the probability of observing the data assuming the null is true, should we ever use null hypothesis significance testing (NHST) when we would rather accept the null hypothesis?

Take the Shapiro-Wilk test through which we might want to establish the normality of our data. Given a p-value of .36 and a conventional $\alpha$ of .05, we would fail to reject the null hypothesis. And the typical follow up is the suggestion that our data are normal.

However, the specific interpretation of $p=.36$ is: assuming our data are normal, the probability of observing these data is 36%. Considering this, it does not seem like good practice to act like failure to reject the null is equivalent to accepting the null. Am I correct?

Absence of evidence is not evidence of absence. Why then is the NHST framework often used when we would want to accept the null?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ You are right, they are not equivalent and "fail to reject the null hypothesis" is preferable (as noted in other related questions -- see the right pane). If a researcher uses "accept the null hypothesis" in NHST framework, I think it is an unfortunate misunderstanding or bad habit. $\endgroup$
    – T.E.G.
    Jul 12 '17 at 4:40
  • $\begingroup$ But there are times when we want to accept the null hypothesis such as in tests for multivariate normality, not just fail to reject. Because absence of evidence - fail to reject - is not evidence of absence - accept the null. $\endgroup$
    – Heteroskedastic Jim
    Jul 13 '17 at 1:15
  • $\begingroup$ I don't think in tests like the one you gave as an example (Shapiro-Wilk test) the null hypothesis can be accepted (see this answer). Maybe the answers to this question would provide some different perspectives. $\endgroup$
    – T.E.G.
    Jul 13 '17 at 2:01
  • $\begingroup$ I believe I know this already. The Shapiro-Wilk cannot in any way be used to accept a null hypothesis. But I guess my question is why do prominent statisticians create such tests, when what we really want to do is accept the null hypothesis? $\endgroup$
    – Heteroskedastic Jim
    Jul 13 '17 at 2:03
3
$\begingroup$

You are right that a lack of (sufficient) evidence of a deviation from the null hypothesis is not necessarily evidence for a lack of a deviation. As already noted one fails to reject the null hypothesis instead of accepting it. In fact the procedure you describe (use of test with normality assumption or rank test depending on a test of normality of regression residuals) is known to lead to tests that do not have the nominal significance level.

The standard solution is to make the null hypothesis the alternative - although that can be difficult. E.g. for the point null treatment A compared to treatment B has no effect on blood pressure (versus the alternative that there is a difference), an "inverted" alternative hypothesis for showing that the treatments are equivalent might be "the absolute difference in blood pressure between treatments A and B is at most 5 mmHg" (with the null being that the difference is either smaller than -5 or greater than 5). To come back yo you example, an irrelevant deviation from normality within some broad general class of distributions is a lot harder to define.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I guess my question is why would a great mathematician like Kolmogorov, or any other great statistician like Mardia, lead us to see a p-value greater than some alpha level as evidence of normality? $\endgroup$
    – Heteroskedastic Jim
    Jul 13 '17 at 1:21
  • 1
    $\begingroup$ @user162986 Is there evidence that they actually did so? If so, I might speculate about a "normality is an assumption, if it's not fulfilled, then we can't use it"-mindset. That might feel very natural when you come from a assumptions-theorem -proof background. And I am not sure when exactly applied statisticians first realized that mild deviations don't matter (while really severe ones do). $\endgroup$
    – Björn
    Jul 19 '18 at 5:47
1
$\begingroup$

First, the p-value is not

the probability of observing the data assuming the null is true,

it is the probability of getting a test statistic at least as extreme as the one we got, given that the null is true.

Second, for normality, there really aren't good tests for using the NHST, regardless of what we want to do. It's much better to use graphical methods. Or, if the reason you are testing normality is to see whether some statistical model is appropriate, then you can do the version that assumes normality and one that doesn't and compare results.

Third, more generally, there are tests of equivalence, tests of superiority and tests of noninferiority.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Would you mind expanding on the difference between the quoted definition and yours? Is it mainly a point of not implying knowledge of the data-generating process or sampling process? $\endgroup$
    – Kuku
    Oct 28 '20 at 19:45
0
$\begingroup$

Yes, NHST can confirm the null hypothesis. An almost sure hypothesis test, using a significance level of $n^{-p}$ for $p>1$, will accept the null whenever it is true and reject the null whenever it is false in any sufficiently large sample with probability one.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ Welcome to our site, Mikenaaman. I have investigated the downvotes each of your answers have received and believe they are because you keep posting the same answer. Once is enough! More than that starts looking like an abuse of the site for self-promotion. Please consult our help center for how this site works and please edit your posts accordingly. $\endgroup$
    – whuber ♦
    Aug 8 '19 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.