Does "failing to reject" the null hypothesis entails rejecting the alternative one?
I think rigorously thinking, we can't, unless alpha is very high.
$\begingroup$
$\endgroup$
2
-
$\begingroup$ This is also related stats.stackexchange.com/questions/125541/… $\endgroup$– Sextus EmpiricusDec 19, 2020 at 19:22
-
$\begingroup$ also stats.stackexchange.com/questions/60670 and stats.stackexchange.com/questions/85903 $\endgroup$– Sextus EmpiricusDec 19, 2020 at 19:24
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
In statistics there are two types of errors:
- Type I: when the null hypothesis is correct. If in this case we reject null, we make this error.
- Type II: when the alternative is correct. If in this case we fail to reject null, we make this error.
A type I error is connected to statistical significance. a type II error is connected to statistical power.
Many frequentists remember about significance, and forget about power. This leads to the situation, that they state, that failing to reject null means accepting null - IT IS WRONG. The true statement is failing to reject null means that we do not know anything. Unless of course we have knowledge about the power.
Let’s imagine an example, that we have a test with 5% significance, but also very low power - let’s say 10%. We failed to reject null. So now, a false positive (making an error of type I) is not our concern. Now we wish to think if we should accept the null (reject alternative), and without knowledge about the power of the test we can do nothing. But if we know the power of this test, which is 10%, we know that when the alternative is true, the test will correctly reject null only in 10% of cases - in 90% of cases where the alternative is correct, we will fail to reject null!
The problem with power is that in most cases it is a function of many aspects connected to the test itself, the sample size, unknown parameters, satisfaction of test assumptions, and probably more. In most cases it can not be calculated directly, and is approximated by Monte Carlo simulations. But every time those conditions change, the power is completely different.
For some more information about this problem, read Valentin et al. (2019) - a short, popular science, article in Nature, which describes the issue in a more elaborate way. For those more curious I'd suggest taking a look at Wasserstein and Lazar (2016) - the original ASA statement.
Amrhein, Valentin, Sander Greenland, and Blake McShane. "Scientists rise up against statistical significance." (2019): 305-307.
Wasserstein, Ronald L., and Nicole A. Lazar. "The ASA statement on p-values: context, process, and purpose." (2016): 129-133.
-
2$\begingroup$ I spent too long trying to write a good answer to the question with same arguments, this explains it very clearly $\endgroup$– RaphaelSDec 18, 2020 at 14:28