What conclusions can we draw if $p>\alpha$? Does not rejecting the $H_0$ mean anything?
$\begingroup$ Not rejecting the H0 does not automatically mean that H0 is true. $\endgroup$– Anton AndreevJul 30, 2015 at 15:12
Statistical hypothesis testing is in some way similar to the technique 'proof by contradiction' in mathematics, i.e. if you want to prove something then assume the opposite and derive a contradiction, i.e. something that is impossible.
In statistics 'impossible' does not exist, but some events are very 'improbable'. So in statistics, if you want to 'prove' something (i.e. $H_1$) then you assume the opposite (i.e. $H_0$) and if $H_0$ is true you try to derive something improbable. 'Improbable' is defined by the confidence level that you choose.
If, assuming $H_0$ is true, you can find something very improbable, then $H_0$ can not be true because it leads to a 'statistical contradiction'. Therefore $H_1$ must be true.
This implies that in statistical hypothesis testing you can only find evidence for $H_1$. If one can not reject $H_0$ then the only conclusion you can draw is 'We can not prove $H_1$' or 'we do not find evidence that $H_0$ is false and so we accept $H_0$ (as long as we do not find evidence against it)'.
But there is more ... it is also about power.
Obviously, as nothing is impossible, one can draw wrong conclusions; we might find 'false evidence' for $H_1$ meaning that we conclude that $H_0$ is false while in reality it is true. This is a type I error and the probability of making a type I error is equal to the signficance level that you have choosen. One may also accept $H_0$ while in reality it is false, this is a type II error and the probability of making one is denoted by $\beta$. The power of the test is defined as $1-\beta$ so 1 minus the probability of making a type II error. This is the same as the probability of not making a type II error.
So $\beta$ is the probability of accepting $H_0$ when $H_0$ is false, therefore $1-\beta$ is the probability of rejecting $H_0$ when $H_0$ is false which is the same as the probability of rejecting $H_0$ when $H_1$ is true.
By the above, rejecting $H_0$ is finding evidence for $H_1$, so the power is $1-\beta$ is the probability of finding evidence for $H_1$ when $H_1$ is true.
If you have a test with very high power (close to 1), then this means that if H1 is true, the test would have found evidence for $H_1$ (almost surely) so if we do not find evidence for $H_1$ (i.e. we do not reject $H_0$) and the test has a very high power, then probably $H_1$ is not true (and thus probably $H_0$ is true).
So what we can say is that if your test has very high power , then not rejecting H0 is ''almost as good as'' finding evidence for $H_0$.
1$\begingroup$ Good answer. Of course, power depends on effect size, so if you say "your test has very high power" that must refer to some pre-specified effect size. I'd agree that failing to reject H0 in this case may be interpreted as evidence in favor of an "extended H0", namely that the true effect size if smaller than the target effect size for which power was computed. $\endgroup$– A. DondaAug 17, 2015 at 17:40
$\begingroup$ I wouldn't use the term "accept H0" as that may be interpreted as a prove that H0 is true, and actually by failing to reject H0 the only thing we can say is "we don't know whether H1 is true or whether H0 is true" or more accurately "we don't have evidence that supports H1 is likely to be true nor that supports H0 is likely to be false" $\endgroup$– MondKinDec 3, 2015 at 4:25
1$\begingroup$ I like your analogy with proof by contradiction, that made it click for me. $\endgroup$ Sep 19, 2016 at 14:19
$\begingroup$ @jeremy radcliff: I am glad it helped you :-) $\endgroup$– user83346Sep 19, 2016 at 14:26
For instance, I'm testing my series for the unit-root, maybe with ADF test. Null in this case means the presence of unit root. Failing to reject null suggests that there might be a unit root in the series. The consequence is that I might have to go with modeling the series with random walk like process instead of autorgressive.
So, although it doesn't mean that I proved unit root's presence, the test outcome is not inconsequential. It steers me towards different kind of modeling than rejecting the null.
Hence, in practice failing to reject often means implicitly accepting it. If you're purist then you'd also have the alternative hypothesis of autoregressive, and accept it when failing to reject null.
$\begingroup$ +1 for pointing out that failing to reject can lead to accepting the null. In theory we never say that, but in practice, this is exactly what occurs. $\endgroup$– soakleyNov 18, 2015 at 18:24
$\begingroup$ One way to reconcile this is in decision theory. You can assign certain costs to errors, then find optimal decisions based on minimization of costs. $\endgroup$– AksakalNov 18, 2015 at 18:52
$\begingroup$ "Hence, in practice failing to reject often means implicitly accepting it." is it, though? I am surprised there is also another comment actually supporting this statement - what have we come to? :p $\endgroup$– gentedJun 3, 2019 at 13:25
$\begingroup$ @gented, time to throw out inference testing? $\endgroup$– AksakalJun 3, 2019 at 13:51
If we fail to reject the null hypothesis, it does not mean that the null hypothesis is true. That's because a hypothesis test does not determine which hypothesis is true, or even which one is very much more likely. What it does assess is whether the evidence available is statistically significant enough to to reject the null hypothesis.
- The data doesn't provide statistically significant evidence in the difference of the means, but it doesn't conclude that it actually is the mean we define in $H_0$.
- We don't have strength of evidence against the mean being different, but the same as part 1. Therefore we can't make finite conclusions on the mean.
Check out this link for more info on P values and Significance tests Click here