What conclusions can we draw if $p>\alpha$? Does not rejecting the $H_0$ mean anything?
Statistical hypothesis testing is in some way similar to the technique 'proof by contradiction' in mathematics, i.e. if you want to prove something then assume the opposite and derive a contradiction, i.e. something that is impossible.
In statistics 'impossible' does not exist, but some events are very 'improbable'. So in statistics, if you want to 'prove' something (i.e. $H_1$) then you assume the opposite (i.e. $H_0$) and if $H_0$ is true you try to derive something improbable. 'Improbable' is defined by the confidence level that you choose.
If, assuming $H_0$ is true, you can find something very improbable, then $H_0$ can not be true because it leads to a 'statistical contradiction'. Therefore $H_1$ must be true.
This implies that in statistical hypothesis testing you can only find evidence for $H_1$. If one can not reject $H_0$ then the only conclusion you can draw is 'We can not prove $H_1$' or 'we do not find evidence that $H_0$ is false and so we accept $H_0$ (as long as we do not find evidence against it)'.
But there is more ... it is also about power.
Obviously, as nothing is impossible, one can draw wrong conclusions; we might find 'false evidence' for $H_1$ meaning that we conclude that $H_0$ is false while in reality it is true. This is a type I error and the probability of making a type I error is equal to the signficance level that you have choosen. One may also accept $H_0$ while in reality it is false, this is a type II error and the probability of making one is denoted by $\beta$. The power of the test is defined as $1-\beta$ so 1 minus the probability of making a type II error. This is the same as the probability of not making a type II error.
So $\beta$ is the probability of accepting $H_0$ when $H_0$ is false, therefore $1-\beta$ is the probability of rejecting $H_0$ when $H_0$ is false which is the same as the probability of rejecting $H_0$ when $H_1$ is true.
By the above, rejecting $H_0$ is finding evidence for $H_1$, so the power is $1-\beta$ is the probability of finding evidence for $H_1$ when $H_1$ is true.
If you have a test with very high power (close to 1), then this means that if H1 is true, the test would have found evidence for $H_1$ (almost surely) so if we do not find evidence for $H_1$ (i.e. we do not reject $H_0$) and the test has a very high power, then probably $H_1$ is not true (and thus probably $H_0$ is true).
So what we can say is that if your test has very high power , then not rejecting H0 is ''almost as good as'' finding evidence for $H_0$.
For instance, I'm testing my series for the unit-root, maybe with ADF test. Null in this case means the presence of unit root. Failing to reject null suggests that there might be a unit root in the series. The consequence is that I might have to go with modeling the series with random walk like process instead of autorgressive.
So, although it doesn't mean that I proved unit root's presence, the test outcome is not inconsequential. It steers me towards different kind of modeling than rejecting the null.
Hence, in practice failing to reject often means implicitly accepting it. If you're purist then you'd also have the alternative hypothesis of autoregressive, and accept it when failing to reject null.
If we fail to reject the null hypothesis, it does not mean that the null hypothesis is true. That's because a hypothesis test does not determine which hypothesis is true, or even which one is very much more likely. What it does assess is whether the evidence available is statistically significant enough to to reject the null hypothesis.
- The data doesn't provide statistically significant evidence in the difference of the means, but it doesn't conclude that it actually is the mean we define in $H_0$.
- We don't have strength of evidence against the mean being different, but the same as part 1. Therefore we can't make finite conclusions on the mean.
Check out this link for more info on P values and Significance tests Click here