Why do statisticians say a non-significant result means “you can't reject the null” as opposed to accepting the null hypothesis?

Traditional statistical tests, like the two sample t-test, focus on trying to eliminate the hypothesis that there is no difference between a function of two independent samples. Then, we choose a confidence level and say that if the difference of means is beyond the 95% level, we can reject the null hypothesis. If not, we "can't reject the null hypothesis". This seems to imply that we can't accept it either. Does it mean we're not sure if the null hypothesis is true?

Now, I want to design a test where my hypothesis is that a function of two samples is the same (which is the opposite of traditional statistics tests where the hypothesis is that the two samples are different). So, my null hypothesis becomes that the two samples are different. How should I design such a test? Will it be as simple as saying that if the p-value is lesser than 5% we can accept the hypothesis that there is no significant difference?

-

Traditionally, the null hypothesis is a point value. (It is typically $0$, but can in fact be any point value.) The alternative hypothesis is that the true value is any value other than the null value. Because a continuous variable (such as a mean difference) can take on a value which is indefinitely close to the null value but still not quite equal and thus make the null hypothesis false, a traditional point null hypothesis cannot be proven.

Imagine your null hypothesis is $0$, and the mean difference you observe is $0.01$. Is it reasonable to assume the null hypothesis is true? You don't know yet; it would be helpful to know what our confidence interval looks like. Let's say that your 95% confidence interval is $(-4.99,\ 5.01)$. Now, should we conclude that the true value is $0$? I would not feel comfortable saying that, because the CI is very wide, and there are many, large non-zero values that we might reasonably suspect are consistent with our data. So let's say we gather much, much more data, and now our observed mean difference is $0.01$, but the 95% CI is $(0.005,\ 0.015)$. The observed mean difference has stayed the same (which would be amazing if it really happened), but the confidence interval now excludes the null value. Of course, this is just a thought experiment, but it should make the basic ideas clear. We can never prove that the true value is any particular point value; we can only (possibly) disprove that it is some point value. In statistical hypothesis testing, the fact that the p-value is > 0.05 (and that the 95% CI includes zero) means that we are not sure if the null hypothesis is true.

As for your concrete case, you cannot construct a test where the alternative hypothesis is that the mean difference is $0$ and the null hypothesis is anything other than zero. This violates the logic of hypothesis testing. It is perfectly reasonable that it is your substantive, scientific hypothesis, but it cannot be your alternative hypothesis in a hypothesis testing situation.

So what can you do? In this situation, you use equivalence testing. (You might want to read through some of our threads on this topic by clicking on the tag.) The typical strategy is to use the two one sided tests approach. Very briefly, you select an interval within which you would consider that the true mean difference might as well be $0$ for all you could care, then you perform a one-sided test to determine if the observed value is less than the upper bound of that interval, and another one-sided test to see if it is greater than the lower bound. If both of these tests are significant, then you have rejected the hypothesis that the true value is outside the interval you care about. If one (or both) are non-significant, you fail to reject the hypothesis that the true value is outside the interval.

For example, suppose anything within the interval $(-0.02,\ 0.02)$ is so close to zero that you think it is essentially the same as zero for your purposes, so you use that as your substantive hypothesis. Now imagine that you get the first result described above. Although $0.01$ falls within that interval, you would not be able to reject the null hypothesis on either one-sided t-test, so you would fail to reject the null hypothesis. On the other hand, imagine that you got the second result described above. Now you find that the observed value falls within the designated interval, and it can be shown to be both less than the upper bound and greater than the lower bound, so you can reject the null. (It is worth noting that you can reject both the hypothesis that the true value is $0$, and the hypothesis that the true value lies outside of the interval $(-0.02,\ 0.02)$, which may seem perplexing at first, but is fully consistent with the logic of hypothesis testing.)

-

Consider the case where the null hypothesis is that a coin is 2 headed, i.e. the probability of heads is 1. Now the data is the result of flipping a coin a single time and seeing heads. This results in a p-value of 1.0 which is greater than every reasonable alpha. Does this mean that the coin is 2 headed? it could be, but it could also be a fair coin and we saw heads due to chance (would happen 50% of the time with a fair coin). So the high p-value in this case says that the observed data is perfectly consistent with the null, but it is also consistent with other possibilities.

Just like a "Not Guilty" verdict in court can mean the defendant is innocent, it can also be because the defendant is guilty but there is not enough evidence. The same with the null hypothesis we fail to reject because the null could be true, or it could be we don't have enough evidence to reject even though it is false.

-
I like the "Not guilty" example. Going one step further, re-opening cases based on DNA evidence that we did not know how to use in the past and having some convictions overturned is a perfect example of how adding more data may be all that's needed to have enough evidence. –  Thomas Speidel Feb 10 at 15:21

The null hypothesis is usually taken to be the thing you have reason to assume. Often times it is the "current state of knowledge" that you wish to show is statistically unlikely.

The usual set-up for hypothesis testing is minimize Type I error. That is, minimize the chance that we reject the null hypothesis in favor of the alternative H_1 even though H_0 is true. This is the error we choose to first minimize because we don't want to overturn common knowledge when that common knowledge is indeed true.

You should always design your test bearing in mind that H_0 should be what you expect.

If we have two samples we expect to be identically distributed then our null hypothesis is the samples are the same. If we have two samples that we would expect to be (wildly) different, our null hypothesis is that they are different.

-
And what if we have no expectations.. it might be that we just don't know. Also, how will the decision rule work if we want to reject the hypothesis that the two samples are different? –  ryu576 Feb 8 at 21:12
In the case you have no expectations you want to keep both both types of errors small but this isn't always possible. You need an extra variable (such as increasing sample size) to do it. –  SomeEE Feb 8 at 21:31
Since we can reject the null but not prove it true the null is usually the opposite of what we want to prove or assume to be true. If we believe that there is a difference then the null should be no difference so that you can disprove that. –  Greg Snow Feb 8 at 22:03
@Greg That is a good approach if you know which one you want to be true which is probably the usual case. –  SomeEE Feb 8 at 22:08
"What you expect" and "that they are different" cannot be statistical hypotheses at all because they are not quantitative. That gets to the crux of the matter: the asymmetry in roles between the null and alternative hypotheses derives from the ability to determine the sampling distribution of the test statistic under the null, compared to the need to parameterize the distribution by the effect size under the alternative hypothesis. Nor is it the case the we "minimize Type I error": that never happens (the minimum is always 0). Tests seek a balance between Type I and II error rates. –  whuber Feb 10 at 15:31

Absence of evidence is not evidence of an absence (the title of an Altman, Bland paper on BMJ). P-values only give us evidence of an absence when we consider them significant. Otherwise, they tell us nothing. Hence, absence of evidence. In other words: we don't know and more data may help.

-