# The rationale behind the "fail to reject the null" jargon in hypothesis testing?

In hypothesis testing with a Null and the Alternative hypothesis (assuming that these two cases are mutually inclusive of all the possible "truths"), we usually base our hypothesis selection criterion on a constraint put on type I error, and choose to say "could not reject the null" as opposed to accepting it, but we say that we "reject the null" in the complimentary case. The reason told to me is this:

Since we usually focus on, and constraint via our selection procedure only the type I error, and do not try to bound the type II error, we do not know how likely is the null if the decision is to go with the Null since we did not know what was the chance of our decision criteria to go with the null if the alternative were true. But if the decision is to go with the alternative, we can reject the null since we set alpha to be very low already.

But this seems fallacious. The probability of selecting the alternative given the Null is the truth may be low, but the corresponding probability given the alternative may be even lower! (In which case our rejecting of the Null was misplaced).

While I agree that the general criteria chosen makes it more likely to go with the alternative given the alternative is the truth than given Null is the truth, the absence of quantifiable information which prevents us from selecting the null should also prevent our rejecting it. To add, since the conditional probability of selecting the Null given the truth is the Null hypothesis is 1 - probability of selecting the alternative given the null is the truth. Hence, by constraining the type I error, we also bound the former to be greater than a particular value.

If type I error is the only one that is constrained, shouldn't not rejecting it be equivalent to accepting the null within the same level of "confidence" as rejecting it had?

I have been doing the excellent course on Mathematical Biostatistics by Hopkins School of Biostats on Coursera, where I came across the above.

• @FransRodenburg Edited. But I did not quite get how the "lack of evidence against is not evidence for" applies to my question? Nov 10, 2017 at 11:55
• Your main question is whether 'not rejecting the null' is equivalent to 'accepting the null with the same level of confidence'. Any such statement of confidence suggests that there is evidence for the null. Nov 10, 2017 at 12:23
• @FransRodenburg in the context of these two hypothesis being mutually inclusive of all possible truths. Can you maybe explain in such a case? Nov 10, 2017 at 12:30
• Maybe this helps: stats.stackexchange.com/questions/163957/…
– user83346
Nov 11, 2017 at 7:22

I think the real reason for this lies deeper. It is a philosophical (precisely, an epistemological) reason, not one constrained by mathematics.

Hypothesis testing is an asymmetric affair in the way you rightly describe it because the person who makes a claim has the burden of proof. (Burden of justification might be a more precise expression since in statistics we are not going to get proof with certainty.) There is a non-arbitrary default position reflected by the null hypothesis. This default position can be rejected, but doesn't have to be proven to remain in it as long as it hasn't been rejected.

This asymmetric setup is also not limited to quantitative analysis. In court for example, innocence is the default assumption that has no burden of proof but guilt must be proven by rejecting that null hypothesis.

Thus the choice of the null hypothesis is not as arbitrary as the mathematics may make us believe. What we put in the null hypothesis corresponds to "no effect" and has no burden of proof. We could mathematically put $$H_0\quad \mu=\gamma$$ and $$H_1\quad \mu\neq\gamma$$ for any given $\gamma$, but only specific null hypotheses make philosophical sense.

This doesn't mean either that $H_0$ must always mathematically be expressed as $\mu=0$. In many cases it is, for example when comparing two means, $H_0$ will be $\mu_{\text{difference}}=0$. When assessing a classifier through a ROC curve for example though, $H_0$ would be $AUC=0.5$ which corresponds to a useless classifier ($AUC=0$ would be a classifier that is perfectly actively harmful). The important point is, that it would be unjustified for philosophical reasons to switch these numbers and have $\mu_{\text{difference}}=0.5$ and $AUC=0$ respectively as null hypotheses.

Mathematically this is doable, but why would you single out one specific difference of means $0.5$ as the null hypothesis that has no burden of proof and posit that all other values (positive or negative or 0, larger or smaller in absolute value) have a burden of proof. Also, rejecting $H_0$ would be close to useless because even after rejecting and observing a smaller mean than 0.5, you still couldn't tell for example whether the value was between 0.5 and 0, exactly 0 or negative. When rejecting $H_0\quad \mu=0$ and looking at the sign of the sample mean, you know if it is a positive or a negative effect.

Furthermore, even due to only the mathematics, the choice of the null hypothesis is already not entirely arbitrary. For example, you couldn't just switch the null and the alternative hypothesis and have $$H_0\quad \mu\neq\gamma$$ and $$H_1\quad \mu=\gamma$$

There are practical applications where this would be in order and the options are still collectively exhaustive, but the mathematics restrain it nevertheless. For example when testing new vaccinations, you cannot have the usual null hypothesis that the vaccination has no medicinal effect and test it against a placebo group. This has obvious ethical problems: you would not be vaccinating your placebo group, then exposing them to the illness on purpose and observe if they survive.

To avoid this ethical problem, we test new vaccinations against other state of the art vaccinations. This deals with one ethical problem, but introduces another one: We cannot use $H_0 \quad \mu_{\text{difference}}=0$. We would have a conflict of interest if we did since the null hypothesis would reflect that the new vaccination is already as good as the state of the art. We cannot just assume that as the default, we need to establish that. If we had this null hypothesis, the researcher would be rewarded for not collecting data since the lower the power of his experiment, the more likely he is to not reject the null which is all he wants to do anyway.

Therefore, equivalence testing was invented where you posit a certain threshold of equivalence tolerance $\delta$ and test for example with two one-sided tests whether the average difference of treatments is at the same time larger than $-\delta$ and smaller than $\delta$, thus equivalent. This detour allows us to test the inverted null and alternative hypotheses and avoids the researcher conflict of interest of wanting to prove the null hypothesis.

Some people say that equivalence testing shows how arbitrary the choice of the null is. I say it shows the exact opposite: how non-arbitrary the choice is. In cases like vaccination where for other ethical reasons the concepts of "mathematical absence of effect" ($\mu_{\text{difference}}=0$) and "philosophical absence of effect" (no medicinal effect above placebo level) don't align, we are forced to make this detour and accept a $\delta$ parameter (which is a horrible researcher degree of freedom) precisely because the choice of the null is non arbitrary.

• A stock example from philosophy of science, as you are likely to be aware, underlines that while observing white swans confirms a hypothesis that all swans are white, observing a single black swan refutes that hypothesis. Hence counter-examples can be decisive. The example and others can be taken further by underlining that the observation of a black swan might be itself in error, or regarded as irrelevant, and so forth. But the emphasis that rejection and acceptance are not symmetric acts remains. I was going to write an answer along these lines (much less developed) but it is not needed! Nov 10, 2017 at 12:44
• I think this is also an possibility: stats.stackexchange.com/questions/163957/…
– user83346
Nov 11, 2017 at 7:23
• Thanks for the answer. To clarify/summarize, mathematically, in the absence of knowing the power, we are as sure about rejecting the null when the outcome selects the alternative as we are about accepting it in the other case. But the jargon of "failing to reject" just represents the philosophy behind choosing the null? Nov 11, 2017 at 14:06
• Well there is a good case to be made for power analysis for that reason and others. (With low power, you also overestimate your effect size int hose cases where you can reject.) But yes, the jargon reflects the philosophical asymmetry of this testing setup. Nov 11, 2017 at 14:15

The Answer by David Ernst is really good.

However, a very short answer is that especially with a point null hypothesis such as $H_0: \mu=0$ for the mean of a normal distribution, it is essentially impossible to exclude the possibility that $\mu$ is just the tiniest little bit away from zero no matter how much data you collect that is perfectly in line with $\mu$ being $0$.

On the other hand, your data can easily be such that it is extremely unlikely to have arisen under $H_0$, in which case you would reject it in favor of $H_A:\mu \neq 0$.

• He may also want input from a Bayesian. They don't only criticize the frequentist NHST epistemology for requiring burden of proof for trivial effects. They as a consequence come much closer to the epistemic model the OP had in mind where you really only compare two theories both of which have a burden of proof. Nov 10, 2017 at 15:38