The rationale behind the "fail to reject the null" jargon in hypothesis testing?

Question

In hypothesis testing with a Null and the Alternative hypothesis (assuming that these two cases are mutually inclusive of all the possible "truths"), we usually base our hypothesis selection criterion on a constraint put on type I error, and choose to say "could not reject the null" as opposed to accepting it, but we say that we "reject the null" in the complimentary case. The reason told to me is this:

Since we usually focus on, and constraint via our selection procedure only the type I error, and do not try to bound the type II error, we do not know how likely is the null if the decision is to go with the Null since we did not know what was the chance of our decision criteria to go with the null if the alternative were true. But if the decision is to go with the alternative, we can reject the null since we set alpha to be very low already.

But this seems fallacious. The probability of selecting the alternative given the Null is the truth may be low, but the corresponding probability given the alternative may be even lower! (In which case our rejecting of the Null was misplaced).

While I agree that the general criteria chosen makes it more likely to go with the alternative given the alternative is the truth than given Null is the truth, the absence of quantifiable information which prevents us from selecting the null should also prevent our rejecting it. To add, since the conditional probability of selecting the Null given the truth is the Null hypothesis is 1 - probability of selecting the alternative given the null is the truth. Hence, by constraining the type I error, we also bound the former to be greater than a particular value.

If type I error is the only one that is constrained, shouldn't not rejecting it be equivalent to accepting the null within the same level of "confidence" as rejecting it had?

I have been doing the excellent course on Mathematical Biostatistics by Hopkins School of Biostats on Coursera, where I came across the above.

Answer 1

I think the real reason for this lies deeper. It is a philosophical (precisely, an epistemological) reason, not one constrained by mathematics.

Hypothesis testing is an asymmetric affair in the way you rightly describe it because the person who makes a claim has the burden of proof. (Burden of justification might be a more precise expression since in statistics we are not going to get proof with certainty.) There is a non-arbitrary default position reflected by the null hypothesis. This default position can be rejected, but doesn't have to be proven to remain in it as long as it hasn't been rejected.

This asymmetric setup is also not limited to quantitative analysis. In court for example, innocence is the default assumption that has no burden of proof but guilt must be proven by rejecting that null hypothesis.

Thus the choice of the null hypothesis is not as arbitrary as the mathematics may make us believe. What we put in the null hypothesis corresponds to "no effect" and has no burden of proof. We could mathematically put $$H_0\quad \mu=\gamma$$ and $$H_1\quad \mu\neq\gamma$$ for any given $\gamma$, but only specific null hypotheses make philosophical sense.

This doesn't mean either that $H_0$ must always mathematically be expressed as $\mu=0$. In many cases it is, for example when comparing two means, $H_0$ will be $\mu_{\text{difference}}=0$. When assessing a classifier through a ROC curve for example though, $H_0$ would be $AUC=0.5$ which corresponds to a useless classifier ($AUC=0$ would be a classifier that is perfectly actively harmful). The important point is, that it would be unjustified for philosophical reasons to switch these numbers and have $\mu_{\text{difference}}=0.5$ and $AUC=0$ respectively as null hypotheses.

Mathematically this is doable, but why would you single out one specific difference of means $0.5$ as the null hypothesis that has no burden of proof and posit that all other values (positive or negative or 0, larger or smaller in absolute value) have a burden of proof. Also, rejecting $H_0$ would be close to useless because even after rejecting and observing a smaller mean than 0.5, you still couldn't tell for example whether the value was between 0.5 and 0, exactly 0 or negative. When rejecting $H_0\quad \mu=0$ and looking at the sign of the sample mean, you know if it is a positive or a negative effect.

Furthermore, even due to only the mathematics, the choice of the null hypothesis is already not entirely arbitrary. For example, you couldn't just switch the null and the alternative hypothesis and have $$H_0\quad \mu\neq\gamma$$ and $$H_1\quad \mu=\gamma$$

There are practical applications where this would be in order and the options are still collectively exhaustive, but the mathematics restrain it nevertheless. For example when testing new vaccinations, you cannot have the usual null hypothesis that the vaccination has no medicinal effect and test it against a placebo group. This has obvious ethical problems: you would not be vaccinating your placebo group, then exposing them to the illness on purpose and observe if they survive.

To avoid this ethical problem, we test new vaccinations against other state of the art vaccinations. This deals with one ethical problem, but introduces another one: We cannot use $H_0 \quad \mu_{\text{difference}}=0$. We would have a conflict of interest if we did since the null hypothesis would reflect that the new vaccination is already as good as the state of the art. We cannot just assume that as the default, we need to establish that. If we had this null hypothesis, the researcher would be rewarded for not collecting data since the lower the power of his experiment, the more likely he is to not reject the null which is all he wants to do anyway.

Therefore, equivalence testing was invented where you posit a certain threshold of equivalence tolerance $\delta$ and test for example with two one-sided tests whether the average difference of treatments is at the same time larger than $-\delta$ and smaller than $\delta$, thus equivalent. This detour allows us to test the inverted null and alternative hypotheses and avoids the researcher conflict of interest of wanting to prove the null hypothesis.

Some people say that equivalence testing shows how arbitrary the choice of the null is. I say it shows the exact opposite: how non-arbitrary the choice is. In cases like vaccination where for other ethical reasons the concepts of "mathematical absence of effect" ($\mu_{\text{difference}}=0$) and "philosophical absence of effect" (no medicinal effect above placebo level) don't align, we are forced to make this detour and accept a $\delta$ parameter (which is a horrible researcher degree of freedom) precisely because the choice of the null is non arbitrary.

Answer 2

The Answer by David Ernst is really good.

However, a very short answer is that especially with a point null hypothesis such as $H_0: \mu=0$ for the mean of a normal distribution, it is essentially impossible to exclude the possibility that $\mu$ is just the tiniest little bit away from zero no matter how much data you collect that is perfectly in line with $\mu$ being $0$.

On the other hand, your data can easily be such that it is extremely unlikely to have arisen under $H_0$, in which case you would reject it in favor of $H_A:\mu \neq 0$.