Does the rejection of the null hypothesis have anything to do with Popper's theory of falsification?

Question

According to Popper, we cannot verify a hypothesis due to the problem of induction - we can only aim to falsify it. If we are repeatedly unable to falsify it, the hypothesis is said to be tentatively accepted. For Popper, all science shall come up with hypotheses and try to falsify them as hard as possible.

In some introductions to statistical hypothesis testing, I could read that scientists aim to falsify the null hypothesis and that this somehow is in accordance with Popper's theory of falsification. Here is a posting, stating this view. (User @Stefan did comment this posting, making exactly my point.)

I have three questions:

1. Doesn't Popper say we should try to falsify the alternative hypothesis?
2. Does falsifying the null hypothesis count as failed falsification of the alternative hypothesis?
3. Might be some semantic sophistry: Shouldn't scientists try to corroborate the null hypothesis instead of trying to falsify it?

(If this posting should be in the "philosophy"-board, please move it there...)

Answer 1

I was also going to point to Deborah Mayo's work as linked in a comment. She is a Popper influenced philosopher who has written a lot about statistical testing.

I'll try to address the questions.

(1a) Popper didn't think of statistical testing as formalising his approach at all. Mayo states that this is because Popper was not expert enough in statistics, but also he probably wouldn't have allowed for an error probability of 5% or 1% as "falsification" (Mayo may also have mentioned this somewhere, but I don't remember).

(1b) There are different approaches for picking the null and alternative hypothesis. In some applications, the null hypothesis is a precise scientific theory of interest, and we check whether the data falsify it. This would be in line with Popper (at least if he allowed for nonzero error probabilities). In some other approaches (in many areas this is found much more often), the null hypothesis formalises the idea that "nothing meaningful is going on", and the alternative is of actual scientific interest. This would not be in line with Popper. (Also, the alternative is not normally specified precisely enough to imply conditions for falsification, and be it statistical.)

(2) According to the standard logic of statistical tests, the null hypothesis can be statistically (i.e. with error probability) falsified, but not the alternative. There is a possibility to argue that an alternative is statistically falisfied, but this basically amounts to running tests the other way round. For example, if you have a $H_0:\ \mu=0$ and an alternative $\mu\neq 0$, you cannot falsify the alternative (as it allows for $\mu$ arbitrarily close to 0, which cannot be distinguished by data from $\mu=0$), but you could state that a meaningful deviation from $\mu=0$ would actually be $|\mu|\ge 2$, and in this case you may reject $|\mu|\ge 2$ in case $\bar x$ is very close to zero. This makes sense if the power of the original test for $|\mu|\ge 2$ is large enough that in that case "$\bar x$ close to zero" would be very unlikely. (This is related to Mayo's concept of "severity"; in such a case we can say that $|\mu|<2$ "with severity".) We could also then say that we have "statistically falsified" $|\mu|\ge 2$.

(3) This is indeed a philosophical question, and I have seen arguments in either direction.

Answer 2

Too long for comments, so here are my thoughts.

Null Hypothesis Statistical Testing (NHST) is only Popperian in the sense that no amount of corroboration proves a hypothesis correct, so often the best you can do is to find out what you can reasonably reject and continue on with hypotheses that have survived the tests thrown at them so far.

Firstly, we should avoid talking of falsifying the null hypothesis, and should stick to "reject" or "do not reject". Being able to reject the null hypothesis does not mean that we have shown it to be false, just that the observations are unlikely under that hypothesis. The observations may be even more unlikely under the alternative hypothesis! Here is the classic example:

In this case the null hypothesis is almost certainly true, even though we have rejected it, the detector was almost certainly giving a random false alarm. This is because the alternative hypothesis was even more unlikely to be true than the null hypothesis, because the prior probability of H1 was vastly smaller than that of H0, but the NHST does not take that into account. This is an example where rejecting the null hypothesis is not a failed falsification of the alternative hypothesis.

Conversely, if an NHST has low statistical power, then a failure to reject the null does not falsify the alternative hypothesis.

As @Dave suggests, sometimes we know for sure a-priori that the null hypothesis is false, for example a coin with two faces is unlikely to be exactly unbiased, i.e. p(head) = p(tail) = 0.5, but we may need a very large amount of coin-flips to detect the bias that is bound to be present, even in a coin that is to all intents and purposes "unbiased". Testing for normality involves a similar issue in most cases, AFAICS. Rejecting a hypothesis that you know to be false from the outset is not very Popperian, but that doesn't mean that such NHSTs cannot perform a useful purpose.

The Quine-Duhem Thesis suggests that in practice it is not that easy to falsify a hypothesis either.

Answer 3

Second-hand from Richard McElreath, but I think no. Popper's famous falsification theory was about falsifying experimental hypotheses not null hypotheses.