21
$\begingroup$

According to Popper, we cannot verify a hypothesis due to the problem of induction - we can only aim to falsify it. If we are repeatedly unable to falsify it, the hypothesis is said to be tentatively accepted. For Popper, all science shall come up with hypotheses and try to falsify them as hard as possible.

In some introductions to statistical hypothesis testing, I could read that scientists aim to falsify the null hypothesis and that this somehow is in accordance with Popper's theory of falsification. Here is a posting, stating this view. (User @Stefan did comment this posting, making exactly my point.)

I have three questions:

  1. Doesn't Popper say we should try to falsify the alternative hypothesis?
  2. Does falsifying the null hypothesis count as failed falsification of the alternative hypothesis?
  3. Might be some semantic sophistry: Shouldn't scientists try to corroborate the null hypothesis instead of trying to falsify it?

(If this posting should be in the "philosophy"-board, please move it there...)

$\endgroup$
4
  • 2
    $\begingroup$ Some of the trouble with this is that we pretty much always know that the null is at least a little bit false. $\endgroup$
    – Dave
    Jul 26, 2021 at 0:58
  • $\begingroup$ @Dave Maybe a stochastic reformulation of Popper's theory of falsification is possible? $\endgroup$
    – Galen
    Jul 26, 2021 at 3:04
  • 4
    $\begingroup$ It's something Deborah Mayo's written about, e.g. here. The philosophy of Statistics is on topic here & probably on Philosophy SE as well. $\endgroup$ Jul 26, 2021 at 9:36
  • 1
    $\begingroup$ Check this answer. $\endgroup$ Jul 26, 2021 at 18:55

3 Answers 3

17
$\begingroup$

I was also going to point to Deborah Mayo's work as linked in a comment. She is a Popper influenced philosopher who has written a lot about statistical testing.

I'll try to address the questions.

(1a) Popper didn't think of statistical testing as formalising his approach at all. Mayo states that this is because Popper was not expert enough in statistics, but also he probably wouldn't have allowed for an error probability of 5% or 1% as "falsification" (Mayo may also have mentioned this somewhere, but I don't remember).

(1b) There are different approaches for picking the null and alternative hypothesis. In some applications, the null hypothesis is a precise scientific theory of interest, and we check whether the data falsify it. This would be in line with Popper (at least if he allowed for nonzero error probabilities). In some other approaches (in many areas this is found much more often), the null hypothesis formalises the idea that "nothing meaningful is going on", and the alternative is of actual scientific interest. This would not be in line with Popper. (Also, the alternative is not normally specified precisely enough to imply conditions for falsification, and be it statistical.)

(2) According to the standard logic of statistical tests, the null hypothesis can be statistically (i.e. with error probability) falsified, but not the alternative. There is a possibility to argue that an alternative is statistically falisfied, but this basically amounts to running tests the other way round. For example, if you have a $H_0:\ \mu=0$ and an alternative $\mu\neq 0$, you cannot falsify the alternative (as it allows for $\mu$ arbitrarily close to 0, which cannot be distinguished by data from $\mu=0$), but you could state that a meaningful deviation from $\mu=0$ would actually be $|\mu|\ge 2$, and in this case you may reject $|\mu|\ge 2$ in case $\bar x$ is very close to zero. This makes sense if the power of the original test for $|\mu|\ge 2$ is large enough that in that case "$\bar x$ close to zero" would be very unlikely. (This is related to Mayo's concept of "severity"; in such a case we can say that $|\mu|<2$ "with severity".) We could also then say that we have "statistically falsified" $|\mu|\ge 2$.

(3) This is indeed a philosophical question, and I have seen arguments in either direction.

$\endgroup$
10
  • 2
    $\begingroup$ " he probably wouldn't have allowed for an error probability of 5% or 1% as "falsification" neither would Fisher, ISTR in one of his books he said the significance level should depend on the nature of the analysis, but that 1:20 is a useful level in many settings. $\endgroup$ Jul 26, 2021 at 12:34
  • $\begingroup$ @Lewian, regarding (1b), isn't it a common convention that H_1 is always the hypothesis a scientist comes up with and thinks explains a phenomenon, while H_0 is always the opposite, often stating no effect? I have read quite a lot of introductions to hypotheses-testing (all in German) and they all defined H_0 and H_1 as I just said. So in that case, is the whole process of hypotheses-testing Popperian? If the answer to this is "no" -> Where or how exactly is Poppers theory of falsification embedded in common hypothesis-testing in e.g. physics or psychology? Is it at all? $\endgroup$ Jul 26, 2021 at 16:55
  • 1
    $\begingroup$ To make things easier: What should I tell my high-school students if they ask why we do the whole null-hypothesis-thing? Is it correct to say that this is a result of Poppers falsificationism (because we try to falsify H_1 and we do this by trying to not-reject H_0, although what we really hope for is that we are able to reject H_0 and therefore kinda tentatively accept H_1)? $\endgroup$ Jul 26, 2021 at 17:21
  • 1
    $\begingroup$ Although it is sometimes possible to falsify $H_1$, not-rejecting $H_0$ does not imply that $H_1$ has been falsified. The most common two results of a hypothesis test are probably that both $H_0$ and $H_1$ can explain the data, or that $H_1$ but not $H_0$ can explain the data. When I first introduce hypothesis testing, I teach my students that a normal hypothesis test on $H_0$ tells you literally nothing about $H_1$. The fact that its often $H_1$ we want to know about is precisely the problem people who say our scientific practice shouldn't be so dependent on hypothesis testing will raise. $\endgroup$ Jul 26, 2021 at 17:39
  • 1
    $\begingroup$ @DikranMarsupial I added something to (2) that makes the connection clearer. $\endgroup$ Jul 27, 2021 at 9:55
7
$\begingroup$

Too long for comments, so here are my thoughts.

Null Hypothesis Statistical Testing (NHST) is only Popperian in the sense that no amount of corroboration proves a hypothesis correct, so often the best you can do is to find out what you can reasonably reject and continue on with hypotheses that have survived the tests thrown at them so far.

Firstly, we should avoid talking of falsifying the null hypothesis, and should stick to "reject" or "do not reject". Being able to reject the null hypothesis does not mean that we have shown it to be false, just that the observations are unlikely under that hypothesis. The observations may be even more unlikely under the alternative hypothesis! Here is the classic example:

enter image description here

In this case the null hypothesis is almost certainly true, even though we have rejected it, the detector was almost certainly giving a random false alarm. This is because the alternative hypothesis was even more unlikely to be true than the null hypothesis, because the prior probability of H1 was vastly smaller than that of H0, but the NHST does not take that into account. This is an example where rejecting the null hypothesis is not a failed falsification of the alternative hypothesis.

Conversely, if an NHST has low statistical power, then a failure to reject the null does not falsify the alternative hypothesis.

As @Dave suggests, sometimes we know for sure a-priori that the null hypothesis is false, for example a coin with two faces is unlikely to be exactly unbiased, i.e. p(head) = p(tail) = 0.5, but we may need a very large amount of coin-flips to detect the bias that is bound to be present, even in a coin that is to all intents and purposes "unbiased". Testing for normality involves a similar issue in most cases, AFAICS. Rejecting a hypothesis that you know to be false from the outset is not very Popperian, but that doesn't mean that such NHSTs cannot perform a useful purpose.

The Quine-Duhem Thesis suggests that in practice it is not that easy to falsify a hypothesis either.

$\endgroup$
2
1
$\begingroup$

Second-hand from Richard McElreath, but I think no. Popper's famous falsification theory was about falsifying experimental hypotheses not null hypotheses.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.