Do testability and falsifiability have statistical definitions?

Question

Psychology: the Core Concepts says

Psychology differs from the pseudosciences in that it employs the scientific method to test its ideas empirically. The scientific method relies on testable theories and falsifiable hypotheses.

Do testability and falsifiability have statistical definitions? If not, what do they mean roughly, and how are they related and different?

Are theory and hypothesis the same concepts?

Answer 1

Testability and falsifiability are general ideas that are discussed at length in the philosophy of science, but they manifest in statistics and aspects of these concepts can be framed in statistical or probabilistic terms. It is useful to have a broad understanding of the philosophy of science and its historical development to understand these concepts, but it is also useful to see how they arise in the context of probability and statistics. Below we examine the latter.

The principle of "falsifiability" is a consequence of the law of total probability

The principle of falsifiability means that in a valid experimental situation relating to a hypothesis, there must be at least one possible outcome that would count as evidence against the hypothesis; if there is not, then the experiment cannot ever be considered to give evidence in favour of the hypothesis. This principle is built into probability theory via the law of total probability, and it occurs in Bayesian reasoning. This rule of probability ensures that if there can be confirmatory evidence for a hypothesis, then it must also be possible for there to be disconfirmatory evidence for that same hypothesis. This property of probability theory is captured in the following simple theorem.

Theorem (Principle of falsifiability for countable space): Consider a hypothesis $H$ and suppose we have a partition of the sample space $\mathscr{E}$ composed of a countable number of events. Suppose that there is at least one piece of confirmatory evidence $E \in \mathscr{E}$ that is in favour of the hypothesis ---i.e., a piece of evidence such that:

$$\mathbb{P}(H|E) > \mathbb{P}(H).$$

Then there must exist at least one event $E' \in \mathscr{E}$ that is disconfirmatory to the hypothesis ---i.e., a piece of evidence such that:

$$\mathbb{P}(H|E') < \mathbb{P}(H).$$

Proof: We will use a proof by contradiction. Suppose ---contra the theorem--- that for all $R \in \mathscr{E}$ we have $\mathbb{P}(H|R) \geqslant \mathbb{P}(H)$. Using the law of total probability, we then have:

$$\begin{align} \mathbb{P}(H) &= \sum_{R} \mathbb{P}(H|R) \mathbb{P}(R) \\[6pt] &= \bigg[ \mathbb{P}(H|E) \mathbb{P}(E) + {\sum_{R \neq E} \mathbb{P}(H|R) \mathbb{P}(R)} \bigg] \\[6pt] &\geqslant \bigg[ \mathbb{P}(H|E) \mathbb{P}(E) + {\sum_{R \neq E} \mathbb{P}(H) \mathbb{P}(R)} \bigg] \\[6pt] &> \bigg[ \mathbb{P}(H) \mathbb{P}(E) + {\sum_{R \neq E} \mathbb{P}(H) \mathbb{P}(R)} \bigg] \\[6pt] &= \mathbb{P}(H) \bigg[ \mathbb{P}(E) + {\sum_{R \neq E} \mathbb{P}(R)} \bigg] \\[6pt] &= \mathbb{P}(H) {\sum_{R} \mathbb{P}(R)} \\[8pt] &= \mathbb{P}(H), \\[6pt] \end{align}$$

which is a contradiction. This establishes the theorem. $\blacksquare$

If you would like to see an application of this principle within Bayesian reasoning, you might be interested in reading O'Neill (2014) on the famous "doomsday argument". This paper argues that the doomsday argument is an example of erroneous reasoning in which there is an argument to a foregone conclusion, in contradiction to the proper application of Bayes' rule. You might also be interested in reading Kadane et al (1996), which talks generally about the notion of "reasoning to a foregone conclusion" (i.e., without the possibility of falsification) and gives sufficient foundational conditions for probabilistic reasoning under which this cannot occur.

The principle of "testability" relates to experimental design and other statistical principles

The notion of "testability" means that it is possible to create an experiment that can provide sufficient evidence to test the hypothesis. This idea therefore forms a part of the field of experimental design, which can be regarded as a subfield of statistics. Wrapped up in this notion is the determination of the requirements that would be needed to form a valid experiment for a hypothesis, any protocols that need to be applied (e.g., randomisation, blinding, etc.), and how much evidence needs to be accumulated in order to get sufficient evidence on the hypothesis of interest to make an inference at some minimum level of confidence/accuracy. The last of these is usually determined by making sample size calculations using statistical rules.

Testability can be framed in many different ways, with stronger or weaker requirements for particular contexts. It is often framed as requiring it to be possible to determine whether the hypothesis is true or false, and in some contexts it might require one to test causal hypothesis, which then imposes additional requirements. In any case, those are strong notions of testability, but if we are willing to think probabilistically, then a very weak notion of testability would merely require that it is possible to create an experiment that can provide either confirmatory or disconfirmatory evidence to some degree. This is a lot weaker than some demands for testability but it does provide a potential starting point.

If we are willing accept a very weak notion of testability then this occurs when it is possible to construct an experiment where the evidence can shift our posterior belief away from our prior belief under some observable evidence. We have already seen above that if it is possible to see confirmatory evidence then it must also be possible to see disconfirmatory evidence, so a belief-shift either way could potentially occur. Weak testability would occur so long as the evidence in the experiment is not (statistically) independent of the hypothesis of interest. If we want to impose a stronger requirement for testability then it might entail having a larger amount of evidence (e.g., a minimal required sample size), or it might require imposing a particular experimental design or experimental protocols.

Answer 2

They don't have statistical definitions but statistics may be helpful in showing either.

"Testable" means ... well "can be tested". If (sticking to psychology) is that, say, "In college, men prefer male professors and women prefer female ones" then that is testable because I can (at least in theory) look at a whole bunch of male and female college students and get them to rate male and female professors. This is a case where statistics will be very helpful in doing the test. On the other hand, if my hypothesis is "every male student dislikes every female professor" then statistics isn't needed. One example is enough.

"Falsifiable" means that there is some possible evidence which would lead me to reject my hypothesis. Maybe statistical, maybe not.

EDIT: In the second paragraph, I mean "reject" in the common language sense, rather than in the sense of statistics (although they could overlap).

Answer 3

This is meant to add an aspect to the other answers, which have some valuable material.

When talking about "falsification" in statistics, in most cases this isn't the same as logical falsification in the sense that a hypothesis would be logically falsified if something happens that under the hypothesis is impossible, so the hypothesis is strictly incompatible with the observed data and must therefore be false (even though philosophy of science teaches that it isn't quite that easy even without taking into account statistical variation, as in many cases something that supposedly refutes a theory/hypothesis can be explained in other ways such as erroneous measurements, or failure of an auxiliary hypothesis that connects the main hypothesis to the data rather than failure of the main hypothesis of interest itself).

In statistics, however, standard models will assign nonzero probability (or at least density) to every potential outcome, be the hypothesis true or not, and this means that any claim that a hypothesis "is falsified by the data" will come with a nonzero error probability, i.e., observed data were just very unlikely but not strictly impossible (regardless of whether we're talking Bayesian or frequentist inference). This also means that if we want to make statements such as "the hypothesis is falsified by the data", this needs to be based on a threshold (how small a probability is small enough to talk of "falsification"), or otherwise we can only make "graded" statements (such as p-value or posterior probability of hypothesis or Bayes factor equal to 0.03), but ultimately an interpretation in words is needed anyway.

Regarding what the hypotheses are, statistical hypotheses are probability models (often parametric with a restrictive specification of parameter values), whereas "research hypotheses" are often informal (in some fields that strongly rely on mathematics, formal research hypotheses are the standard, but in some other fields almost everything is informal). So a research hypothesis needs to be "translated" into a statistical model, and this usually involves model assumptions such as independence or certain distributional shapes that can be doubted (and checked, but only to a limited extent) and may affect the interpretation of any outcome of the statistical analyses. In fact this adds an additional source of uncertainty on top of any uncertainty already modelled by the statistical model.

The question regarding theory vs. hypothesis makes sense regarding the "research hypothesis", but chances are that in different fields and situations connections between what people call "theory" and what people call "research hypothesis" may differ (a research hypothesis may often be a more specified instance/special case of a more general theory); in any case the statistical hypothesis will not normally be identical to the research hypothesis let alone the scientific theory of interest, but will come with additional restrictions (and/or add-ons, as the theory to be tested may not involve observational variation, which is modelled by the statistical hypothesis).

This also implies that for any result of a statistical analysis it makes sense to ask: "Could these data have led us to a different conclusion had we chosen another statistical model that is as well compatible with the research hypothesis of interest and - as far as this can be tested - the data?"

Another uncomfortable implication is that if many statistical hypothesis are up for "falsification" at a certain probability standard, the probability that an error ("statistically falsifying" a hypothesis that is in fact true) occurs can become quite large, as the probability that one out of many (falsification) events obtains can be quite large even if the probability for every single such event is very small (referred to as the problem of multiple testing in statistics).

Answer 4

These are important concepts that are in many ways more fundamental than statistics, they are about philosophy of science. I know only a little about this, so what follows is mainly a summary of my thinking and not an expert answer.

Theories are organising frameworks that contain several related ideas. A single theory can lead to or contain several hypotheses. A hypothesis is much more specific: it is an explanation for a phenomenon that includes a specific mechanism. Note that these are research or scientific hypotheses, not the statistical hypotheses that we frame in our models¹. A research or scientific hypothesis has to be translated into a statistical hypothesis to be amenable to quantitative analysis (or perhaps 'quantitative falsification', to coin a phrase). I've yet to find a good discussion about how best to go about this; it is an interesting and complex challenge.

A hypothesis can lead to one or more predictions, which can be tested. The more specific and novel the predictions that arise from a hypothesis, the more useful that hypothesis is. If a precise prediction is found to be wrong through appropriate experimentation or observation, we can judge the hypothesis to be incorrect i.e. falsified. This falsification step usually involves statistical analysis but it not may require it, especially if a single observation is sufficient for falsification (this can be captured using Bayesian philosophy/approaches, but it may not be needed). The proverbial black swan is of course the usual go-to example for the power of a single observation. Real-world examples are harder to come by, but Eddington's observation of light bending around the sun during a solar eclipse (predicted by Einstein's general theory of relativity) might be one; I don't know enough about the details to be sure though.

I've found Popper's essay on Science as Falsification² to be a short, clear and insightful exposition of this viewpoint. His vision is an excellent goal for science to aspire to, even if it is an incomplete or incorrect description of how science actually functions. It's a strongly contested view, and I'd encourage looking at alternate viewpoints as well.

I find 'theory' and 'testable' are harder to pin down than the other terms. Especially the latter - one can in principle refer to theories or predictions as testable, which muddies the waters somewhat. On this front, Kuhn and Lakatos raised important and interesting ideas about how it is that theories rise and fall. The processes they describe have a strong social dynamic and are much messier than the Popperian falsification of hypotheses described above. Hypotheses that form part of a theory can be falsified while the theory itself survives in some reduced or modified form, for example. Often, the theory is jettisoned only when a substantial portion of its core ideas are falsified or replaced by a theory that improves on it - by making new predictions, or more precise and accurate ones.

---

¹ I think statistics has done science a disservice by causing so much confusion around the term 'hypothesis'. Null hypothesis statistical testing has arguably led to a great deal of poor science in part because people think that they are pursuing some high scientific goal by 'testing hypotheses' even though it's the uninteresting kind of hypotheses.

² Popper, K. R. (1963). Science as falsification. Conjectures and refutations, 1(1963), 33-39.