Why use a null hypothesis not predicted by theory?

Question

The current question is inspired by this one: Which one is the null hypothesis? Conflict between science theory, logic and statistics?

Paul Meehl attributed the first occurrence of using a null hypothesis other than that predicted by theory to Karl Pearson. Are there earlier examples of this behavior? Also, what justifications for doing so have been put forward since?

Paul Meel wrote:

As I pointed out in my 1967 article, when the physicist uses a probable error in this second way, improvement in the quality and number of measurements leading to a lessened standard error subjects the theory to a greater risk of falsification, because here a “significant deviation” means a deviation from the predicted point value or curve type. That is how Karl Pearson’s original invention of chi square at the turn of the century worked. His idea of chi square was as an indicator of frequency discordance, asking for example, does an observed distribution depart significantly from the frequencies in class intervals as given by the Gaussian (or other theoretical) function? This I call the strong use of a significance test. But then occurs a development in the use of chi square, at Pearson’s own hands admittedly, in which the “theoretical” or “expected” values of cell frequencies, rather than being positively generated by an affirmative substantive theory generating a certain mathematical form, are instead specified by the hypothesis that two variables are not related to one another. So the expected values of cell tallies are provided by multiplying the marginals on the hypothesis of independence, using the product theorem of the probability calculus. There is, of course, nothing wrong with the mathematics of that procedure. But social scientists seem unaware of the great shift methodo-logically that takes place in that reverse-direction use of a significance test, where now the substantive theory is supported by the achievement of significance in departing from the “empty” hypothesis that two things are unrelated. In the strong use of a significance test, the more precise the experiment, the more dangerous for the theory. Whereas the social scientist’s use of chi square in a fourfold table, where H0 is that “These things are not related,” I call the weak use. Here, getting a significant result depends solely on the statistical power function, because the null hypothesis is always literally false.

Source: (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant using it. Psychological Inquiry, 1, 108-141, 173-180

The 1967 article mentioned: Meehl, Paul E. (1967). "Theory-Testing in Psychology and Physics: A Methodological Paradox". Philosophy of Science 34 (2): 103–115.

The Pearson paper: Pearson, K (1904). "On the Theory of Contingency and Its Relation to Association and Normal Correlation". Drapers' Company Research Memoirs Biometric Series 1: 1–35.

Answer 1

@Nick Cox's comment above "most null hypotheses aren't predicted by theory" gets pretty close to the distinction that Meehl is trying to draw in your excerpt.

When used as a goodness of fit test, the null hypothesis has to be specified by a theory. Specifically, it requires a generative model that can calculate the expected frequency $E_i$ of each event $i$. Suppose you're wondering whether a coin is fair. The null hypothesis comes from a specific theory ("the coin is fair iff $E_{\textrm{heads}}=E_{\textrm{tails}} = \frac{1}{2}$") and calculating the $\chi^2$ statistic provides a specific test of that theory. In fact, the $\chi^2$ test provides a way to test almost any model against your data (you need a to be able to calculate a CDF), but you have to provide the model, or at the very least a procedure for getting the model (people often test against the best-fitting distribution from a specific family). This makes these results fairly "strong".

On the other hand, when used as a test of independence, that's all you really get. You don't need to provide a reason why the two variables are independent or a specific hypothesis about how the two covary, which makes this result conceptually "weaker".

Meehl's broader point is that the null hypothesis is almost always slightly wrong: there are always factors that are beyond your ability to control--or even identify. If you have enough statistical power, you may therefore reject null hypotheses for the most mundane reasons (People advocate for effect size measurements or confidence intervals instead of $t$-tests for similar reasons). To avoid this, he wants researchers to build theories/models that make specific, quantifiable predictions and then measure their success by evaluating the model's performance. This ensures that the underlying phenomenon is understood.

So, why doesn't everyone do this? Some parts of physics actually do operate this way. For example, the Standard Model makes specific predictions about what a Higgs Boson would look like. These closely match the experimental data, which is why Peter Higgs and François Englert are presumably enjoying their Nobel Prize. In other fields, however, we just don't know enough. We can often predict whether a manipulation will make it harder or easier for people to notice a light or tone, but we know so little about perception and the brain that it's very difficult to predict exactly how big that effect will be (and trust me, it's not for want of trying!).

---

There have been several attempts to defend significance testing. Some are pragmatic arguments ("We need to make an actual decision" or, more snarkily, how else would you phrase $H_0$: "I am pregnant"?) or philosophical arguments ("Any deviation from the theory, no matter how small, is actually interesting"). No-effect null hypotheses also seem more defensible when the experimental conditions can be tightly controlled (e.g., in a within-subjects psychophysics experiment with a good design) than when dealing with messy observational data.

Tukey has argued that the null hypothesis testing is valuable not because we're particularly interested in whether a population mean is exactly zero, but because if we cannot rule out the fact that it is zero, we also cannot rule out small effects in either the positive OR negative direction, which leaves us ignorant as to the sign of the effect. He and a few others (Kaiser, Braver) propose treating a hypothesis test as a three-way decision: $u_1 > u_2$, $u_1 < u_2$, or "we can't tell." Of course, a confidence interval could also lead you in the same direction, but on the other hand, reporting whether a CI includes zero is just a hypothesis test in disguise.

R. Chris Farley has taught a class on the "statistical testing controversy." Though it's ten years old, the syllabus might still be of interest. Weeks 4, 5, and 13 "defend" significance testing from the critiques advanced elsewhere in the reading list.

Bill Thompson has compiled a list of 19 articles supporting statistical hypothesis tests; however, it is a companion to his immense list of 402 citations arguing against their indiscriminate use in observational studies. (Note the subtle shift in title though…)

All that said, I think both parameter estimation and hypothesis testing would still count as "weak" according to Meehl. He'd prefer that we test generative models, built prior to the experiments, than perform post-hoc analyses of the resulting data.