4
$\begingroup$

The current question is inspired by this one: Which one is the null hypothesis? Conflict between science theory, logic and statistics?

Paul Meehl attributed the first occurrence of using a null hypothesis other than that predicted by theory to Karl Pearson. Are there earlier examples of this behavior? Also, what justifications for doing so have been put forward since?

Paul Meel wrote:

As I pointed out in my 1967 article, when the physicist uses a probable error in this second way, improvement in the quality and number of measurements leading to a lessened standard error subjects the theory to a greater risk of falsification, because here a “significant deviation” means a deviation from the predicted point value or curve type. That is how Karl Pearson’s original invention of chi square at the turn of the century worked. His idea of chi square was as an indicator of frequency discordance, asking for example, does an observed distribution depart significantly from the frequencies in class intervals as given by the Gaussian (or other theoretical) function? This I call the strong use of a significance test. But then occurs a development in the use of chi square, at Pearson’s own hands admittedly, in which the “theoretical” or “expected” values of cell frequencies, rather than being positively generated by an affirmative substantive theory generating a certain mathematical form, are instead specified by the hypothesis that two variables are not related to one another. So the expected values of cell tallies are provided by multiplying the marginals on the hypothesis of independence, using the product theorem of the probability calculus. There is, of course, nothing wrong with the mathematics of that procedure. But social scientists seem unaware of the great shift methodo-logically that takes place in that reverse-direction use of a significance test, where now the substantive theory is supported by the achievement of significance in departing from the “empty” hypothesis that two things are unrelated. In the strong use of a significance test, the more precise the experiment, the more dangerous for the theory. Whereas the social scientist’s use of chi square in a fourfold table, where H0 is that “These things are not related,” I call the weak use. Here, getting a significant result depends solely on the statistical power function, because the null hypothesis is always literally false.

Source: (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant using it. Psychological Inquiry, 1, 108-141, 173-180

The 1967 article mentioned: Meehl, Paul E. (1967). "Theory-Testing in Psychology and Physics: A Methodological Paradox". Philosophy of Science 34 (2): 103–115.

The Pearson paper: Pearson, K (1904). "On the Theory of Contingency and Its Relation to Association and Normal Correlation". Drapers' Company Research Memoirs Biometric Series 1: 1–35.

$\endgroup$
  • 1
    $\begingroup$ The answer depends on whether you want the terminology of null hypothesis to be explicit and on what you call theory. In a broad sense, Arbuthnott and Daniel Bernoulli both used similar ideas roughly two centuries before Karl Pearson. amazon.co.uk/History-Probability-Statistics-Wiley/dp/0471471291 is good. $\endgroup$ – Nick Cox Dec 12 '13 at 23:58
  • $\begingroup$ @NickCox Do they use null hypotheses not predicted by theory or "common sense"? I do not have easy access to that book but will get access to read their reasoning. The real important aspect is whether the null hypothesis is plausible or not. $\endgroup$ – Flask Dec 13 '13 at 0:10
  • $\begingroup$ I don't think most null hypotheses are predicted by theory. I don't know what you mean by "common sense" here. $\endgroup$ – Nick Cox Dec 13 '13 at 0:17
  • $\begingroup$ @NickCox Predicted by theory could be that telekinesis does not exist or that the force due to gravity equals a certain value. Common sense could be that men and women are born at the same rate. $\endgroup$ – Flask Dec 13 '13 at 1:25
  • $\begingroup$ One thing to keep in mind with hypothesis testing is the need to compute the distribution of the test statistic under the null. Imagine you have a point null (usually the case with scientific theories). You might want to ponder the problems with doing that for 'non-null' hypotheses. $\endgroup$ – Glen_b Dec 13 '13 at 5:06
4
$\begingroup$

@Nick Cox's comment above "most null hypotheses aren't predicted by theory" gets pretty close to the distinction that Meehl is trying to draw in your excerpt.

When used as a goodness of fit test, the null hypothesis has to be specified by a theory. Specifically, it requires a generative model that can calculate the expected frequency $E_i$ of each event $i$. Suppose you're wondering whether a coin is fair. The null hypothesis comes from a specific theory ("the coin is fair iff $E_{\textrm{heads}}=E_{\textrm{tails}} = \frac{1}{2}$") and calculating the $\chi^2$ statistic provides a specific test of that theory. In fact, the $\chi^2$ test provides a way to test almost any model against your data (you need a to be able to calculate a CDF), but you have to provide the model, or at the very least a procedure for getting the model (people often test against the best-fitting distribution from a specific family). This makes these results fairly "strong".

On the other hand, when used as a test of independence, that's all you really get. You don't need to provide a reason why the two variables are independent or a specific hypothesis about how the two covary, which makes this result conceptually "weaker".

Meehl's broader point is that the null hypothesis is almost always slightly wrong: there are always factors that are beyond your ability to control--or even identify. If you have enough statistical power, you may therefore reject null hypotheses for the most mundane reasons (People advocate for effect size measurements or confidence intervals instead of $t$-tests for similar reasons). To avoid this, he wants researchers to build theories/models that make specific, quantifiable predictions and then measure their success by evaluating the model's performance. This ensures that the underlying phenomenon is understood.

So, why doesn't everyone do this? Some parts of physics actually do operate this way. For example, the Standard Model makes specific predictions about what a Higgs Boson would look like. These closely match the experimental data, which is why Peter Higgs and François Englert are presumably enjoying their Nobel Prize. In other fields, however, we just don't know enough. We can often predict whether a manipulation will make it harder or easier for people to notice a light or tone, but we know so little about perception and the brain that it's very difficult to predict exactly how big that effect will be (and trust me, it's not for want of trying!).


There have been several attempts to defend significance testing. Some are pragmatic arguments ("We need to make an actual decision" or, more snarkily, how else would you phrase $H_0$: "I am pregnant"?) or philosophical arguments ("Any deviation from the theory, no matter how small, is actually interesting"). No-effect null hypotheses also seem more defensible when the experimental conditions can be tightly controlled (e.g., in a within-subjects psychophysics experiment with a good design) than when dealing with messy observational data.

Tukey has argued that the null hypothesis testing is valuable not because we're particularly interested in whether a population mean is exactly zero, but because if we cannot rule out the fact that it is zero, we also cannot rule out small effects in either the positive OR negative direction, which leaves us ignorant as to the sign of the effect. He and a few others (Kaiser, Braver) propose treating a hypothesis test as a three-way decision: $u_1 > u_2$, $u_1 < u_2$, or "we can't tell." Of course, a confidence interval could also lead you in the same direction, but on the other hand, reporting whether a CI includes zero is just a hypothesis test in disguise.

R. Chris Farley has taught a class on the "statistical testing controversy." Though it's ten years old, the syllabus might still be of interest. Weeks 4, 5, and 13 "defend" significance testing from the critiques advanced elsewhere in the reading list.

Bill Thompson has compiled a list of 19 articles supporting statistical hypothesis tests; however, it is a companion to his immense list of 402 citations arguing against their indiscriminate use in observational studies. (Note the subtle shift in title though…)

All that said, I think both parameter estimation and hypothesis testing would still count as "weak" according to Meehl. He'd prefer that we test generative models, built prior to the experiments, than perform post-hoc analyses of the resulting data.

$\endgroup$
  • $\begingroup$ I agree with all, but I am still wondering what justifications people give for performing significance tests rather than doing parameter estimation when we don't know enough to make a prediction. Has anyone ever published an argument justifying a null hypothesis of no difference? $\endgroup$ – Flask Dec 13 '13 at 15:04
  • $\begingroup$ Harold Jeffreys' Theory of probability although emphatically not cast in any Fisher or Neyman-Pearson mould has as a central theme that you prize simplicity and choose more complicated models only when the evidence is compelling. Something like that is, I think, one basic argument for null hypotheses. $\endgroup$ – Nick Cox Dec 13 '13 at 17:30
  • $\begingroup$ @Flask, sorry! I answered your title more than your actual question. I edited in some links which might be of interest $\endgroup$ – Matt Krause Dec 17 '13 at 5:48
  • $\begingroup$ @MattKrause If you haven't seen it I would highly recommend this pre-print: To P or not to P: on the evidential nature of P-values and their place in scientific inference. Its the only paper I have read to give me an explanation for why a p-value is useful. It always bugged me that they clearly indicated something, but I could never accept the process of using them because I couldnt make sense of it. I remember thinking this when I first learned stats, then accepting that if everyone did it I must not have enough background to understand. Ill check out yours. $\endgroup$ – Flask Dec 17 '13 at 6:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.