I recently came across the paper "The Insignificance of Null Hypothesis Significance Testing", Jeff Gill (1999). The author raised a few common misconceptions regarding hypothesis testing and p-values, about which I have two specific questions:

  1. The p-value is technically $P({\rm observation}|H_{0})$, which, as pointed out by the paper, generally does not tell us anything about $P(H_{0}|{\rm observation})$, unless we happen to know the marginal distributions, which is rarely the case in "everyday" hypothesis testing. When we obtain a small p-value and "reject the null hypothesis," what exactly is the probabilistic statement that we are making, since we cannot say anything about $P(H_{0}|{\rm observation})$?
  2. The second question relates to a particular statement from page 6(652) of the paper:

Since the p-value, or range of p-values indicated by stars, is not set a priori, it is not the long-run probability of making a Type I error but is typically treated as such.

Can anyone help to explain what is meant by this statement?


(Technically, the P-value is the probability of observing data at least as extreme as that actually observed, given the null hypothesis.)

Q1. A decision to reject the null hypothesis on the basis of a small P-value typically depends on 'Fisher's disjunction': Either a rare event has happened or the null hypothesis is false. In effect, it is rarity of the event is what the P-value tells you rather than the probability that the null is false.

The probability that the null is false can be obtained from the experimental data only by way of Bayes' theorem, which requires specification of the 'prior' probability of the null hypothesis (presumably what Gill is referring to as "marginal distributions").

Q2. This part of your question is much harder than it might seem. There is a great deal of confusion regarding P-values and error rates which is, presumably, what Gill is referring to with "but is typically treated as such." The combination of Fisherian P-values with Neyman-Pearsonian error rates has been called an incoherent mishmash, and it is unfortunately very widespread. No short answer is going to be completely adequate here, but I can point you to a couple of good papers (yes, one is mine). Both will help you make sense of the Gill paper.

Hurlbert, S., & Lombardi, C. (2009). Final collapse of the Neyman-Pearson decision theoretic framework and rise of the neoFisherian. Annales Zoologici Fennici, 46(5), 311–349. (Link to paper)

Lew, M. J. (2012). Bad statistical practice in pharmacology (and other basic biomedical disciplines): you probably don't know P. British Journal of Pharmacology, 166(5), 1559–1567. doi:10.1111/j.1476-5381.2012.01931.x (Link to paper)

  • $\begingroup$ Thanks for the clarification. Is it technically incorrect to make statement such as "the small p-value indicates that the sample mean(or regression coefficient, etc) is significantly different from zero"? The source of confusion seems to be that no real probabilistic claim is being made to the null hypothesis when we say the null is "rejected." $\endgroup$ – user13587 Jan 3 '13 at 5:20
  • 2
    $\begingroup$ @ezbentley, that really depends on what you mean by significant. That word is not really very meaningful in most contexts because it has been contaminated by the Fisher-Neyman-Pearson hybrid. If you obtained a very small P-value then it is fair to say that the true mean is probably not zero, but it is important to say what the observed mean was, and indicate its variability (SEM or confidence interval), and don't forget to say what the sample size was. A P-value is not a substitute for specification of the observed effect size. $\endgroup$ – Michael Lew Jan 3 '13 at 5:26
  • $\begingroup$ Thank you for the explanation. I need to dig deeper into the Fisher and Neyman-Pearson paradigm. $\endgroup$ – user13587 Jan 3 '13 at 5:36
  • $\begingroup$ @Michael Lew: Maybe it could be interesting to take a look at my answer: stats.stackexchange.com/questions/166323/… $\endgroup$ – user83346 Aug 15 '15 at 15:25
  • 1
    $\begingroup$ @nalzok I was working from a combination reprint of three of Fisher's books amazon.com/… It was published in 1990. $\endgroup$ – Michael Lew Dec 27 '19 at 6:12

+1 to @MichaelLew, who has provided you with a good answer. Perhaps I can still contribute by providing a way of thinking about Q2. Consider the following situation:

  • The null hypothesis is true. (Note that if the null hypothesis is not true, no type I errors are possible, and it's not clear what meaning the $p$-value has.)
  • $\alpha$ has been set conventionally at $0.05$.
  • The computed $p$-value is $0.01$.

Now, the probability of getting data as extreme or more extreme than your data is 1% (that's what the $p$-value means). You have rejected the null hypothesis, making a type I error. Is it true that the long run type I error rate in this situation is also 1%, which many people might intuitively conclude? The answer is no. The reason is that if you had gotten a $p$-value of $0.02$, you would still have rejected the null. In fact, you would have rejected the null even if $p$ had been $0.04\bar{9}$, and in the long run, $p$'s up to this large will occur $\approx$5% of the time and all of such rejections will be type I errors. Thus, the long run type I error rate is 5% (where you had set $\alpha$).

(Disclosure: I have not read Gill's paper, so I cannot guarantee that this is what he meant, but it does make sense of the claim that the $p$-value is not [necessarily] the same as the long run type I error rate.)

  • 1
    $\begingroup$ Working in a field (epi) where oftentimes it's extremely difficult to believe that the hypothesis H_0=0 is actually true, I think that this point is overlooked and deserves much more attention. $\endgroup$ – boscovich Jan 3 '13 at 17:18
  • 1
    $\begingroup$ Just to make sure my understanding is correct. P-value itself is a random variable, and Type-I error is the probability that this random variable is less than $\alpha$. Is this right? $\endgroup$ – user13587 Jan 3 '13 at 19:07
  • 1
    $\begingroup$ +1, but the suggestion that the meaning of a P-value is unclear when the null is false is misleading. The smaller the P-value the larger the discrepancy between the null and the observed. The larger the sample size, the closer it can be assumed that the true effect size is to the observed effect size. It is very useful to note that significance testing is analogous to estimation. $\endgroup$ – Michael Lew Jan 3 '13 at 20:13
  • 3
    $\begingroup$ @MichaelLew, I'm not sure that the p-value means these things on its own. In conjunction w/ N (& specifically, holding N constant) a smaller p will correspond to a larger discrepancy b/t the null & observed. Even then, that's more of something that can be inferred from p rather than something p means. It is also true that w/ larger N observed effect sizes should be closer to true ES's, but it's less clear to me what role p plays there. EG, w/ a false null, the true effect could still be very small, & w/ large N we would expect the observed ES to be close, but p could still be large. $\endgroup$ – gung - Reinstate Monica Jan 3 '13 at 20:50
  • 1
    $\begingroup$ @gung, I am reading this paper <ftp.stat.duke.edu/WorkingPapers/03-26.pdf> by Hubbard. On page 12, he states ...this fallacy shows up in statistics textbooks, as when Canavos and Miller (1999, p.255) stipulate: "If the null hypothesis is true, then a type I error occurs if (due to sampling error) the P-value is less than or equal to $alpha$" I thought that type-I error is exactly the probability that p-value is less than $\alpha$ if the null is true. So why is Hubbard saying that the quoted statement is a fallacy? Am I misinterpreting it somehow? $\endgroup$ – user13587 Jan 7 '13 at 18:01

I'd like to make a comment related to "the insignificance of null hypothesis significance testing" but which does not answer the question of the OP.

In my opinion, the main problem is not the misinterpretation of the $p$-value. Many practitioners often test for a "significant difference" for instance, and they wrongly believe that a significant difference means that there is a "big" difference. More precisely they are in the context of a "precise" null hypothesis $H_0$ having form $H_0\colon\{\theta=0\}$. This hypothesis will be rejected when $\theta=\epsilon$ even for a very small $\epsilon$ when the sample size increases. But in the real world, there's no difference between a small $\epsilon$ and $0$ (we say there is equivalence between a small $\epsilon$ and $0$ and equivalence testing is the way to go in such a situation).

  • 3
    $\begingroup$ +1 Yes, the real problem with conventional hypothesis testing is that it answers a question that you are not really interested in having answered, i.e. "is there significant evidence of a difference?", rather than "is there evidence of a significant difference?". Of course what is really desired is generally "what is the probability that my research hypothesis is true?", but this cannot be answered within a frequentist framework. The misinterpretation generally arises from attempts to treat the frequentist test in Bayesian terms. $\endgroup$ – Dikran Marsupial Jan 3 '13 at 13:23
  • 1
    $\begingroup$ It is not a good idea to separate the meaning of P-values and sample size. A smaller P-value indicates a larger effect size at any particular sample size, and for any particular P-value a larger sample size indicates that the true effect size is probably closer to the observed effect size. Significance tests should be thought of in the context of estimation, not errors. A larger sample always gives more information -- how to interpret it is up to the experimenter. The large sample negligible effect complaint is only a problem for Neyman-Pearsonian hypothesis testing. $\endgroup$ – Michael Lew Jan 3 '13 at 20:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy