# Interpretation of p-value in hypothesis testing

I recently came across the paper "The Insignificance of Null Hypothesis Significance Testing", Jeff Gill (1999). The author raised a few common misconceptions regarding hypothesis testing and p-values, about which I have two specific questions:

1. The p-value is technically $P(observation|H_{0})$, which, as pointed out by the paper, generally does not tell us anything about $P(H_{0}|observation)$, unless we happen to know the marginal distributions, which is rarely the case in "everyday" hypothesis testing. When we obtain a small p-value and "reject the null hypothesis," what exactly is the probabilistic statement that we are making, since we cannot say anything about $P(H_{0}|observation)$?
2. The second question relates to a particular statement from page 6(652) of the paper:

Since the p-value, or range of p-values indicated by stars, is not set a prior, it is not the long-run probability of making a Type I error but is typically treated as such.

Can anyone help to explain what is meant by this statement?

-
TY for the reference to the paper –  lkuty Dec 1 at 16:39

(Technically, the P-value is the probability of observing data at least as extreme as that actually observed, given the null hypothesis.)

Q1. A decision to reject the null hypothesis on the basis of a small P-value typically depends on 'Fisher's disjunction': Either a rare event has happened or the null hypothesis is false. In effect, it is rarity of the event is what the P-value tells you rather than the probability that the null is false.

The probability that the null is false can be obtained from the experimental data only by way of Bayes' theorem, which requires specification of the 'prior' probability of the null hypothesis (presumably what Gill is referring to as "marginal distributions").

Q2. This part of your question is much harder than it might seem. There is a great deal of confusion regarding P-values and error rates which is, presumably, what Gill is referring to with "but is typically treated as such." The combination of Fisherian P-values with Neyman-Pearsonian error rates has been called an incoherent mishmash, and it is unfortunately very widespread. No short answer is going to be completely adequate here, but I can point you to a couple of good papers (yes, one is mine). Both will help you make sense of the Gill paper.

Hurlbert, S., & Lombardi, C. (2009). Final collapse of the Neyman-Pearson decision theoretic framework and rise of the neoFisherian. Annales Zoologici Fennici, 46(5), 311–349. (Link to paper)

Lew, M. J. (2012). Bad statistical practice in pharmacology (and other basic biomedical disciplines): you probably don't know P. British Journal of Pharmacology, 166(5), 1559–1567. doi:10.1111/j.1476-5381.2012.01931.x (Link to paper)

-
Thanks for the clarification. Is it technically incorrect to make statement such as "the small p-value indicates that the sample mean(or regression coefficient, etc) is significantly different from zero"? The source of confusion seems to be that no real probabilistic claim is being made to the null hypothesis when we say the null is "rejected." –  ezbentley Jan 3 at 5:20
@ezbentley, that really depends on what you mean by significant. That word is not really very meaningful in most contexts because it has been contaminated by the Fisher-Neyman-Pearson hybrid. If you obtained a very small P-value then it is fair to say that the true mean is probably not zero, but it is important to say what the observed mean was, and indicate its variability (SEM or confidence interval), and don't forget to say what the sample size was. A P-value is not a substitute for specification of the observed effect size. –  Michael Lew Jan 3 at 5:26
Thank you for the explanation. I need to dig deeper into the Fisher and Neyman-Pearson paradigm. –  ezbentley Jan 3 at 5:36

+1 to @MichaelLew, who has provided you with a good answer. Perhaps I can still contribute by providing a way of thinking about Q2. Consider the following situation:

• The null hypothesis is true. (Note that if the null hypothesis is not true, no type I errors are possible, and it's not clear what meaning the $p$-value has.)
• $\alpha$ has been set conventionally at $0.05$.
• The computed $p$-value is $0.01$.

Now, the probability of getting data as extreme or more extreme than your data is 1% (that's what the $p$-value means). You have rejected the null hypothesis, making a type I error. Is it true that the long run type I error rate in this situation is also 1%, which many people might intuitively conclude? The answer is no. The reason is that if you had gotten a $p$-value of $0.02$, you would still have rejected the null. In fact, you would have rejected the null even if $p$ had been $0.04\bar{9}$, and in the long run, $p$'s up to this large will occur $\approx$5% of the time and all of such rejections will be type I errors. Thus, the long run type I error rate is 5% (where you had set $\alpha$).

(Disclosure: I have not read Gill's paper, so I cannot guarantee that this is what he meant, but it does make sense of the claim that the $p$-value is not [necessarily] the same as the long run type I error rate.)

-
+1, especally for "Note that if the null hypothesis is not true, no type I errors are possible, and it's not clear what meaning the p-value has." –  andrea Jan 3 at 17:13
Working in a field (epi) where oftentimes it's extremely difficult to believe that the hypothesis H_0=0 is actually true, I think that this point is overlooked and deserves much more attention. –  andrea Jan 3 at 17:18
Just to make sure my understanding is correct. P-value itself is a random variable, and Type-I error is the probability that this random variable is less than $\alpha$. Is this right? –  ezbentley Jan 3 at 19:07
@ezbentley, yeah, you've got it. –  gung Jan 3 at 19:58
+1, but the suggestion that the meaning of a P-value is unclear when the null is false is misleading. The smaller the P-value the larger the discrepancy between the null and the observed. The larger the sample size, the closer it can be assumed that the true effect size is to the observed effect size. It is very useful to note that significance testing is analogous to estimation. –  Michael Lew Jan 3 at 20:13
In my opinion, the main problem is not the misinterpretation of the $p$-value. Many practitioners often test for a "significant difference" for instance, and they wrongly believe that a significant difference means that there is a "big" difference. More precisely they are in the contexte of a "precise" null hypothesis $H_0$ having form $H_0\colon\{\theta=0\}$. This hypothesis will be rejected when $\theta=\epsilon$ even for a very small $\epsilon$ when the sample size increases. But in the real world, there's no difference between a small $\epsilon$ and $0$.