Interpretation of p-value in hypothesis testing

Question

I recently came across the paper "The Insignificance of Null Hypothesis Significance Testing", Jeff Gill (1999). The author raised a few common misconceptions regarding hypothesis testing and p-values, about which I have two specific questions:

1. The p-value is technically $P({\rm observation}|H_{0})$, which, as pointed out by the paper, generally does not tell us anything about $P(H_{0}|{\rm observation})$, unless we happen to know the marginal distributions, which is rarely the case in "everyday" hypothesis testing. When we obtain a small p-value and "reject the null hypothesis," what exactly is the probabilistic statement that we are making, since we cannot say anything about $P(H_{0}|{\rm observation})$?
2. The second question relates to a particular statement from page 6(652) of the paper:

Since the p-value, or range of p-values indicated by stars, is not set a priori, it is not the long-run probability of making a Type I error but is typically treated as such.

Can anyone help to explain what is meant by this statement?

Answer 1

(Technically, the P-value is the probability of observing data at least as extreme as that actually observed, given the null hypothesis.)

Q1. A decision to reject the null hypothesis on the basis of a small P-value typically depends on 'Fisher's disjunction': Either a rare event has happened or the null hypothesis is false. In effect, it is rarity of the event is what the P-value tells you rather than the probability that the null is false.

The probability that the null is false can be obtained from the experimental data only by way of Bayes' theorem, which requires specification of the 'prior' probability of the null hypothesis (presumably what Gill is referring to as "marginal distributions").

Q2. This part of your question is much harder than it might seem. There is a great deal of confusion regarding P-values and error rates which is, presumably, what Gill is referring to with "but is typically treated as such." The combination of Fisherian P-values with Neyman-Pearsonian error rates has been called an incoherent mishmash, and it is unfortunately very widespread. No short answer is going to be completely adequate here, but I can point you to a couple of good papers (yes, one is mine). Both will help you make sense of the Gill paper.

Hurlbert, S., & Lombardi, C. (2009). Final collapse of the Neyman-Pearson decision theoretic framework and rise of the neoFisherian. Annales Zoologici Fennici, 46(5), 311–349. (Link to paper)

Lew, M. J. (2012). Bad statistical practice in pharmacology (and other basic biomedical disciplines): you probably don't know P. British Journal of Pharmacology, 166(5), 1559–1567. doi:10.1111/j.1476-5381.2012.01931.x (Link to paper)

Answer 2

+1 to @MichaelLew, who has provided you with a good answer. Perhaps I can still contribute by providing a way of thinking about Q2. Consider the following situation:

• The null hypothesis is true. (Note that if the null hypothesis is not true, no type I errors are possible, and it's not clear what meaning the $p$-value has.)
• $\alpha$ has been set conventionally at $0.05$.
• The computed $p$-value is $0.01$.

Now, the probability of getting data as extreme or more extreme than your data is 1% (that's what the $p$-value means). You have rejected the null hypothesis, making a type I error. Is it true that the long run type I error rate in this situation is also 1%, which many people might intuitively conclude? The answer is no. The reason is that if you had gotten a $p$-value of $0.02$, you would still have rejected the null. In fact, you would have rejected the null even if $p$ had been $0.04\bar{9}$, and in the long run, $p$'s up to this large will occur $\approx$5% of the time and all of such rejections will be type I errors. Thus, the long run type I error rate is 5% (where you had set $\alpha$).

(Disclosure: I have not read Gill's paper, so I cannot guarantee that this is what he meant, but it does make sense of the claim that the $p$-value is not [necessarily] the same as the long run type I error rate.)

Answer 3

I'd like to make a comment related to "the insignificance of null hypothesis significance testing" but which does not answer the question of the OP.

In my opinion, the main problem is not the misinterpretation of the $p$-value. Many practitioners often test for a "significant difference" for instance, and they wrongly believe that a significant difference means that there is a "big" difference. More precisely they are in the context of a "precise" null hypothesis $H_0$ having form $H_0\colon\{\theta=0\}$. This hypothesis will be rejected when $\theta=\epsilon$ even for a very small $\epsilon$ when the sample size increases. But in the real world, there's no difference between a small $\epsilon$ and $0$ (we say there is equivalence between a small $\epsilon$ and $0$ and equivalence testing is the way to go in such a situation).

Answer 4

I’ve attempted to describe this as a distinction between assertion probabilities and decision errors here. Part of the issue is that essentially $\alpha = \Pr(p < 0.05 | H_{0})$ and that setting $\alpha$ does not have the desired effect of controlling decision errors. This is elaborated on in the section on Bayesian operating characteristics here.