Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I recently came across the paper "The Insignificance of Null Hypothesis Significance Testing", Jeff Gill (1999). The author raised a few common misconceptions regarding hypothesis testing and p-values, about which I have two specific questions:

  1. The p-value is technically $P({\rm observation}|H_{0})$, which, as pointed out by the paper, generally does not tell us anything about $P(H_{0}|{\rm observation})$, unless we happen to know the marginal distributions, which is rarely the case in "everyday" hypothesis testing. When we obtain a small p-value and "reject the null hypothesis," what exactly is the probabilistic statement that we are making, since we cannot say anything about $P(H_{0}|{\rm observation})$?
  2. The second question relates to a particular statement from page 6(652) of the paper:

Since the p-value, or range of p-values indicated by stars, is not set a priori, it is not the long-run probability of making a Type I error but is typically treated as such.

Can anyone help to explain what is meant by this statement?

share|improve this question
    
TY for the reference to the paper –  lkuty Dec 1 '13 at 16:39
add comment

3 Answers

up vote 17 down vote accepted

(Technically, the P-value is the probability of observing data at least as extreme as that actually observed, given the null hypothesis.)

Q1. A decision to reject the null hypothesis on the basis of a small P-value typically depends on 'Fisher's disjunction': Either a rare event has happened or the null hypothesis is false. In effect, it is rarity of the event is what the P-value tells you rather than the probability that the null is false.

The probability that the null is false can be obtained from the experimental data only by way of Bayes' theorem, which requires specification of the 'prior' probability of the null hypothesis (presumably what Gill is referring to as "marginal distributions").

Q2. This part of your question is much harder than it might seem. There is a great deal of confusion regarding P-values and error rates which is, presumably, what Gill is referring to with "but is typically treated as such." The combination of Fisherian P-values with Neyman-Pearsonian error rates has been called an incoherent mishmash, and it is unfortunately very widespread. No short answer is going to be completely adequate here, but I can point you to a couple of good papers (yes, one is mine). Both will help you make sense of the Gill paper.

Hurlbert, S., & Lombardi, C. (2009). Final collapse of the Neyman-Pearson decision theoretic framework and rise of the neoFisherian. Annales Zoologici Fennici, 46(5), 311–349. (Link to paper)

Lew, M. J. (2012). Bad statistical practice in pharmacology (and other basic biomedical disciplines): you probably don't know P. British Journal of Pharmacology, 166(5), 1559–1567. doi:10.1111/j.1476-5381.2012.01931.x (Link to paper)

share|improve this answer
    
Thanks for the clarification. Is it technically incorrect to make statement such as "the small p-value indicates that the sample mean(or regression coefficient, etc) is significantly different from zero"? The source of confusion seems to be that no real probabilistic claim is being made to the null hypothesis when we say the null is "rejected." –  ezbentley Jan 3 '13 at 5:20
    
@ezbentley, that really depends on what you mean by significant. That word is not really very meaningful in most contexts because it has been contaminated by the Fisher-Neyman-Pearson hybrid. If you obtained a very small P-value then it is fair to say that the true mean is probably not zero, but it is important to say what the observed mean was, and indicate its variability (SEM or confidence interval), and don't forget to say what the sample size was. A P-value is not a substitute for specification of the observed effect size. –  Michael Lew Jan 3 '13 at 5:26
    
Thank you for the explanation. I need to dig deeper into the Fisher and Neyman-Pearson paradigm. –  ezbentley Jan 3 '13 at 5:36
add comment

+1 to @MichaelLew, who has provided you with a good answer. Perhaps I can still contribute by providing a way of thinking about Q2. Consider the following situation:

  • The null hypothesis is true. (Note that if the null hypothesis is not true, no type I errors are possible, and it's not clear what meaning the $p$-value has.)
  • $\alpha$ has been set conventionally at $0.05$.
  • The computed $p$-value is $0.01$.

Now, the probability of getting data as extreme or more extreme than your data is 1% (that's what the $p$-value means). You have rejected the null hypothesis, making a type I error. Is it true that the long run type I error rate in this situation is also 1%, which many people might intuitively conclude? The answer is no. The reason is that if you had gotten a $p$-value of $0.02$, you would still have rejected the null. In fact, you would have rejected the null even if $p$ had been $0.04\bar{9}$, and in the long run, $p$'s up to this large will occur $\approx$5% of the time and all of such rejections will be type I errors. Thus, the long run type I error rate is 5% (where you had set $\alpha$).

(Disclosure: I have not read Gill's paper, so I cannot guarantee that this is what he meant, but it does make sense of the claim that the $p$-value is not [necessarily] the same as the long run type I error rate.)

share|improve this answer
    
+1, especally for "Note that if the null hypothesis is not true, no type I errors are possible, and it's not clear what meaning the p-value has." –  andrea Jan 3 '13 at 17:13
    
Working in a field (epi) where oftentimes it's extremely difficult to believe that the hypothesis H_0=0 is actually true, I think that this point is overlooked and deserves much more attention. –  andrea Jan 3 '13 at 17:18
    
Just to make sure my understanding is correct. P-value itself is a random variable, and Type-I error is the probability that this random variable is less than $\alpha$. Is this right? –  ezbentley Jan 3 '13 at 19:07
    
@ezbentley, yeah, you've got it. –  gung Jan 3 '13 at 19:58
1  
@MichaelLew, I'm not sure that the p-value means these things on its own. In conjunction w/ N (& specifically, holding N constant) a smaller p will correspond to a larger discrepancy b/t the null & observed. Even then, that's more of something that can be inferred from p rather than something p means. It is also true that w/ larger N observed effect sizes should be closer to true ES's, but it's less clear to me what role p plays there. EG, w/ a false null, the true effect could still be very small, & w/ large N we would expect the observed ES to be close, but p could still be large. –  gung Jan 3 '13 at 20:50
show 3 more comments

I'd like to make a comment related to "the insignificance of null hypothesis significance testing" but which does not answer the question of the OP.

In my opinion, the main problem is not the misinterpretation of the $p$-value. Many practitioners often test for a "significant difference" for instance, and they wrongly believe that a significant difference means that there is a "big" difference. More precisely they are in the contexte of a "precise" null hypothesis $H_0$ having form $H_0\colon\{\theta=0\}$. This hypothesis will be rejected when $\theta=\epsilon$ even for a very small $\epsilon$ when the sample size increases. But in the real world, there's no difference between a small $\epsilon$ and $0$.

share|improve this answer
1  
+1 Yes, the real problem with conventional hypothesis testing is that it answers a question that you are not really interested in having answered, i.e. "is there significant evidence of a difference?", rather than "is there evidence of a significant difference?". Of course what is really desired is generally "what is the probability that my research hypothesis is true?", but this cannot be answered within a frequentist framework. The misinterpretation generally arises from attempts to treat the frequentist test in Bayesian terms. –  Dikran Marsupial Jan 3 '13 at 13:23
    
It is not a good idea to separate the meaning of P-values and sample size. A smaller P-value indicates a larger effect size at any particular sample size, and for any particular P-value a larger sample size indicates that the true effect size is probably closer to the observed effect size. Significance tests should be thought of in the context of estimation, not errors. A larger sample always gives more information -- how to interpret it is up to the experimenter. The large sample negligible effect complaint is only a problem for Neyman-Pearsonian hypothesis testing. –  Michael Lew Jan 3 '13 at 20:19
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.