6
$\begingroup$

The p-postulate is the notion that equal p-values provide equal evidence against the null hypothesis.

Wagenmakers et al (2008) write:

If p-values truly reflect evidence, a minimum requirement is that equal p-values provide equal evidence against the null hypothesis (i.e., the p-postulate). According to the p-postulate, p = .05 with 10 observations constitutes just as much evidence against the null hypothesis as does p = .05 after 50 observations.

They cite Royall (1986) as the source for their definition of the p-postulate. They also go on saying that this postulate is false. Is it?


$\endgroup$
  • 4
    $\begingroup$ What do you understand "evidence" to be here? Do you have an exact definition in mind? $\endgroup$ – Erik May 3 '13 at 10:32
  • $\begingroup$ I do not have an exact definition in mind. I'm open to the answer that it depends on what definition of evidence one adopts. In that case, I'm curious about what definitions respondents feel are most defensible, and why they suggest the p-postulate is true or false. $\endgroup$ – user1205901 - Reinstate Monica May 4 '13 at 1:08
  • $\begingroup$ Maybe this helps stats.stackexchange.com/questions/166323/… $\endgroup$ – user83346 Jun 14 '16 at 15:21
2
$\begingroup$

The Royall paper begins with two quotes that providing apparently contradictory interpretations of the p-value. Both rely on interpreting the p-value in light of the sample size and as such both are flawed interpretations of the p-value.

A p-value tells us one thing and one thing only--the probability of observing a statistic as extreme or more extreme than that observed in a sample as a result of random sampling error. A p-value of .05 with a sample of 10 or a sample of 50 (or any sample size for that matter) yields the same interpretation in any case. Under the assumptions of the model, a difference of the magnitude observed or greater would be observed in just 5% of samples if the null hypothesis were actually true.

So, in response to your specific question and focusing on interpreting only the p-value, the answer is yes--equal p-values provide equal evidence against the null hypothesis at any sample size.

This does not tell us anything about the magnitude of the difference or the effect size. Indeed, all else being equal, the same difference in an observed effect size will yield lower p-values as sample size increases. Strength of evidence against the null (p-value) and magnitude of the difference (effect size) should be interpreted together.

$\endgroup$
  • 1
    $\begingroup$ (1) Royall's paper starts here and confronts this with two other (conflicting) claims about p-values. He concludes with "despite their apparent inconsistencies, interpretations can be given of all three statements ... that make each of them correct." It would be nice to see an explicit acknowledgment of, and response to, that conclusion. (2) re your last line: given that p-values are random, "will" is too strong to be correct. Perhaps you mean "has a greater chance to"? $\endgroup$ – whuber Jun 14 '16 at 15:10
  • $\begingroup$ @whuber 1. I edited to address. 2. I added "observed" to effect size, otherwise this is a mathematical result, no? All else equal in the inputs, increase n in calculating p = smaller p? $\endgroup$ – Brett Jun 14 '16 at 15:57
  • $\begingroup$ It's hard to see how one could keep everything equal in the inputs while increasing $n$. The closest I can imagine is a sequence of independent draws from a distribution--but even then, the p-values constructed from the first $2, 3, \ldots, n$ elements of the sequence can actually increase almost as often as they decrease. $\endgroup$ – whuber Jun 14 '16 at 19:31
  • $\begingroup$ I am talking about the calculation in a hypothetical situation. Increase n it decreases p. $\endgroup$ – Brett Jun 14 '16 at 20:19
  • $\begingroup$ As far as I can tell, there is no such hypothetical situation! To increase $n$, you have to obtain one more random result. Because it is random, frequently the p-value goes up. $\endgroup$ – whuber Jun 14 '16 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.