Can someone please offer a nice succinct explanation why it is not a good idea to teach students that a p-value is the prob(their findings are due to [random] chance). My understanding is that a p-value is the prob(getting more extreme data | null hypothesis is true).

My real interest is what is the harm of telling them it is the former (aside from the fact it is simply not so).

  • 43
    $\begingroup$ Because it's wrong? $\endgroup$ – whuber Oct 13 '11 at 3:11
  • 6
    $\begingroup$ Maybe what you want is a simple example to show it's not just wrong but bad? $\endgroup$ – Karl Oct 13 '11 at 3:23
  • 2
    $\begingroup$ Some things are simply matters of fact, Patrick, not of opinion: Pi is not equal to three (despite attempts to legislate it so), for instance. But your comment is indeed a useful clarification: it suggests you're not asking about the harm of teaching the wrong thing, but are really seeking reasons to explain the difference to people. $\endgroup$ – whuber Oct 13 '11 at 3:27
  • 2
    $\begingroup$ There's good discussion of these issues at stats.stackexchange.com/questions/5591/…, even among the lower-voted replies (IMHO). $\endgroup$ – whuber Oct 13 '11 at 3:32
  • 1
    $\begingroup$ Yes Karl, I suppose I am looking for real world examples. Those dealing with observational based studies (e.g., environmental science, ecology, wildlife sciences) would be great. I read that thread (whuber) before I posted this, along with several pubs. Thank you for that though. $\endgroup$ – Patrick Oct 13 '11 at 3:42

I have a different interpretation of the meaning of the wrong statement than @Karl does. I think that it is a statement about the data, rather than about the null. I understand it as asking for the probability of getting your estimate due to chance. I don't know what that means---it's not a well-specified claim.

But I do understand what is likely meant by the probability of getting my estimate by chance given that the true estimate is equal to a particular value. For example, I can understand what it means to get a very large difference in average heights between men and women given that their average heights are actually the same. That's well specified. And that is what the p-value gives. What's missing in the wrong statement is the condition that the null is true.

Now, we might object that this isn't statement perfect (the chance of getting an exact value for an estimator is 0, for example). But it's far better than the way that most would interpret a p-value.

The key point that I say over and over again when I teach hypothesis testing is "Step one is to assume that the null hypothesis is true. Everything is calculated given this assumption." If people remember that, that's pretty good.

  • $\begingroup$ Oh, that looks good to me. I see I've been making the same point without noticing [sigh] (+1) $\endgroup$ – conjugateprior Dec 13 '12 at 21:24
  • $\begingroup$ But what of "what is the harm"? $\endgroup$ – rolando2 Jan 28 '17 at 2:00

I've seen this interpretation a lot (perhaps more often than the correct one). I interpret "their findings are due to [random] chance" as "$\text{H}_0$ is true", and so really what they are saying is $\Pr(\text{H}_0)$ [which actually should be $\Pr(\text{H}_0 | \text{data})$; say, "given what we have seen (the data), what is the probability that only chance is operating?"] This can be a meaningful statement (if you are willing to assign priors and do Bayes), but it is not the p-value.

$\Pr(\text{H}_0 | \text{data})$ can be quite different than the p-value, and so to interpret a p-value in that way can be seriously misleading.

The simplest illustration: say the prior, $\Pr(H_0)$ is quite small, but one has rather little data, and so the p-value is largish (say, 0.3), but the posterior, $\Pr(\text{H}_0|\text{data})$, would still be quite small. [But maybe this example isn't so interesting.]

  • $\begingroup$ So Pr(H0|data) == to prob(their findings are due to [random] chance)? $\endgroup$ – Patrick Oct 13 '11 at 3:18
  • $\begingroup$ @Patrick - yes. $\endgroup$ – Karl Oct 13 '11 at 3:21
  • 1
    $\begingroup$ @Patrick - no, definitely not. In classical hypothesis testing, $\Pr(H_0|\text{anything})$ makes no sense. $\endgroup$ – whuber Oct 13 '11 at 3:24
  • $\begingroup$ @whuber - but that's the point. "prob(their findings are due to [random] chance)" is really $\Pr(H_0)$ which I think should be written as $\Pr(H_0 | \text{data})$. It can make sense (with priors + Bayes), but it's not the p-value. $\endgroup$ – Karl Oct 13 '11 at 3:32
  • 2
    $\begingroup$ Hmm, I think I'm still not following, although I'm grateful you have invoked Bayes and prior distributions to give a context to your reply and comments, which otherwise are puzzling. That's perhaps because I am interpreting "findings" to mean "data," not "$H_0$". It's hard to wrap my mind around the concept "the null hypothesis is due to chance," even in a Bayesian model. (In the Bayesian setup, that statement wouldn't be adding any information not already assumed at the outset: all hypotheses are random variables.) $\endgroup$ – whuber Oct 13 '11 at 3:39

I'll add a late answer from the (ex) student perspective: IMHO the harm cannot be separated from its being wrong.

This type of wrong "didactic approximations/shortcut" can create a lot of confusion for students who realize that they cannot logically understand the statement, but assuming that what is taught to them is right they do not realize that they are not able to understand it because it is not right.

This does not affect students who just memorize rules presented to them. But it requires students who learn by understanding to be good enough to

  • arrive at the correct solution by themselves and
  • be good enough so they can be sure they are right
  • and conclude that they are taught bullshit (for some allegedly didactic reason).

I'm not saying that there aren't valid didactic shortcuts. But IMHO when such a shortcut is taken, this should be mentioned (e.g. as "for the ease of the argument, we assume/approximate that ...").
In this particular case, however, I think it is too misleading to be of any use.

  • 1
    $\begingroup$ +1 This is a very good point, if you teach students something that is incorrect, you encourage them to construct a model of how statistics work that is faulty, and may well cause them to misunderstand other elements of statistics that are on the syllabus (e.g. what a confidence interval - if you encourage students to think a frequentist probability can be attached to a hypothesis, then why can't it be applied to the hypothesis that the true value lies in a particular interval). Understanding is the true aim of education, and this requires accuracy. $\endgroup$ – Dikran Marsupial Dec 20 '12 at 10:04

Referring directly to the question: Where is the harm?

In my opinion, the answer to this question lies in the converse of the statement, "A p-value is the probability that the findings are due to random chance." If one believes this, then one also probably believes the following: "[1-(p-value)] is the probability that the findings are NOT due to random chance."

The harm then lies in the second statement, because, given the way most people's brains work, this statement grossly overestimates how confident we should be in the specific values of an estimated parameter.


Here is a simple example that I use:

Suppose our null hypothesis is that we are flipping a 2-headed coin (so prob(heads) = 1). Now we flip the coin one time and get heads, the p-values for this is 1, so does that mean that we have a 100% chance of having a 2-headed coin?

The tricky thing is that if we had flipped a tails then the p-value would have been 0 and the probability of having a 2-headed coin would have been 0, so they match in this case, but not the above. The p-value of 1 above just means that what we have observed is perfectly consistent with the hypothesis of a 2-headed coin, but it does not prove that the coin is 2-headed.

Further, if we are doing frequentist statistics then the null hypothesis is either True or False (we just don't know which) and making (frequentist) probability statements about the null hypothesis is meaningless. If you want to talk about the probability of the hypothesis, then do proper Bayesian statistics, use the Bayesian definition of probability, start with a prior and calculate the posterior probability that the hypothesis is true. Just don't confuse a p-value with a Bayesian posterior.


OK another, slightly different take on this:

A first basic problem is the phrase "due to [random] chance". The idea of unspecified 'chance' comes naturally to students but it is hazardous for thinking clearly about uncertainty and catastrophic for doing sensible statistics. With something like a sequence of coin flips it is easy to assume that 'chance' is described by the Binomial setup with a probability of 0.5. There is a certain naturalness to it for sure, but from a statistical point of view it's not more natural than assuming 0.6 or something else. And for other less 'obvious' examples, e.g. involving real parameters it's utterly unhelpful to think about what 'chance' would look like.

With respect to the question, the key idea is understanding what sort of 'chance' is described by H0, i.e. what actual likelihood/DGP H0 names. Once that concept is in place, students finally stop talking about things happening 'by chance', and start asking what H0 actually is. (They also figure out that things can be consistent with a rather wide variety of Hs so they get a head start on confidence intervals, via inverted tests).

The second problem is that if you're on the way to Fisher's definition of p-values, you should (imho) always explain it first in terms of the data's consistency with H0 because the point of p is to see that, not to interpret the tail area as some sort of 'chance' activity, (or frankly to interpret it at all). This is purely a matter of rhetorical emphasis, obviously, but it seems to help.

In short, the harm is that this way of describing things will not generalise to any non-trivial model they might subsequently try to think about. At worst it may just add to sense of mystery that the study of statistics already generates in the sorts of people such bowdlerised descriptions are aimed at.


If I take apart, "p-value is the probability that an effect is due to chance," it seems to be implying that the effect is caused by chance. But every effect is partially caused by chance. In a statistics lesson where one is explaining the need to try to see through random variability this is a pretty magical and overreaching statement. It imbues p-values with powers they do not have.

If you define chance in a specific case to be the null hypothesis then you're stating that the p-value yields the probability that the observed effect is caused by the null hypothesis. That seems awfully close to the correct statement but claiming that a condition on probability is the cause of that probability is again overreaching. The correct statement, that the p-value is the probability of the effect given the null hypothesis is true, does not ascribe cause to the null effect. The causes are various including the true effect, the variability around the effect, and random chance. The p-value doesn't measure the probability of any of those.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.