Explain the xkcd jelly bean comic: What makes it funny?

I see that one time out of the twenty total tests they run, $p < 0.05$, so they wrongly assume that during one of the twenty tests, the result is significant ($0.05 = 1/20$).

xkcd jelly bean comic - "Significant"

• Title: Significant
• Hover text: "'So, uh, we did the green study again and got no link. It was probably a--' 'RESEARCH CONFLICTED ON GREEN JELLY BEAN/ACNE LINK; MORE STUDY RECOMMENDED!'"

-
95% confidence will mean that on average in 5% of experiments (one out of 20) we'll get an opposite conclusion. Which is exactly what has happened here. I.e., if you also make the same experiment with orange jelly beans 1000 times, ~ 50 of those will give positive result. :) –  sashkello Feb 27 at 0:37
Is this a candidate for CW, perhaps? –  Glen_b Feb 27 at 0:47
Who said it's funny? –  whuber Feb 27 at 1:12
–  Jeromy Anglim Feb 27 at 3:21
@Glen_b, the favorite data analysis cartoon thread is appropriately CW, however, I see no reason this one should be. 'Why funny' aside, the question asks for an understanding of the statistical point at issue in the cartoon, which has an answer & should be on-topic & not-CW (& which I think you handled well below). –  gung Feb 27 at 3:31

Humor is a very personal thing - some people will find it amusing, but it may not be funny to everyone. Indeed not all xkcd's are even intended to be actually funny. Many do, however make important points in a way that's thought provoking, and at least sometimes they're amusing while doing that. (I personally find it funny, but I find it hard to clearly explain what, exactly, makes it funny to me. I think partly it's the recognition of the way that a doubtful, or even dubious result turns into a media circus (on which see also this PhD comic), and perhaps partly the recognition of the way some research may actually be done - if usually not consciously.)

However, one can appreciate the point whether or not it tickles your funnybone.

The point is about doing multiple hypothesis tests at some moderate significance level like 5%, and then publicizing the one that came out significant. Of course, if you do 20 such tests when there's really nothing of any importance going on, the expected number of those tests to give a significant result is 1. Doing a rough in-head approximation for $n$ tests at significance level $\frac{1}{n}$, there's roughly a 37% chance of no significant result, roughly 37% chance of one and roughly 26% chance of more than one (I just checked the exact answers; they're close enough to that).

In the comic, Randall depicted 20 tests, so this is no doubt his point (that you expect to get one significant even when there's nothing going on). The fictional newspaper article even emphasizes the problem with the subhead "Only 5% chance of coincidence!". (If the one test that ended up in the papers was the only one done, that might be the case.)

Of course, there's also the subtler issue that an individual researcher may behave much more reasonably, but the problem of rampant publicizing of false positives still occurs. Let's say that these researchers only do 5 tests, each at the 1% level, so their overall chance of discovering a bogus result like that is only about five percent.

So far so good. But now imagine there are 20 such research groups, each testing whichever random subset of colors they think they have reason to try. Or 100 research groups... what chance of a headline like the one in the comic now?

So more broadly, the comic may be referencing publication bias more generally. If only significant results are trumpeted, we won't hear about the dozens of groups that found nothing for green jellybeans, only the one that did.

Indeed, that's one of the major points being made in this article, which has been in the news in the last few months (e.g. here, even though it's a 2005 article).

A response to that article emphasizes the need for replication. Note that if there were to be several replications of the study that was published, the "Green jellybeans linked to acne" result would be very unlikely to stand.

(And indeed, the hover text for the comic makes a clever reference to the same point.)

-

What people overlook is that the actual p-value for the green jelly bean case is not .05 but around .64. Only the pretend (nominal) p-value is .05. There’s a difference between actual and pretend p-values. The probability of finding 1 in 20 that reach the nominal level even if all the nulls are true is NOT .05, but .64. On the other hand, if you appraise evidence looking at comparative likelihoods—the most popular view aside from the error statistical one (within which p-values reside) you WILL say there’s evidence for H: green jelly beans are genuinely correlated with acne. That’s because P(x;no effect) < P(x; H). The left side is < .05, whereas the right side is fairly high: if green jelly beans did cause acne then finding the observed association would be probable. Likelihoods alone fail to pick up on error probabilities because they condition on the actual data attained. There’s no difference in the appraisal than if there had just been this one test of the green jelly beans and acne. So although this cartoon is often seen as making fun of p-values, the very thing that’s funny about it demonstrates why we need to consider the overall error probability (as non-pretend p values do) and not merely likelihoods. Bayesian inference is also conditioned on the outcome, ignoring error probabilities. The only way to avoid finding evidence for H, for a Bayesian would be to have a low prior in H. But we would adjust the p-value no matter what the subject matter, and without relying on priors, because of the hunting procedure used to find the hypothesis to test. Even if the H that was hunted was believable, it's still a lousy test. Errorstatistics.com

-
It is very hard to tell exactly what this post is trying to say. Let me focus on one part of it, hoping that a clarification might reveal the meaning of the rest: exactly what do you mean by "the overall error probability"? –  whuber Aug 21 at 19:41
@whuber I believe that the post is referring to the multiple comparisons problem. –  Matt Aug 26 at 21:03