1
$\begingroup$

I recently questioned my understanding of multiple comparison when I saw this xkcd comic: https://xkcd.com/882/.

The comic is illustrating multiple comparison by showing a study that concludes that green jelly beans cause acne. It is illustrating that, by testing 20 different hypotheses with the same $\alpha$ level (0.05), it is highly likely that a false positive conclusion will be drawn.

What I don't understand, is why 20 different hypotheses are the cause of this. Since these hypotheses about the colors of jelly beans are independent, and the testing of each of those hypotheses do not influence each other, it doesn't seem to me how this could be an issue?

More precisely, I understand why it is an issue if the same hypothesis is tested 20 times, e.g. the hypothesis that "green jelly beans cause acne". But, in this comic it is only evaluated once. The hypothesis "jelly beans cause acne" is tested 20 times, but we don't draw a conclusion on that, we only draw a conclusion on green jelly beans.

It doesn't make sense to me why evaluating a hypothesis once is considered multiple comparison?

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
4
$\begingroup$

Because the research hypothesis wasn't specifically about green jelly beans, but about whether there is a colour of jelly bean that is associated with acne. If you throw 100 tests at a problem (whether the same exact hypothesis or tests all in the same family of related tests) then the chance (assume H0 true in all cases) is high that some of the results will pass the 5% level of significance spuriously. That's why research hypotheses should be specified in advance. Here is an excellent treatment of related issues.

Improve this answer
$\endgroup$
2
  • $\begingroup$ So, if I understand correctly, you're saying that there actually were many evaluations of the same hypothesis, because the hypothesis is "is there a color of jelly bean that is associated with acne"? In that case, the proper thing to do would be to dismiss the hypothesis after the second block of the comic strip? $\endgroup$
    – makansij
    Feb 16, 2016 at 4:49
  • 2
    $\begingroup$ There are too many unstated things in the comic to say. In theory, if the initial evaluation included the full range of colours and had sufficient power in an omnibus test to have a good chance to detect a single colour deviating from the rest, then yes - it should have stopped there. In real life, that's rarely the case. The real problem is in panel three - when the character states that it's only a certain colour, but doesn't state which. This sets up a family of tests that should have protection against Type I error inflation. $\endgroup$
    – J Taylor
    Feb 16, 2016 at 6:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.