Please convince me why I should bother correcting p values with multiple comparisons, because I don't see the point

Question

Statistical inference using p-values results in accepting or rejecting a hypothesis given some p-value, say 0.05. So, if p = 0.05, we can conclude an effect is "significantly" different from zero with 95% confidence. However, I then test a second hypothesis with the same data set. Again, p = 0.05, and I can make the same claim. But wait, many statistical textbooks will preach that because its a second look at the data this p-value must be 'corrected' (read: penalized) to account for the fact that if you test enough hypotheses you will get positive results by random chance, even if nothing is truly happening. If you make 100 comparisons, 5 will erroneously detect an effect when none is present at a threshold of p = 0.05. To deal with this researchers perform a bonferroni correction, or apply the false discovery rate correction, or some other multiple-comparison correction method.

This does not make sense to me for two reasons:

1. The statement that an effect was detected at p=0.05 already communicates this information. Anyone who has been taught statistics should understand there is still some probability that the effect is spurrious, and the p-value communicates that probability.
2. The proposed correction methods guard against Type I errors (false positives), at the expense of artifically inflating the Type II error rate (false negatives). However, to my knowledge, there is no a priori reason why one error should be considered "worse" than the other. While some might argue that you want to be conservative in hypothesis testing and err on the side of caution, a far more effective strategy would be lowering your p-value threshold rather than ad hoc correcting your p-values.

All in all, a p-value is a pretty straightforward statistic. Its the probability a result may have been detected by chance. Its no more and no less than that. "Correcting" a p-value seems to do more harm than good in the context of interpretation.

Note: this is different than misunderstanding the jelly bean problem. I understand that if you run 20 tests, one will spuriously detect an effect (green jelly beans cause acne - though it should have been equally probable that some jelly bean color prevented acne...) What I do not understand is why multiple comparison corrections are an appropriate way to deal with this problem. The problem is always there.

Answer 1

In your point one you use the word "should" and that is the key, while everyone should understand this, most do not. Google for Andrew Gelman's blog (or other sources) for examples of cases where researchers who "should" know better don't and the reviewers that "should" know better don't, and the editors who "should" know better don't and even when this mistake is pointed out, they still don't. In the comic, we know that there were 20 tests done, but does it spell that out in the article? probably not. That is one of the problems is that many times multiple tests are done, but only the interesting ones are reported, so the reader does not know how many tests to adjust for unless the researcher specifically talks about them or does the adjustment. Even worse, some research is done by throwing in as many predictors as can be thought of and declaring "success" if at least one p-value is less than 0.05. If these researchers were not continually told that they needed to adjust for multiple comparisons then they would be rewarded for behaviors like throwing in extra, meaningless predictors, or breaking a single large study into several small studies, or other strategies that increase type I error rates and therefore give them a better chance of "success".

You ask why we multiply the p-value rather than divide the significance level. Mathematically it is equivalent, though conceptually there is a difference, and I agree with you that it makes much more conceptual sense to adjust the significance level, but apparently for most people it is easier to let the computer multiply the p-value than think about a different significance level (and in the case of FDR you would need to calculate a different significance level for each comparison).

You are right to think about both type I and type II errors, but type I error is much easier to control, so it tends to get much more attention. I did once need to respond to an article review that we intentionally did not control for multiple comparisons because this was a case where the type II error was much more serious than a type I error.

Here are a couple of articles that spell some of these issues out in more detail (and give good reasons why sometimes you don't want to adjust as well as times you do):

Multiplicity in randomised trials I: endpoints and treatments Kenneth F Schulz, David A Grimes. Lancet 2005; 365: 1591–95

Multiplicity in randomised trials II: subgroup and interim analyses Kenneth F Schulz, David A Grimes. Lancet 2005; 365: 1657–61