Statistical inference using p-values results in accepting or rejecting a hypothesis given some p-value, say 0.05. So, if p = 0.05, we can conclude an effect is "significantly" different from zero with 95% confidence. However, I then test a second hypothesis with the same data set. Again, p = 0.05, and I can make the same claim. But wait, many statistical textbooks will preach that because its a second look at the data this p-value must be 'corrected' (read: penalized) to account for the fact that if you test enough hypotheses you will get positive results by random chance, even if nothing is truly happening. If you make 100 comparisons, 5 will erroneously detect an effect when none is present at a threshold of p = 0.05. To deal with this researchers perform a bonferroni correction, or apply the false discovery rate correction, or some other multiple-comparison correction method.
This does not make sense to me for two reasons:
- The statement that an effect was detected at p=0.05 already communicates this information. Anyone who has been taught statistics should understand there is still some probability that the effect is spurrious, and the p-value communicates that probability.
- The proposed correction methods guard against Type I errors (false positives), at the expense of artifically inflating the Type II error rate (false negatives). However, to my knowledge, there is no a priori reason why one error should be considered "worse" than the other. While some might argue that you want to be conservative in hypothesis testing and err on the side of caution, a far more effective strategy would be lowering your p-value threshold rather than ad hoc correcting your p-values.
All in all, a p-value is a pretty straightforward statistic. Its the probability a result may have been detected by chance. Its no more and no less than that. "Correcting" a p-value seems to do more harm than good in the context of interpretation.
Note: this is different than misunderstanding the jelly bean problem. I understand that if you run 20 tests, one will spuriously detect an effect (green jelly beans cause acne - though it should have been equally probable that some jelly bean color prevented acne...) What I do not understand is why multiple comparison corrections are an appropriate way to deal with this problem. The problem is always there.