# Please convince me why I should bother correcting p values with multiple comparisons, because I don't see the point

Statistical inference using p-values results in accepting or rejecting a hypothesis given some p-value, say 0.05. So, if p = 0.05, we can conclude an effect is "significantly" different from zero with 95% confidence. However, I then test a second hypothesis with the same data set. Again, p = 0.05, and I can make the same claim. But wait, many statistical textbooks will preach that because its a second look at the data this p-value must be 'corrected' (read: penalized) to account for the fact that if you test enough hypotheses you will get positive results by random chance, even if nothing is truly happening. If you make 100 comparisons, 5 will erroneously detect an effect when none is present at a threshold of p = 0.05. To deal with this researchers perform a bonferroni correction, or apply the false discovery rate correction, or some other multiple-comparison correction method.

This does not make sense to me for two reasons:

1. The statement that an effect was detected at p=0.05 already communicates this information. Anyone who has been taught statistics should understand there is still some probability that the effect is spurrious, and the p-value communicates that probability.
2. The proposed correction methods guard against Type I errors (false positives), at the expense of artifically inflating the Type II error rate (false negatives). However, to my knowledge, there is no a priori reason why one error should be considered "worse" than the other. While some might argue that you want to be conservative in hypothesis testing and err on the side of caution, a far more effective strategy would be lowering your p-value threshold rather than ad hoc correcting your p-values.

All in all, a p-value is a pretty straightforward statistic. Its the probability a result may have been detected by chance. Its no more and no less than that. "Correcting" a p-value seems to do more harm than good in the context of interpretation.

Note: this is different than misunderstanding the jelly bean problem. I understand that if you run 20 tests, one will spuriously detect an effect (green jelly beans cause acne - though it should have been equally probable that some jelly bean color prevented acne...) What I do not understand is why multiple comparison corrections are an appropriate way to deal with this problem. The problem is always there.

• @whuber The jelly bean responses do not deal with these two questions I pose here. I get that if you do 20 tests one will be significant (green jelly beans cause acne). I state that in my introduction of the question. I don't understand why multiple comparison corrections are an appropriate solution to that problem. Please explai how this question is a duplicate of people not understanding the jelly bean comic. I understand the jelly bean comic. – colin Aug 18 '17 at 18:08
• This doesn't seem a duplicate question to me, but it does contain some straw-man arguments. We do adjust significance levels instead of p-values - I'd even say that's the default in my field. FDR q-values are not adjusted p-values. And a priori declaring that Type I errors are worse is the entire basis of NHST: to give a practical example, -omics experiments routinely test $>10^5$ markers, so declaring thousands of significant findings in each study would immediately flood any research community with false leads. – juod Aug 18 '17 at 18:32
• Id like to point out that your sentence saying that a p-value is a straightforward statistic misinterprets the p-value. It is not the probability that a result would have been detected by chance. You need to add "assuming the null hypothesis" at the very least, and "equally or more extreme" to lock it down. – Matthew Drury Aug 18 '17 at 18:44
• "...a far more effective strategy would be lowering your p-value threshold rather than ad hoc correcting your p-values." This is exactly what multiple comparison corrections are doing. – klumbard Aug 18 '17 at 19:05