1
$\begingroup$

Statistical inference using p-values results in accepting or rejecting a hypothesis given some p-value, say 0.05. So, if p = 0.05, we can conclude an effect is "significantly" different from zero with 95% confidence. However, I then test a second hypothesis with the same data set. Again, p = 0.05, and I can make the same claim. But wait, many statistical textbooks will preach that because its a second look at the data this p-value must be 'corrected' (read: penalized) to account for the fact that if you test enough hypotheses you will get positive results by random chance, even if nothing is truly happening. If you make 100 comparisons, 5 will erroneously detect an effect when none is present at a threshold of p = 0.05. To deal with this researchers perform a bonferroni correction, or apply the false discovery rate correction, or some other multiple-comparison correction method.

This does not make sense to me for two reasons:

  1. The statement that an effect was detected at p=0.05 already communicates this information. Anyone who has been taught statistics should understand there is still some probability that the effect is spurrious, and the p-value communicates that probability.
  2. The proposed correction methods guard against Type I errors (false positives), at the expense of artifically inflating the Type II error rate (false negatives). However, to my knowledge, there is no a priori reason why one error should be considered "worse" than the other. While some might argue that you want to be conservative in hypothesis testing and err on the side of caution, a far more effective strategy would be lowering your p-value threshold rather than ad hoc correcting your p-values.

All in all, a p-value is a pretty straightforward statistic. Its the probability a result may have been detected by chance. Its no more and no less than that. "Correcting" a p-value seems to do more harm than good in the context of interpretation.

Note: this is different than misunderstanding the jelly bean problem. I understand that if you run 20 tests, one will spuriously detect an effect (green jelly beans cause acne - though it should have been equally probable that some jelly bean color prevented acne...) What I do not understand is why multiple comparison corrections are an appropriate way to deal with this problem. The problem is always there.

$\endgroup$
  • $\begingroup$ Much more about this issue: stats.stackexchange.com/search?q=jelly+bean. $\endgroup$ – whuber Aug 18 '17 at 18:02
  • $\begingroup$ @whuber The jelly bean responses do not deal with these two questions I pose here. I get that if you do 20 tests one will be significant (green jelly beans cause acne). I state that in my introduction of the question. I don't understand why multiple comparison corrections are an appropriate solution to that problem. Please explai how this question is a duplicate of people not understanding the jelly bean comic. I understand the jelly bean comic. $\endgroup$ – colin Aug 18 '17 at 18:08
  • 2
    $\begingroup$ This doesn't seem a duplicate question to me, but it does contain some straw-man arguments. We do adjust significance levels instead of p-values - I'd even say that's the default in my field. FDR q-values are not adjusted p-values. And a priori declaring that Type I errors are worse is the entire basis of NHST: to give a practical example, -omics experiments routinely test $>10^5$ markers, so declaring thousands of significant findings in each study would immediately flood any research community with false leads. $\endgroup$ – juod Aug 18 '17 at 18:32
  • 4
    $\begingroup$ Id like to point out that your sentence saying that a p-value is a straightforward statistic misinterprets the p-value. It is not the probability that a result would have been detected by chance. You need to add "assuming the null hypothesis" at the very least, and "equally or more extreme" to lock it down. $\endgroup$ – Matthew Drury Aug 18 '17 at 18:44
  • 4
    $\begingroup$ "...a far more effective strategy would be lowering your p-value threshold rather than ad hoc correcting your p-values." This is exactly what multiple comparison corrections are doing. $\endgroup$ – klumbard Aug 18 '17 at 19:05
5
$\begingroup$

In your point one you use the word "should" and that is the key, while everyone should understand this, most do not. Google for Andrew Gelman's blog (or other sources) for examples of cases where researchers who "should" know better don't and the reviewers that "should" know better don't, and the editors who "should" know better don't and even when this mistake is pointed out, they still don't. In the comic, we know that there were 20 tests done, but does it spell that out in the article? probably not. That is one of the problems is that many times multiple tests are done, but only the interesting ones are reported, so the reader does not know how many tests to adjust for unless the researcher specifically talks about them or does the adjustment. Even worse, some research is done by throwing in as many predictors as can be thought of and declaring "success" if at least one p-value is less than 0.05. If these researchers were not continually told that they needed to adjust for multiple comparisons then they would be rewarded for behaviors like throwing in extra, meaningless predictors, or breaking a single large study into several small studies, or other strategies that increase type I error rates and therefore give them a better chance of "success".

You ask why we multiply the p-value rather than divide the significance level. Mathematically it is equivalent, though conceptually there is a difference, and I agree with you that it makes much more conceptual sense to adjust the significance level, but apparently for most people it is easier to let the computer multiply the p-value than think about a different significance level (and in the case of FDR you would need to calculate a different significance level for each comparison).

You are right to think about both type I and type II errors, but type I error is much easier to control, so it tends to get much more attention. I did once need to respond to an article review that we intentionally did not control for multiple comparisons because this was a case where the type II error was much more serious than a type I error.

Here are a couple of articles that spell some of these issues out in more detail (and give good reasons why sometimes you don't want to adjust as well as times you do):

Multiplicity in randomised trials I: endpoints and treatments Kenneth F Schulz, David A Grimes. Lancet 2005; 365: 1591–95

Multiplicity in randomised trials II: subgroup and interim analyses Kenneth F Schulz, David A Grimes. Lancet 2005; 365: 1657–61

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.