I am wary of stepping into the civil war between Fisher vs Neyman-Pearson interpretations of $p$-value (which has been well elucidated here and here), but I've been pondering a question that I keep going in circles on. Is it appropriate to make a correction for multiple comparisons to a $p$-value in a Fisher paradigm of interpreting $p$-values?

Now, as I understand it, corrections for multiple comparisons are officially made to $\alpha$, not to $p$-values. For example, Bonferroni's correction is $\alpha/k$, where $k$ is the number of comparisons. But it's an easy switch to multiply the $p$-value by $k$, and equivalent conclusions are drawn in a Neyman-Pearson framework (it's also easier to present to folks more comfortable seeing $\alpha=0.05$).

However, the Fisher framework doesn't have an $\alpha$ value. $P$-value is thought more as evidence against the null hypothesis instead of a hard decision criterion. Given that, is it still reasonable to correct this $p$-value for multiple comparisons?

For some additional context, I work in a drug discovery oriented environment, and the necessary conditions for using Neyman-Pearson (particularly power requirements) aren't met. The Fisher framework seems much more appropriate to guide pursuing promising treatments (in a discovery mode, not a confirmatory mode). An example would be if there are $5$ potential treatment options compared to a control, and the best option showed a clinically meaningful difference with significance at $p=0.045$. If that isn't corrected, it seems promising and should be pursued. If I correct it using Bonferroni however, $p=0.225$, and I would likely go back to the drawing board and throw out all $5$ treatments.

[Note: I don't have to correct using Bonferroni either; it's just the easiest to use an example. I'm more interested in the theory of applying corrections for multiple comparisons and family-wise error rates.]


Following @MichaelLew's answer (+1), I changed my point of view to the opposite one; now I think that $p$-values should NOT be corrected. I have reworked my answer.

To make the discussion more lively, I will refer to the famous XKCD comic where $20$ colours of jelly beans are independently tested to be linked to acne, and green jelly beans yield $p<0.05$; for concreteness, let us assume it was $p=0.02$:

green jelly beans

The Fisher approach is to consider $p$-value as quantifying the strength of evidence, or rather as a measure of surprise ("surprisingness") -- I like this expression and find it intuitively clear and the same time quite precise. We pretend that the null is true and quantify how surprised we should then be to observe such results. This yields a $p$-value. In the "hybrid" Fisher-Neyman-Pearson approach, if we are surprised more than some chosen surprisingness threshold ($p<\alpha$) then we additionally call the results "significant"; this allows to control type I error rate.

Importantly, the threshold should represent our prior beliefs and expectations. For example, "extraordinary claims require extraordinary evidence": we would need to be very surprised to believe the evidence of e.g. clairvoyance, and so would like to set a very low threshold.

In the jelly beans example, each individual $p$-value reflects the surprisingness of each individual correlation. Bonferroni correction replaces $\alpha$ with $\alpha/k$ to control the overall type I error rate. In the first version of this answer, I argued that we should also be less surprised (and should consider that we have less evidence) by getting $p=0.02$ for green jelly beans if we know that we ran $20$ tests, hence Fisher's $p$-values should also be replaced with $kp$.

Now I think it is wrong, and $p$-values should not be adjusted.

First of all, let's point out that for the hybrid approach to be coherent, we cannot possibly adjust both, $p$-values and $\alpha$ threshold. Only one or another can be adjusted. Here are two arguments for why it should be $\alpha$.

  1. Consider exactly the same jelly beans setting, but now we a priori expected that green jelly beans are likely to be linked to acne (say, somebody suggested a theory with this prediction). Then we would be happy to see $p=0.02$ and would not make any adjustments to anything. But nothing about the experiment has changed! If $p$-value is a measure of surprisingness (of each individual experiment), then $p=0.02$ should stay the same. What changes is our $\alpha$, and it is only natural, because as I argued above, the threshold always in one way or another reflects our assumptions and expectations.

  2. $P$-value has a clear interpretation: it is a probability of obtaining the observed (or even less favorable) results under the null hypothesis. If there is no link between green jelly beans and acne, then this probability is $p=0.02$. Replacing it with $kp=20\cdot 0.02=0.4$ ruins this interpretation; this is now not a probability of anything anymore. Moreover, imagine that not the $20$ colours were tested, but $100$. Then $kp=2$, which is larger than $1$, and obviously cannot be a probability. Whereas reducing $\alpha$ by $100$ still makes sense.

To put it in terms of evidence, the "evidence" that green jelly beans are linked to acne is measured as $p=0.02$ and that's that; what changes depending on the circumstances (in this case, on the number of performed tests), is how we treat this evidence.

I should stress that "how we treat the evidence" is something that is very much not fixed in the Fisher's framework either (see this famous quote). When I say that $p$-values should better not be adjusted, it does not mean that Sir Ronald Fisher would look at $p=0.02$ for green jelly beans and consider that a convincing result. I am sure he would still be wary of it.

Concluding metaphor: the process of cherry picking does not modify the cherries! It modifies how we treat these cherries.

  • $\begingroup$ I came to a similar thought myself. If I make 5 comparisons, I should be less surprised at a small p-value than if I just made one. $\endgroup$ – Ashe Feb 26 '15 at 13:57
  • 1
    $\begingroup$ @amoeba Those arguments against adjustment seem pretty sound to me, but note that questions of how to calculate P-values should be considered in light of how the P-values are going to be used. If they are going to be plugged into a thoughtless NHST algorithm then the adjusted P-values are probably safer then the unadjusted. However, thoughtful inferences should be informed by the unadjusted P-values as well as the characteristics of the experimental design and analytical method. $\endgroup$ – Michael Lew Feb 28 '15 at 22:48
  • 2
    $\begingroup$ You both hit on the mental struggle I was going back and forth on. I very much appreciate @Michael's way of wording it (experimental method vs evidential p-values). Adjustments are meant to control that long-running Type I error rate, yet I feel this need to be less surprised if I make k comparisons. Perhaps that is better done more informally in the process weighing all the evidence together (P-value, effect size, design, sampling method, etc) rather than formally by adjusting a P-value. $\endgroup$ – Ashe Mar 2 '15 at 15:58
  • 1
    $\begingroup$ @Joshua: yes. The ultimate reason why you feel the need to be less surprised if you make $k$ comparisons, is that you are looking at the lowest $p$-value across all $k$ of them and then a posteriori think about its surprisingness. As you of course know, this is called cherry picking. What Michael and me are essentially saying, is that the process of cherry picking does not modify the cherries! Instead, it modifies how you should treat these cherries. $\endgroup$ – amoeba Mar 2 '15 at 16:04
  • 1
    $\begingroup$ Yes, I like that very much. This thread has been quite helpful. Much obliged. $\endgroup$ – Ashe Mar 2 '15 at 16:27

Amoeba's answer is good, but it not the answer that I would give, as he notes.

The answer is, of course, it depends. It depends on whether you want the P-value to be conditioned on the actual results of the particular comparison or to be conditioned additionally on the number of comparisons that you have made. In the former case you do not need to adjust the P-value for the multiplicity. In the latter case you should do so.

Why would you want to condition on the number of comparisons? To allow an algorithm-based decision process to provide a guarantee regarding the long term rate of a false positive errors.

Why would you not want to condition on the number of comparisons? To allow the P-value to represent the evidence in the particular experimental results of interest without being modified by the presence of other comparisons involving other data.

The long run error rate is a property of the method, and the evidence is a property of the particular set of data under consideration. A frequentist adjustment of the P-value for multiplicity treats the method properties as more important than the evidential meaning of the P-value. It is my opinion that the performance of the method that yielded the data in question is a useful piece of information to have when making inference from the evidence, but it should be kept as a separate piece of information rather than being rolled into the evidence by `correction' of the P-value. To roll it into the evidence is to modify the evidence in a manner that takes the responsibility for inference away from the analyst.

  • $\begingroup$ +1. Thanks for joining the discussion, @Michael. I must say that your answer made me re-think the whole issue and I have actually changed my mind. I think I agree with you now! I have re-written my answer. $\endgroup$ – amoeba Feb 28 '15 at 17:08
  • $\begingroup$ Thank you for chiming in. The way you word it (algorithm-based decision process vs individual experimental evidence) is very helpful. $\endgroup$ – Ashe Mar 2 '15 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.