Multiple comparison correction (Bonferroni vs. Holm Bonferroni) I have four questions:


*

*If i have to run a test on 4 colors; red, blue, green, yellow.  I
ran the same test 4 times (one time on each color). (H1~H4) If i
already have a predefined or existing reason to do each of these
tests (not exploratory), do i have to correct for multiple
comparison (no. of H/4)? What If it was exploratory?

*If i have only 2 colored flowers; Black flowers and White flowers. I
ran the same test 2 times (one time on each color). (H1~H2) But then
i found something interesting to further investigate. E.g. the
flower stem and leaves. If i were to correct for multiple comparisons for the stem and leaves. Do i divide the p-value by 2? or 4?

*Which is better to use for the above Bonferroni or Holm-Bonferroni?

*If i am correcting using the Holm-Bonferroni. Is it possible to report adjusted p values? If yes, how do you do that? e.g. in Bonferroni you multiply by the no. of tests run.

 A: *

*It depends on context. Let us say you really want to write an article about how males vs females react to colors. If you are going to publish if you find a significant difference for at least one color then you absolutely need to correct for multiple testing. Usually even one significant result is considered interesting, it is almost always better to correct.  

*For your original hypotheses, you count only the number original
hypotheses. For any data-driven hypotheses you are out of luck. You
will not be able to correct for multiple testing after having
explored the data. You would have to put the number of all
interesting things you might potentially have noticed in, and that
is not quantifiable. In general, you should test your new hypotheses
in a new experiment with new data.

*Holm-Bonferroni is always superior. It has more power and still
strictly controls the familywise error rate.

*For Bonferroni you multiply (not divide) by the number of
p-values to get adjusted p-values. Holm-Bonferroni is slightly more
complicated to correct, but most statistical packages have functions for
that and you can find the description of the procedure in the
article by Wright here: Adjusted P-Values for Simultaneous
Inference.

