5
$\begingroup$

I have four questions:

  1. If i have to run a test on 4 colors; red, blue, green, yellow. I ran the same test 4 times (one time on each color). (H1~H4) If i already have a predefined or existing reason to do each of these tests (not exploratory), do i have to correct for multiple comparison (no. of H/4)? What If it was exploratory?
  2. If i have only 2 colored flowers; Black flowers and White flowers. I ran the same test 2 times (one time on each color). (H1~H2) But then i found something interesting to further investigate. E.g. the flower stem and leaves. If i were to correct for multiple comparisons for the stem and leaves. Do i divide the p-value by 2? or 4?
  3. Which is better to use for the above Bonferroni or Holm-Bonferroni?
  4. If i am correcting using the Holm-Bonferroni. Is it possible to report adjusted p values? If yes, how do you do that? e.g. in Bonferroni you multiply by the no. of tests run.
$\endgroup$
1
  • $\begingroup$ I have one final Q. If i am correcting using the Holm-Bonferroni. Is it possible to report adjusted p values? If yes, how do you do that? e.g. in Bonferroni you divide by the no. of tests run. $\endgroup$ Jan 9 '18 at 5:07
4
$\begingroup$
  1. It depends on context. Let us say you really want to write an article about how males vs females react to colors. If you are going to publish if you find a significant difference for at least one color then you absolutely need to correct for multiple testing. Usually even one significant result is considered interesting, it is almost always better to correct.
  2. For your original hypotheses, you count only the number original hypotheses. For any data-driven hypotheses you are out of luck. You will not be able to correct for multiple testing after having explored the data. You would have to put the number of all interesting things you might potentially have noticed in, and that is not quantifiable. In general, you should test your new hypotheses in a new experiment with new data.
  3. Holm-Bonferroni is always superior. It has more power and still strictly controls the familywise error rate.
  4. For Bonferroni you multiply (not divide) by the number of p-values to get adjusted p-values. Holm-Bonferroni is slightly more complicated to correct, but most statistical packages have functions for that and you can find the description of the procedure in the article by Wright here: Adjusted P-Values for Simultaneous Inference.
$\endgroup$
1
  • 1
    $\begingroup$ Thank you Erik. Ok so for number one, you mean that you have to correct for multiple comparison whether you have a predefined hypothesis or not as long as you get one significant result, right? In #2, you mean that i can't even explore these things in my current study? Even if the flower population is rare? Sorry i don't get what you mean by what you said. Could you please elaborate? Why can't i do post hoc analyses, if i already have the data, why should it be a new data and a new study? #3, #4 Got it, thanks! $\endgroup$ Jan 12 '18 at 8:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.