I have run an experiment with a 2 x 2 factorial design. From what I understand there is no need to adjust the alpha-level for multiple comparisons because all main-effects and interactions are planned.

However I have had a reviewer come back and suggest analyses of some simple effects and as a result I am starting to wonder about error-correction for multiple post-hoc comparisons.

My question is when calculating something like a Bonferroni error correction, how many comparisons should be in the formula? Should I only correct for the post-hoc tests or does doing post-hoc tests mean I should include all planned contrasts as well? So, if performing the bonferroni procedure, what should the k in $\alpha$/$k$ be for a 2 x 2 ANOVA with 2 tests of simple effects?


It's not so straightforward to answer because developing formal rules for the number of corrections would require to invoke Bayesian approaches.

Basically, the idea is not that you correct on whether a test was "planned" or not. Corrections should be done for all tests, if you want to maintain the type I error across the experiment. However, if you add known predictors to your model, you are pretty sure that the null hypothesis is false for that predictor, so you do not expect that test to contribute to the overall type I error. Blind post-hoc tests, on the other hand, are not supported by external evidence, and you would be more strict towards them.

To give an applied example, in large-throughput biology, an experiment can be designed to test 20,000 genes - all of these tests are planned, yet you apply a correction because the probability of null being true is high for each gene. In an epidemiological analysis of lung cancer one would include smoking without worrying about correcting for that predictor.

In short, I would suggest that you correct for the number of post-hoc tests only, and discuss any further issues in the text of the paper.

  • $\begingroup$ Thank you @juod. The reviewer's suggestions are not blind in the sense that there are differences that appear on the graph, but they are blind in the sense that they were not originally planned and are not justified given that there is no two-way interaction between the two categorical predictors. In other words these tests would be well and truly post-hoc. So would you suggest $\alpha$/2 then if there are two of these thoroughly post-hoc comparisons? $\endgroup$ – llewmills Feb 28 '17 at 13:33
  • 1
    $\begingroup$ If there's only two of these comparisons, I actually would present the uncorrected p-values, and then discuss the possible interpretations more extensively. Something like, "the obtained p-values (0.042 and 0.035) were significant, but note that two of the tests were post-hoc and the significance would be lost if a Bonferroni correction for the 2 tests were applied". It is much clearer to present all p-values as uncorrected, and the reader can then divide by 2 himself, if he considers it necessary. $\endgroup$ – juod Feb 28 '17 at 14:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.