A reviewer asked me to do multiple comparisons corrections for two-way and three-way interactions that showed significance in my manuscript. I did my data analysis with linear mixed model in R.

I had three different tasks for the same participants. Among 3 tasks, the other 3 outcomes were drawn from one task. So, I had 5 outcome measures. Then, my research questions of particular interest were two-way and three-way interactions within each outcome variable indicated by asterisks below. In this case, should I divide 0.05 by 10 (2 research questions x 5 outcomes) or by 6 (2 research questions x 3 outcomes drawn from the same task) if I apply the Bonferronni correction? Or should I just divide 0.05 by 5 (outcomes) or 3 (outcomes that were drawn from the same task)?

Variables entered in the models were SES, Condition, Group, Time, Time x Condition, Group x Time, Condition X Group**, Time x Condition x Group**.

Also, is there any other powerful but less conservative method that I can easily apply for repeated measure analysis? (I was told that Holm-Bonferroni is more powerful than Boferroni, but in my case, the results were very similar).

  • $\begingroup$ I'm concerned about the design: you have a well controlled experiment. But your analysis loses focus when you separately present statistical results for two model parameters when your overarching interest seems to be whether "condition" interacts with either group or the group-by-time growth/interaction term. Why not perform the 2 degree of freedom test for nested models taking as a null model the same as the above but omitting the 2- and 3- way interactions with condition? This is a kind of interrupted time series analysis, and is more powerful as a single test. $\endgroup$
    – AdamO
    Feb 8, 2018 at 17:20
  • $\begingroup$ Thank you for suggesting the other analyses. Yes, my research interests are whether condition interacts with group or with group and time. I do not understand what you meant by 2 degree of freedom test for nested models. Would you mind sharing R scripts or references that I can look into? Also, I do not think my analyses are inappropriate. Given I use the above analyses, what would be the answer for the above questions in the second paragraph? Thank you! $\endgroup$
    – Grace
    Feb 9, 2018 at 21:28
  • $\begingroup$ Let me be clear: I'm suggesting this analysis as a omnibus not as another test. Searches for "interrupted time series" will be fruitful on this site and elsewhere. This post I thought was well written. The basic way you would test the models is: aov(lm(outcome ~ condition*time*group), lm(outcome ~ condition+time*group). or replace lm with lmer, gee, or whatever modeling approach you've used. $\endgroup$
    – AdamO
    Feb 9, 2018 at 22:03

1 Answer 1


Dunn's 'Bonferroni adjustment' method is almost 60 years old at this point, being first published in 1961. The Bonferroni adjustment is commonly taught as a way to control the family-wise error rate (FWER) when introducing the notion of multiple comparisons and Type I error inflation. Holm's step-up procedure, developed about 40 years ago, was an improvement on the Bonferroni adjustment in terms of statistical power, but this improvement really only starts to show when the number of comparisons gets large.

About two decades ago some problems with the FWER—namely that 'family' has no formal definition, and that rejection probabilities are dependent on family size—were pointed out by Benjamini and Hochberg in a more refined approach to multiple comparisons adjustments termed the false discovery rate (FDR). Benjamini and Hochberg's step-down procedure has more statistical power than the Holm adjustment, does not require a definition of family, and is scalable across different numbers of comparisons, as when adding new comparisons to an existing set of comparisons.

Important note: adjusted p-values alone resulting from step-up and step-down multiple comparisons adjustments are insufficient to determine rejections, as these methods also employ stopping rules that depend on the original—unadjusted—ordering of test p-values.

As with all multiple comparisons adjustments, the Benjamini and Hochberg FDR method can be performed either as an adjustment to the rejection criterion $\alpha$, or as an adjustment to the p-values. Here's how to perform the way:

  1. Compute the p-value for each of $m$ comparisons as you would for a single test. These are unadjusted p-values. (I am assuming they are "two-sided" p-values corresponding to something like $p=P(|T|\ge|t|)$.)

  2. Order the $m$ p-values from largest to smallest (hence "step-down").

  3. For the first pairwise comparison ($i=1$), compare the p-value to $(\alpha)\times((m+1-1)/m)$

  4. For the second pairwise comparison ($i=2$), compare the p-value to $(\alpha)\times((m+1-2)/m)$

  5. For the $i^{\text{th}}$ pairwise comparison, compare the p-value to $(\alpha)\times((m+1-i)/m)$

  6. Reject null hypotheses for all tests including and following the first test (stepping down) for which we reject the null hypothesis.


Benjamini, Y. and Hochberg, Y. (2000). On the adaptive control of the false discovery rate in multiple testing with independent statistics. Journal of Educational and Behavioral Statistics, 25(1):60–83.

Dunn, O. J. (1961). Multiple comparisons among means. Journal of the American Statistical Association, 56(293):52–64.

Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(65-70):1979.

  • $\begingroup$ Thank you for your explanation. With regard to the number of comparisons, what would be the appropriate number for m? Please see the above questions in the second paragraph. $\endgroup$
    – Grace
    Feb 9, 2018 at 21:22
  • $\begingroup$ @Grace $m$ = the number of p-values you are comparing... literally the number of tests you conducted. $\endgroup$
    – Alexis
    Feb 9, 2018 at 22:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.