multiple correction comparisons (Interactions)

Question

A reviewer asked me to do multiple comparisons corrections for two-way and three-way interactions that showed significance in my manuscript. I did my data analysis with linear mixed model in R.

I had three different tasks for the same participants. Among 3 tasks, the other 3 outcomes were drawn from one task. So, I had 5 outcome measures. Then, my research questions of particular interest were two-way and three-way interactions within each outcome variable indicated by asterisks below. In this case, should I divide 0.05 by 10 (2 research questions x 5 outcomes) or by 6 (2 research questions x 3 outcomes drawn from the same task) if I apply the Bonferronni correction? Or should I just divide 0.05 by 5 (outcomes) or 3 (outcomes that were drawn from the same task)?

Variables entered in the models were SES, Condition, Group, Time, Time x Condition, Group x Time, Condition X Group**, Time x Condition x Group**.

Also, is there any other powerful but less conservative method that I can easily apply for repeated measure analysis? (I was told that Holm-Bonferroni is more powerful than Boferroni, but in my case, the results were very similar).

Answer 1

Dunn's 'Bonferroni adjustment' method is almost 60 years old at this point, being first published in 1961. The Bonferroni adjustment is commonly taught as a way to control the family-wise error rate (FWER) when introducing the notion of multiple comparisons and Type I error inflation. Holm's step-up procedure, developed about 40 years ago, was an improvement on the Bonferroni adjustment in terms of statistical power, but this improvement really only starts to show when the number of comparisons gets large.

About two decades ago some problems with the FWER—namely that 'family' has no formal definition, and that rejection probabilities are dependent on family size—were pointed out by Benjamini and Hochberg in a more refined approach to multiple comparisons adjustments termed the false discovery rate (FDR). Benjamini and Hochberg's step-down procedure has more statistical power than the Holm adjustment, does not require a definition of family, and is scalable across different numbers of comparisons, as when adding new comparisons to an existing set of comparisons.

Important note: adjusted p-values alone resulting from step-up and step-down multiple comparisons adjustments are insufficient to determine rejections, as these methods also employ stopping rules that depend on the original—unadjusted—ordering of test p-values.

As with all multiple comparisons adjustments, the Benjamini and Hochberg FDR method can be performed either as an adjustment to the rejection criterion $\alpha$, or as an adjustment to the p-values. Here's how to perform the way:

1. Compute the p-value for each of $m$ comparisons as you would for a single test. These are unadjusted p-values. (I am assuming they are "two-sided" p-values corresponding to something like $p=P(|T|\ge|t|)$.)

2. Order the $m$ p-values from largest to smallest (hence "step-down").

3. For the first pairwise comparison ($i=1$), compare the p-value to $(\alpha)\times((m+1-1)/m)$

4. For the second pairwise comparison ($i=2$), compare the p-value to $(\alpha)\times((m+1-2)/m)$

5. For the $i^{\text{th}}$ pairwise comparison, compare the p-value to $(\alpha)\times((m+1-i)/m)$

6. Reject null hypotheses for all tests including and following the first test (stepping down) for which we reject the null hypothesis.

References

Benjamini, Y. and Hochberg, Y. (2000). On the adaptive control of the false discovery rate in multiple testing with independent statistics. Journal of Educational and Behavioral Statistics, 25(1):60–83.

Dunn, O. J. (1961). Multiple comparisons among means. Journal of the American Statistical Association, 56(293):52–64.

Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(65-70):1979.