Dunn's 'Bonferroni adjustment' method is almost 60 years old at this point, being first published in 1961. The Bonferroni adjustment is commonly taught as a way to control the family-wise error rate (FWER) when introducing the notion of multiple comparisons and Type I error inflation. Holm's step-up procedure, developed about 40 years ago, was an improvement on the Bonferroni adjustment in terms of statistical power, but this improvement really only starts to show when the number of comparisons gets large.
About two decades ago some problems with the FWER—namely that 'family' has no formal definition, and that rejection probabilities are dependent on family size—were pointed out by Benjamini and Hochberg in a more refined approach to multiple comparisons adjustments termed the false discovery rate (FDR). Benjamini and Hochberg's step-down procedure has more statistical power than the Holm adjustment, does not require a definition of family, and is scalable across different numbers of comparisons, as when adding new comparisons to an existing set of comparisons.
Important note: adjusted p-values alone resulting from step-up and step-down multiple comparisons adjustments are insufficient to determine rejections, as these methods also employ stopping rules that depend on the original—unadjusted—ordering of test p-values.
As with all multiple comparisons adjustments, the Benjamini and Hochberg FDR method can be performed either as an adjustment to the rejection criterion $\alpha$, or as an adjustment to the p-values. Here's how to perform the way:
Compute the p-value for each of $m$ comparisons as you would for a single test. These are unadjusted p-values. (I am assuming they are "two-sided" p-values corresponding to something like $p=P(|T|\ge|t|)$.)
Order the $m$ p-values from largest to smallest (hence "step-down").
For the first pairwise comparison ($i=1$), compare the p-value to $(\alpha)\times((m+1-1)/m)$
For the second pairwise comparison ($i=2$), compare the p-value to $(\alpha)\times((m+1-2)/m)$
For the $i^{\text{th}}$ pairwise comparison, compare the p-value to $(\alpha)\times((m+1-i)/m)$
Reject null hypotheses for all tests including and following the first test (stepping down) for which we reject the null hypothesis.
References
Benjamini, Y. and Hochberg, Y. (2000). On the adaptive control of the false discovery rate in multiple testing with independent statistics. Journal of Educational and Behavioral Statistics, 25(1):60–83.
Dunn, O. J. (1961). Multiple comparisons among means. Journal of the American Statistical Association, 56(293):52–64.
Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(65-70):1979.
aov(lm(outcome ~ condition*time*group), lm(outcome ~ condition+time*group)
. or replacelm
withlmer
,gee
, or whatever modeling approach you've used. $\endgroup$