Adjustment for multiple comparisons in complex cases

Question

I use the emmeans package for post-hoc pairwise comparisons with different types of models. Standard contrast families are e.g. pairwise or dunnett which would respectively compare each factor level to every other or each factor level to baseline level.

These families are very handy until you have more complex models with interactions between two terms. But this package (which is absolutely great BTW) allows for the definition of custom contrast families. I have defined such families but I am unsure about which adjustment for multiple comparisons I should set by default.

Here some examples about which comparisons are made:

letr <- factor(LETTERS[1:2])
num <- factor(1:3)

# within.pairwise~num*letr -> All pairwise comparisons within the second factor
# A1-A2, A1-A3, A2-A3
# B1-B2, B1-B3, B2-B3
# Would use tukey adjustment

# within.dunnett~num*letr -> All Dunnett comparisons within the second factor
# A1-A2, A1-A3
# B1-B2, B1-B3
# Would use dunnett adjustment

# within.between~num*letr -> All pairwise comparisons within both factors
# A1-A2, A1-A3, A2-A3
# B1-B2, B1-B3, B2-B3
# A1-B1, A2-B2, A3-B3
# Would use tukey adjustment

I know for example that with Holm adjustment few assumptions about the nature of the comparisons are made (e.g. in contrast to Šidák), but I tried to keep adjustment functions from the closest form of simple (with only one factor) multiple comparisons.

1. Is this correct or are there assumptions of these methods I do not know about?
2. Can you suggest some resources summarizing which adjustment methods are available and what their assumptions are?
3. Should I change the methods used currently?

I hope the question is understandable (even for people not used to the R or emmeans package syntax). Thank you for your input!

Answer 1

I suggest using the “mvt” adjustment. It is based on the multivariate t distribution and is the exact one-step adjustment when the estimates have a multivariate normal distribution with the given correlation structure. So when you are looking at all pairwise comparisons, mvt is equivalent to Tukey; and when you are looking at each treatment versus a control, mvt is equivalent to Dunnett.

Note however that the calculations are done using a Monte Carlo method, so the results differ slightly if you repeat the analysis.

For more info, see the documentation in the multcomp and mvtnorm packages