I use the emmeans
package for post-hoc pairwise comparisons with different types of models. Standard contrast families are e.g. pairwise
or dunnett
which would respectively compare each factor level to every other or each factor level to baseline level.
These families are very handy until you have more complex models with interactions between two terms. But this package (which is absolutely great BTW) allows for the definition of custom contrast families. I have defined such families but I am unsure about which adjustment for multiple comparisons I should set by default.
Here some examples about which comparisons are made:
letr <- factor(LETTERS[1:2])
num <- factor(1:3)
# within.pairwise~num*letr -> All pairwise comparisons within the second factor
# A1-A2, A1-A3, A2-A3
# B1-B2, B1-B3, B2-B3
# Would use tukey adjustment
# within.dunnett~num*letr -> All Dunnett comparisons within the second factor
# A1-A2, A1-A3
# B1-B2, B1-B3
# Would use dunnett adjustment
# within.between~num*letr -> All pairwise comparisons within both factors
# A1-A2, A1-A3, A2-A3
# B1-B2, B1-B3, B2-B3
# A1-B1, A2-B2, A3-B3
# Would use tukey adjustment
I know for example that with Holm adjustment few assumptions about the nature of the comparisons are made (e.g. in contrast to Šidák), but I tried to keep adjustment functions from the closest form of simple (with only one factor) multiple comparisons.
- Is this correct or are there assumptions of these methods I do not know about?
- Can you suggest some resources summarizing which adjustment methods are available and what their assumptions are?
- Should I change the methods used currently?
I hope the question is understandable (even for people not used to the R or emmeans package syntax). Thank you for your input!