how to perform multiple comparisons on a linear mixed model with unbalanched set of groups

Question

I have built a linear mixed model (rt.m6) with 2 fixed factors, namely condition and suffix, which also interact with each other. However, my experimental design isn't fully crossed, meaning that not all suffixes are included in all the experimental conditions. When running the model (rt.m6), R is dropping 1 column/coefficient because of that.

rt.m6 <- lmer (RT ~ condition * suffix + (1 | num_item) + (1 | subject), data= data_trimmed_RT, REML = F)

The problem occurs when I'm trying to perform multiple comparisons via emmeans including only one of the fixed factors, the one of the condition. For the condition that doesn't include all the suffixes, comparisons cannot be made, and the output in noted as NA.

pairs(emmeans(rt.m6, "condition", simple = "each", lmer.df = "asymp"))

> > pairs(emmeans(rt.m6, "condition", simple = "each", lmer.df = "asymp")) NOTE: Results may be misleading due to involvement in interactions  
contrast             estimate  SE  df z.ratio p.value 
> ArgStrViol - AspViol   nonEst  NA  NA      NA      NA  
> ArgStrViol - CatViol    499.9 164 Inf   3.053  0.0192  
> ArgStrViol - GramW     458.2 158 Inf   2.903  0.0304  
> ArgStrViol - Novel     -138.5 171 Inf  -0.808  0.9284  
> AspViol - CatViol      nonEst  NA  NA      NA      NA  
> AspViol - GramW        nonEst  NA  NA      NA      NA  
> AspViol - Novel        nonEst  NA  NA      NA      NA  
> CatViol - GramW         -41.7 150 Inf -0.278  0.9987  
> CatViol - Novel        -638.4 166 Inf  -3.839  0.0012  
> GramW - Novel          -596.7 159 Inf  -3.752  0.0016
> 
> Results are averaged over the levels of: suffix  Degrees-of-freedom
> method: asymptotic  P value adjustment: tukey method for comparing a
> family of 5 estimates

Do you know how to solve this?

Answer 1

You seem to not have taken adequate note of this message:

NOTE: Results may be misleading due to involvement in interactions

You have a model that includes the interaction of condition and suffix. If the interaction is at all important, it means that the conditions compare differently at each suffix -- and hence that you probably should not be averaging over suffix, as was done.

Accordingly, I invite you to consider doing the comparisons separately for each level of suffix:

EMM <- emmeans(rt.m6, "condition", by = "suffix", lmer.df = "asymp")
pairs(EMM)

This of course will flag as non-estimable the comparisons for which you have no data.

The simple = "each" specification shown applies only to cases where you obtain EMMs of combinations of two or more factors; and that does not seem to be what you are interested in.

If, and only if, the interaction effect is negligible, then you can re-fit the model with the interaction excluded, then do what you did before. You will not get the advisory message about interactons, because there is no interaction.