Comparing models: which model to choose?

Question

I'm new to mixed effects modeling, so I need help understanding when it's appropriate to choose a model. So far I've been incrementally building my modeling with main effects and then adding in the interactions.

mod1 includes all 3 main effects and the 3-way interaction. When I write the model out like this, the 3-way interaction is significant. In mod2, only session:trialtype is significant.

mod1 <- lmer(rt ~ group + session + trialtype + 
             group:session:trialtype + (1 | subject), 
             REML = FALSE, data = data)

mod2 <- lmer(rt ~ group * session * trialtype + (1 | subject), 
             REML = FALSE, data = data)

Output for mod2

anova(mod2)
Type III Analysis of Variance Table with Satterthwaite's method
                          Sum Sq Mean Sq NumDF DenDF  F value    Pr(>F)    
group                      16431    8215     2    90   0.8911   0.41380    
session                   211734  211734     1   630  22.9657 2.059e-06 ***
trialtype               14558623 4852874     3   630 526.3673 < 2.2e-16 ***
group:session              35237   17618     2   630   1.9110   0.14879    
group:trialtype            14767    2461     6   630   0.2670   0.95223    
session:trialtype          84680   28227     3   630   3.0616   0.02766 *  
group:session:trialtype    39074    6512     6   630   0.7064   0.64459 

How do I decide which is the appropriate model? Or do I use a separate, simpler model that only includes session:trialtype interaction?

Answer 1

A couple of points:

• Even there are exceptions, the general rule involving interactions is that you need to include the lower order terms. That is, in a model in which you want to include a 3-way interaction, you also need to include the main effects, and all 2-way interactions. In that regard, model mod2 does that.
• In general interactions are complex terms to include in a model requiring most often a sufficiently large sample size to estimate them accurately. This holds for 2-way interactions. For a 3-way interaction the required sample size would be even larger. Hence, are you certain that you want to include it in your model?

• Assuming that you have prior reasons (and sufficient sample size) to include the 3-way interaction, you could first test whether including any interaction terms improves the model, i.e., to compare the interaction model with the additive model:

  fm_add <- lmer(rt ~ group + session + trialtype + (1 | subject), REML = FALSE, data = data)

  fm_int <- lmer(rt ~ group * session * trialtype + (1 | subject), REML = FALSE, data = data)

  anova(fm_add, fm_int)

• If this test gives you a p-value smaller than 0.15, then there could be indications that some of the interaction terms may improve the fit of the model. You could then proceed to look which interaction terms improve the fit, y first testing the 3-way interaction, and then the 2-way interactions. It would be appropriate to correct the p-values of these post-hoc tests for multiple testing.