# Comparing models: which model to choose?

I'm new to mixed effects modeling, so I need help understanding when it's appropriate to choose a model. So far I've been incrementally building my modeling with main effects and then adding in the interactions.

mod1 includes all 3 main effects and the 3-way interaction. When I write the model out like this, the 3-way interaction is significant. In mod2, only session:trialtype is significant.

mod1 <- lmer(rt ~ group + session + trialtype +
group:session:trialtype + (1 | subject),
REML = FALSE, data = data)

mod2 <- lmer(rt ~ group * session * trialtype + (1 | subject),
REML = FALSE, data = data)


Output for mod2

anova(mod2)
Type III Analysis of Variance Table with Satterthwaite's method
Sum Sq Mean Sq NumDF DenDF  F value    Pr(>F)
group                      16431    8215     2    90   0.8911   0.41380
session                   211734  211734     1   630  22.9657 2.059e-06 ***
trialtype               14558623 4852874     3   630 526.3673 < 2.2e-16 ***
group:session              35237   17618     2   630   1.9110   0.14879
group:trialtype            14767    2461     6   630   0.2670   0.95223
session:trialtype          84680   28227     3   630   3.0616   0.02766 *
group:session:trialtype    39074    6512     6   630   0.7064   0.64459


How do I decide which is the appropriate model? Or do I use a separate, simpler model that only includes session:trialtype interaction?

• Mod1 is problematic, because 3-way interaction is in the model and related 2-way interactions are not. Mod2 is good. If you want to simplify the model, fit a model without 3 way interaction, then check if it is to exclude the 2-way interaction. – user158565 Aug 12 '19 at 19:53

A couple of points:

• Even there are exceptions, the general rule involving interactions is that you need to include the lower order terms. That is, in a model in which you want to include a 3-way interaction, you also need to include the main effects, and all 2-way interactions. In that regard, model mod2 does that.
• In general interactions are complex terms to include in a model requiring most often a sufficiently large sample size to estimate them accurately. This holds for 2-way interactions. For a 3-way interaction the required sample size would be even larger. Hence, are you certain that you want to include it in your model?
• Assuming that you have prior reasons (and sufficient sample size) to include the 3-way interaction, you could first test whether including any interaction terms improves the model, i.e., to compare the interaction model with the additive model:

fm_add <- lmer(rt ~ group + session + trialtype + (1 | subject), REML = FALSE, data = data)

fm_int <- lmer(rt ~ group * session * trialtype + (1 | subject), REML = FALSE, data = data)

anova(fm_add, fm_int)

• If this test gives you a p-value smaller than 0.15, then there could be indications that some of the interaction terms may improve the fit of the model. You could then proceed to look which interaction terms improve the fit, y first testing the 3-way interaction, and then the 2-way interactions. It would be appropriate to correct the p-values of these post-hoc tests for multiple testing.

• (+1) I think you mean "mod1" in your first bullet rather than "mod2." Good answer otherwise. Important to note that if you intend to report p-values or confidence intervals in the end, doing a lot of model selection with, e.g., anova, will hurt the reliability of your inference. Best scientific practice, OP, is to specify a priori the model you believe makes scientific sense, and then only make small tweaks after you fit that model to improve poor aspects of the fit. – Sheridan Grant Aug 13 '19 at 6:24