# does comparing models make sense?

Let's say I have 3 factors f1, f2 and f3 and fit these models

m1 <- lmer(y ~ f1 + (1|sub))
m2 <- lmer(y ~ f1*f2 + (1|sub))
m3 <- lmer(y ~ f1*f2*f3 + (1|sub))


Does it make sense to compare the models with anova(m1,m2,m3)? Or would you first test every single step

m1 <- lmer(y ~ f1 + (1|sub))
m2 <- lmer(y ~ f1 + f2 + (1|sub))
m3 <- lmer(y ~ f1 + f2 + f3 + (1|sub))
m4 <- lmer(y ~ f1*f2 + f3 + (1|sub))
etc


I suppose the second example is correct. But what if m1 and m2 do not differ statistically? Does that mean I should not include f2 and f3 in my model? But what if for example f2 and f3 are significant in model m3? This happens with my real data, for example m1 is not significantly different from m2, but f1 interacts significantly with f2 when the interaction is added to the model. I just don't see the point in comparing models then.

• compare for what purpose? if it's forecasting performance then you compare anything with anything Nov 15, 2018 at 1:10

It would be better to perform the omnibus test for all interactions together, i.e.,

fm_additive <- lmer(y ~ f1 + f2 + f3 + (1 | sub))
fm_inter <- lmer(y ~ f1 * f2 * f3 + (1 | sub))

The second approach you describe is called stepwise regression and is not a good idea. You should base the inclusion of variables first and foremost on theoretical basis. If you believe f1, f2 and f3 are only meaningful if their interaction with one another is included, then the first method for comparing them would be better.
• Thanks @Frans Rodenburg but the first method seems strange to me because m1 <- y ~ f1 and m2 <- y ~ f1*f2 differ not only in that m2 has f2 in addition to f1 but also that m2 has all interactions that are not included in m1. So it seems that we are comparing very different models, does that make sense? Nov 28, 2018 at 0:19