# lmer: comparing model fit and null hypothesis significance testing

We recently used lmer to analyze some reaction time data. There were three fixed effects variables, and the model included their interactions and a fully specified random effects structure. Something like

lm1 <- lmer(RT ~ A*B*C + (1+(A*B*C)|participant) + (1+(A*B*C)|item), data, REML = TRUE)

We found a three-way interaction, and interpreted it in a way that I thought was reasonable. However, a referee commented that, in order to interpret the coefficients associated with the interaction, we first has to compare nested models in order to ascertain the necessity of the different terms starting from the highest order interaction (and report the log likelihood Chi Square for the stats).

My understanding was that assessing nested models via likelihood ratio tests (to determine the best fitting model) and null hypothesis significance testing (e.g., to see if factor C has any effect) were separate issues. For example:

"LR tests can assess the significance of particular factors or, equivalently, choose the better of a pair of nested models, but some researchers have criticized model selection via such pairwise comparisons as an abuse of hypothesis testing..." from Bolker et al. (2008).

I know that something like lmerTest would do this kind of thing automatically, but I want to be sure I understand this. Does the referee comment make sense? Is this a disputed topic? Are there recommendations for further reading that I can do on this issue?

I would be rather more concerned that you are working with a maximal model which may have converged to a degenerate solution. Do you really need random coefficients for all 3 main effects and their interactions on both grouping factors ? What kind of variables are A, B and C? The advice to "keep it maximal" is very often very bad advice and has been shown to lead to incorrect inferences. See Bates et al 2015 for an in depth discussion (note that Douglas Bates was the original author of the lme4 package you are using.