Model simplification I am currently looking at a linear mixed model of with the formula x ~ y * z
I'm struggling with simplifying the model.


*

*When I run an ANOVA of my model it said there was no interaction so I reduced the model to remove the interaction term.

*However when I simplify my model I find my $R^2$ value drops and the significance values change (e.g., stuff becomes significant that isn't significant in my full model). To add to my confusion there actually is an interaction between one of my z categories and one of my y values! Despite this, my AICc value states that the reduced model is better...!


I've been made aware of the issues in model simplification giving false effects and was wondering if it's best to just keep my full model?
 A: A couple of points:


*

*You have not stated what test exactly are you performing when you run ANOVA. I would suggest that you best fit your linear mixed model using restricted maximum likelihood (REML), which is the default in most software, and then you do an F-test for the interaction using the Satterthwaite's degrees of freedom. In R you can do this using the lmerTest package. This will work better especially if you have a relatively small sample size.

*The coefficients for the variables y and z have a different interpretation when the interaction y:z is in or out of the model. Hence, it could be well the case that their statistical significance changes. Moreover, instead of looking only at statistical significance, you should also have a look at practical significance, and see what the two models say for your outcome. For example, doing an effects plots using the effects package in R. 

*The $R^2$ is a bit controversial outside the case of simple linear regression. There are several versions of it for (linear) mixed effects models, each one doing something different.

*Likewise it is not clear which version of the AICc you are using. Note that the standard version of it only works for univariate normal data, and therefore is not directly appropriate for linear mixed models that are designed for multivariate normal data. 

*Moreover, in mixed models there are also different AICs depending on which version of the model you are interested in, namely, the marginal model in which your integrate the random effects out or the hierarchical model that is defined conditional on the random effects. For the latter you can have a look at the cAIC4 package.

A: Dimitris comments are useful, but I think the fundamental issue here is the assumption that you have to simplify your model. 
If your initial hypothesis was x ~ y * z, fit this model, check residuals, consider if you might be overfitting (i.e. do you have enough data to fit this model), and if everything is fine report the results and move on. 
Removing predictors that are n.s., or model selection before significance testing is not helpful if you are interested in calibrated p-values and unbiased regression estimates. 
