Model-selection for linear mixed models over alternative sets of parameters (nlme function in R)

My models look like:

lme1 = lme(y~X+Y+V, random=~1|Subject, data=mydata, method ="ML")
lme2 = lme(y~X+Y+V2+V3, random=~1|Subject, data=mydata, method ="ML")
lme3 = lme(y~X+Y+V4, random=~1|Subject, data=mydata, method ="ML")


where X and Y are factors, but V, V2, V3,and V4 are continuous variables (modeled as covariates). I am using Method ="ML" in the hope that I could compare the likelihood values across the models.

My research question has to do with whether V4 (in lme3) was a better predictor than V2 and V3 together, V2+V3 was better than V, etc. What goodness of fits measure is valid here? Can I use AIC values to compare models of different sets of parameters?

I've also found some references on computing $R^2$ for mixed models. In particular, I am interested in the likelihood ratio test $R^2$ (Magee, 1990) which computes a $R^2$ by comparing each of these models to the null model. Using this method, I'd be comparing all three of my models to the same null model with just y~1. Is it then a valid approach to compare the $R^2$s generated?

I am not a statistician but I would like to use a valid (at least justifiable) measure for my analysis. Any feedback would be greatly appreciated.

• Since the models are fit by maximum likelihood, AIC would be defensible here. – Macro Jun 19 '12 at 21:28
• @Macro: thanks for your comment. Could you elaborate on how AIC would be defensible here, when it comes down to comparing models Lme2 and LME3, even though they are non-nested and the number of variables are different? There seemed to be some disagreements over whether AICs can be used in non-nested models. Should I be concerned about that? Thanks again for your help! – Wynn Jun 20 '12 at 18:54
• The AIC is an estimate of the (relative) Kullback-Leibler information. There is no reason to believe that it is applicable only for nested models (I actually don't know why people think that). – Néstor Jun 20 '12 at 19:09
• @Néstor Very true. You can use likelihood-ratio tests for nested models. AIC and the like are especially useful for non-nested models (i.e., situations where you cannot use likelihood-ratio tests). – Henrik Jun 20 '12 at 20:43
• I agree with everyone about AIC, but I just wanted to point out that in the code you have above, y is being modelled as continuous, even though you say it's a factor. You might want to consider using lmer in the lme4 package or the MCMCglmm package depending on what kind of factor y is. – smillig Aug 28 '12 at 6:59

I would use Akaike’s Information Criterion ($AIC$) for model selection where: $$AIC = -2ln(L)-2k$$ Though a better alternative is often $AIC_c$, which is the second-order Akaike’s Information Criterion ($AIC_c$). $AIC_c$ is corrected for small sample size with an addtion bias-correction term because $AIC$ can perform poorly when the ratio of sample size to the number parameters in the model is small (Burnham and Anderson 2002). $$AIC_c = -2ln(L)-2k+\frac{2k(k+1)}{(n-k-1)}$$ In fact, I would always use $AIC_c$ since the bias-correction term goes to zero as sample size increases. However, there are some types of models where it is difficult to determine sample size (i.e., hierarchical models of abundance see links to these model types here).
$AIC$ or $AIC_c$ can be recaled to $\mathsf{\Delta}_i=AIC_i-minAIC$ where the best model will have $\mathsf{\Delta}_i=0$. Further, these values can be used to estimate relative strength of evidence ($w_i$) for the alternative models where: $$w_i = \frac{e^{(-0.5\mathsf{\Delta}_i)}}{\sum_{r=1}^Re^{(-0.5\mathsf{\Delta}_i)}}$$ This is often refered to as the "weight of evidence" for model $i$ from the model set. As $\mathsf{\Delta}_i$ increases, $w_i$ decreases suggesting model $i$ is less plausible. Also, the weights of evidence for the models in a model set can be use in model averaging and multi-model inference.