How should mixed effects models be compared and or validated? How are (linear) mixed effects models normally compared against each other? I know likelihood ratio tests can be used, but this doesn't work if one model is not a 'subset' of the other correct?
Is the estimation of the models df always straightforward? Number of fixed effects + number of variance components estimated? Do we ignore the random effects estimates?
What about validation? My first thought is cross validation, but random folds might not work given the structure of the data. Is a methodology of 'leave one subject/cluster out' appropriate? What about leave one observation out?
Mallows Cp can be interpreted as an estimate of the models prediction error. Model selection via AIC attempts to minimize the prediction error (So Cp and AIC should pick the same model if the errors are Gaussian I believe). Does this mean AIC or Cp can be used to pick an 'optimal' linear mixed effects model from a collection of some un-nested models in terms of prediction error? (provided they are fit on the same data) Is BIC still more likely to pick the 'true' model amongst candidates?
I am also under the impression that when comparing mixed effects models via AIC or BIC we only count the fixed effects as 'parameters' in the calculation, not the actual models df.  
Is there any good literature on these topics? Is it worth investigating cAIC or mAIC? Do they have specific application outside of AIC?
 A: One way to compare models (whether mixed or otherwise) is to plot results. Suppose you havae model A and model B; produce the fitted values from each and graph them against each other in a scatter plot. If the values are very similar (using your judgement as to whether they are) choose the simpler model. Another idea is to find the differences between the fitted values and graph these against the independent values; you  can also make a density plot of the differences. 
In general, I am a proponent of not using statistical tests to compare models (although AIC and its variants certainly have virtues) but rather using judgement. Of course, this has the (dis)advantage of not giving precise answers.  
A: The main problem on model selection in mixed models is to define the degrees of freedom (df) of a model, truly. To compute df of a mixed model, one has to define the number of estimated parameters including fixed and random effects. And this is not straightforward. This paper by Jiming Jiang and others (2008) entitled "Fence methods for mixed model selection" could be applied in such situations. A new related work is this one by Greven, S. & Kneib, T. (2010) entitled "On the behavior of marginal and conditional AIC in linear mixed models". Hope this could be helpful.
