How are (linear) mixed effects models normally compared against each other? I know likelihood ratio tests can be used, but this doesn't work if one model is not a 'subset' of the other correct?
Is the estimation of the models df always straightforward? Number of fixed effects + number of variance components estimated? Do we ignore the random effects estimates?
What about validation? My first thought is cross validation, but random folds might not work given the structure of the data. Is a methodology of 'leave one subject/cluster out' appropriate? What about leave one observation out?
Mallows Cp can be interpreted as an estimate of the models prediction error. Model selection via AIC attempts to minimize the prediction error (So Cp and AIC should pick the same model if the errors are Gaussian I believe). Does this mean AIC or Cp can be used to pick an 'optimal' linear mixed effects model from a collection of some un-nested models in terms of prediction error? (provided they are fit on the same data) Is BIC still more likely to pick the 'true' model amongst candidates?
I am also under the impression that when comparing mixed effects models via AIC or BIC we only count the fixed effects as 'parameters' in the calculation, not the actual models df.
Is there any good literature on these topics? Is it worth investigating cAIC or mAIC? Do they have specific application outside of AIC?