# K-Fold Cross Validation for mixed-effect models: how to score them?

I'm trying to use k-fold cross validation for model selection for a mixed-effect model (fitted with the lme function).

But, what exactly do I use as the score for each fold? Presumably I don't just fit each candidate model to the validation subset, calculating new coefficients based on the new data. If I understand correctly, I'm supposed to score the models according to how well a model with coefficients calculated using the training data fits the validation data.

But how does one calculate AIC, BIC, logLik, adjR^2, etc on an artificial model that gets its coefficients from one source and its data from another? With so many people advocating cross-validation, I thought there would be more information and code available for calculating the scores by which models will be compared. I can't be the first one trying to cross-validate lme fits in R, yet I see absolutely nothing about what to use as the score... how does everyone else do this? What am I overlooking?

-
Can you not just calculate the R^2 on each test fold? i.e. how well the model fits the test cases for each fold. –  BGreene Oct 27 '12 at 10:37
I've been thinking about that... even though R^2 always goes down with each additional term, the reason that's a problem is overfitting. If it is also down for an out-of-sample prediction, that's evidence against overfitting. Is that the reasoning behind it being okay to use R^2 in this case? –  f1r3br4nd Oct 29 '12 at 7:23
Basically R^2 is a measure of how well your model fits the data. This is only useful for prediction when calculated on unseen (test fold) data. If you have a high R^2 on your training set and low R^2 on you test set this is evidence of overfitting. The mean R^2 averaged across all test folds in cross validation could be a good measure of the generalized model fit. –  BGreene Oct 29 '12 at 21:48