Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite

I'm trying to use k-fold cross validation for model selection for a mixed-effect model (fitted with the lme function).

But, what exactly do I use as the score for each fold? Presumably I don't just fit each candidate model to the validation subset, calculating new coefficients based on the new data. If I understand correctly, I'm supposed to score the models according to how well a model with coefficients calculated using the training data fits the validation data.

But how does one calculate AIC, BIC, logLik, adjR^2, etc on an artificial model that gets its coefficients from one source and its data from another? With so many people advocating cross-validation, I thought there would be more information and code available for calculating the scores by which models will be compared. I can't be the first one trying to cross-validate lme fits in R, yet I see absolutely nothing about what to use as the score... how does everyone else do this? What am I overlooking?

share|improve this question
2  
Can you not just calculate the R^2 on each test fold? i.e. how well the model fits the test cases for each fold. – BGreene Oct 27 '12 at 10:37
I've been thinking about that... even though R^2 always goes down with each additional term, the reason that's a problem is overfitting. If it is also down for an out-of-sample prediction, that's evidence against overfitting. Is that the reasoning behind it being okay to use R^2 in this case? – f1r3br4nd Oct 29 '12 at 7:23
Basically R^2 is a measure of how well your model fits the data. This is only useful for prediction when calculated on unseen (test fold) data. If you have a high R^2 on your training set and low R^2 on you test set this is evidence of overfitting. The mean R^2 averaged across all test folds in cross validation could be a good measure of the generalized model fit. – BGreene Oct 29 '12 at 21:48

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.