9
$\begingroup$

I'm trying to use k-fold cross validation for model selection for a mixed-effect model (fitted with the lme function).

But, what exactly do I use as the score for each fold? Presumably I don't just fit each candidate model to the validation subset, calculating new coefficients based on the new data. If I understand correctly, I'm supposed to score the models according to how well a model with coefficients calculated using the training data fits the validation data.

But how does one calculate AIC, BIC, logLik, adjR^2, etc on an artificial model that gets its coefficients from one source and its data from another? With so many people advocating cross-validation, I thought there would be more information and code available for calculating the scores by which models will be compared. I can't be the first one trying to cross-validate lme fits in R, yet I see absolutely nothing about what to use as the score... how does everyone else do this? What am I overlooking?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ Can you not just calculate the R^2 on each test fold? i.e. how well the model fits the test cases for each fold. $\endgroup$
    – BGreene
    Commented Oct 27, 2012 at 10:37
  • $\begingroup$ I've been thinking about that... even though R^2 always goes down with each additional term, the reason that's a problem is overfitting. If it is also down for an out-of-sample prediction, that's evidence against overfitting. Is that the reasoning behind it being okay to use R^2 in this case? $\endgroup$
    – f1r3br4nd
    Commented Oct 29, 2012 at 7:23
  • 1
    $\begingroup$ Basically R^2 is a measure of how well your model fits the data. This is only useful for prediction when calculated on unseen (test fold) data. If you have a high R^2 on your training set and low R^2 on you test set this is evidence of overfitting. The mean R^2 averaged across all test folds in cross validation could be a good measure of the generalized model fit. $\endgroup$
    – BGreene
    Commented Oct 29, 2012 at 21:48

2 Answers 2

Reset to default
3
$\begingroup$

I've mostly seen cross-validation used in a machine-learning context where one thinks in terms of a loss function that one is trying to minimize. The natural loss function associated with linear models is mean squared error (which is basically the same as $R^2$). Calculating this for test data is very simple.

You could also use other loss functions (mean absolute error, rank correlation, etc.). However, since the linear model learns by minimizing $R^2$, it might be advisable to try a different model in this case that maximizes whatever loss function you chose (e.g. quantile regression for the mean absolute error).

Improve this answer
$\endgroup$
3
$\begingroup$

The goal with cross validation is to estimate how well your model will perform on new data. So you are correct in that you'll fit the model on a subset of your data ($k-1$ folds). Then you'll use the test set (fold $k$) to make predictions using the model you just built.

You'll now have the true values and predicted values for fold k (your test set), which is generally all you need to calculate different performance measures. Repeat $k$ times and average to get the average performance of your model. Chapter 5 of An Introduction to Statistical Learning provides a good overview of k-fold cross validation.

Edit: If the concern is that you need people from each group/cohort in both your train and test sets, then you could do stratified sampling of each of your groups, such that you end up with members of each cohort in both your test and train sets.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.