In leave one out cross validation, if I have 10 data points, and I am trying to fit a linear regression model (as an example), then I generate 10 different models, each time using 9 data points and testing the model on the 10th data point.

After that, do I just select the model that gave the lowest error? or do I take an average of the coefficients of the 10 different models for my final model?

  • $\begingroup$ Do you by any chance have the idea that there is actually only one model but you estimate it using 10 different subsamples of the data? Because if you actually have 10 different models (potentially with different variables), how can you average the coefficients? Also, the number of subsamples and the number of models need not coincide. You may have 5 candidate models and 100 subsamples. $\endgroup$ – Richard Hardy Sep 24 '15 at 18:59
  • $\begingroup$ Yes, there is one model y=a + bx; But (a,b) will have different values for each iteration. In the end, do I just take the average? $\endgroup$ – Victor Sep 24 '15 at 19:07
  • 1
    $\begingroup$ What example are you following? In other words, where did you get this idea? Why do you want to use cross validation to obtain point estimates of the regression coefficients? Using cross validation not for model selection may be more common when you want to look at variation in the point estimates; think about jackknife variance estimator. However, when it comes to point estimates, OLS on the full data sample will give you the minimum-variance linear unbiased estimator (if the model is well specified), so why would you choose an estimator that is still unbiased but has a higher variance? $\endgroup$ – Richard Hardy Sep 24 '15 at 19:24

Cross validation gives you an idea of how reliable your model will be in data it wasn't trained on. In practice, i look at the performance characteristic using some appropriate metric on the residuals of the holdout sets. Then once you're satisfied that the model type selected is appropriate, you train the model on all the data to put into production. Selecting just one of your hold-out trials wouldn't give you any information on future performance.

Here's an illustration. Imagine a linear regression using n-fold CV like you're describing. Put one major outlier in the data. The poorest performing trial would be the one with the outlier in the holdout test set. But since outliers are rare, it's actually a better model for production because it's trained on data that it's likely to see in practice.

So we typically don't use CV to pick a single trial, but to test different models, input variables, etc.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.