10-fold Cross-validation vs leave-one-out cross-validation

Question

I'm doing nested cross-validation. I have read that leave-one-out cross-validation can be biased (don't remember why).

Is it better to use 10-fold cross-validation or leave-one-out cross-validation apart from the longer runtime for leave-one-out cross-validation?

Answer 1

Just to add slightly to the answer of @SubravetiSuraj (+1)

Cross-validation gives a pessimistically biased estimate of performance because most statistical models will improve if the training set is made larger. This means that k-fold cross-validation estimates the performance of a model trained on a dataset $100\times\frac{(k-1)}{k}\%$ of the available data, rather than on 100% of it. So if you perform cross-validation to estimate performance, and then use a model trained on all of the data for operational use, it will perform slightly better than the cross-validation estimate suggests.

Leave-one-out cross-validation is approximately unbiased, because the difference in size between the training set used in each fold and the entire dataset is only a single pattern. There is a paper on this by Luntz and Brailovsky (in Russian).

Luntz, Aleksandr, and Viktor Brailovsky. "On estimation of characters obtained in statistical procedure of recognition." Technicheskaya Kibernetica 3.6 (1969): 6–12.

see also

Lachenbruch,Peter A., and Mickey, M. Ray. "Estimation of Error Rates in Discriminant Analysis." Technometrics 10.1 (1968): 1–11.

However, while leave-one-out cross-validation is approximately unbiased, it tends to have a high variance (so you would get very different estimates if you repeated the estimate with different initial samples of data from the same distribution). As the error of the estimator is a combination of bias and variance, whether leave-one-out cross-validation is better than 10-fold cross-validation depends on both quantities.

Now the variance in fitting the model tends to be higher if it is fitted to a small dataset (as it is more sensitive to any noise/sampling artifacts in the particular training sample used). This means that 10-fold cross-validation is likely to have a high variance (as well as a higher bias) if you only have a limited amount of data, as the size of the training set will be smaller than for LOOCV. So k-fold cross-validation can have variance issues as well, but for a different reason. This is why LOOCV is often better when the size of the dataset is small.

However, the main reason for using LOOCV in my opinion is that it is computationally inexpensive for some models (such as linear regression, most kernel methods, nearest-neighbour classifiers, etc.), and unless the dataset were very small, I would use 10-fold cross-validation if it fitted in my computational budget, or better still, bootstrap estimation and bagging.

Answer 2

In my opinion, leave one out cross validation is better when you have a small set of training data. In this case, you can't really make 10 folds to make predictions on using the rest of your data to train the model.

If you have a large amount of training data on the other hand, 10-fold cross validation would be a better bet, because there will be too many iterations for leave one out cross-validation, and considering these many results to tune your hyperparameters might not be such a good idea.

According to ISL, there is always a bias-variance trade-off between doing leave one out and k fold cross validation. In LOOCV(leave one out CV), you get estimates of test error with lower bias, and higher variance because each training set contains n-1 examples, which means that you are using almost the entire training set in each iteration. This leads to higher variance too, because there is a lot of overlap between training sets, and thus the test error estimates are highly correlated, which means that the mean value of the test error estimate will have higher variance.

The opposite is true with k-fold CV, because there is relatively less overlap between training sets, thus the test error estimates are less correlated, as a result of which the mean test error value won't have as much variance as LOOCV.

Answer 3

The existing answers focus on getting good estimates of the out of sample prediction error. This is not the only perspective on the LOOCV versus K-fold CV decision. In particular, some readers may believe there is a true model and may wish to recover it.

In this case, Shao 1993 famously showed that for linear models, LOOCV is inconsistent for recovering the true model and something resembling K-fold CV is consistent. (Shao considered a train-test split where the $n_{test}/n_{train}$ goes to 1, i.e. the test set dominates.)

Model selection is a deep and confusing topic, so let me add a couple of references to related discussions.

• A related discussion here on LASSO reinforces this dichotomy of prediction vs inference and has interesting, thought-provoking comments about LASSO and CV in specific.
• This site has maybe hundreds of related threads that describe AIC and BIC. In some cases, these are asymptotically equivalent to LOOCV (AIC) and K-fold CV (BIC) (More info and orignal sources in this answer: https://stats.stackexchange.com/a/414610/86176). These threads reinforce what I wrote above, because like LOOCV, AIC emphasizes prediction. Like K-fold, BIC emphasizes selecting the true model. Yuhong Yang showed that no procedure can be optimal for both purposes.
• Rob Hyndman has a useful cross-validation overview here mentioning AIC, BIC, LOO, and leave-more-out CV. His position is that consistency in model selection is irrelevant because the true model is rarely in the set under consideration.