# 10-fold Cross-validation vs leave-one-out cross-validation

I'm doing nested cross-validation. I have read that leave-one-out cross-validation can be biased (don't remember why).

Is it better to use 10-fold cross-validation or leave-one-out cross-validation apart from the longer runtime for leave-one-out cross-validation?

• Do you remember where you read that? May 31, 2015 at 12:41
• Have you seen this post about bias? Also, this answer has a quote from a very good book that recommends 5-fold or 10-fold cross validation. May 31, 2015 at 12:46
• This post is a little related. May 31, 2015 at 12:49
• Thank you. So all in all it can be said that I should go with 10-fold CV instead of leave-one-out CV? Does this also hold for a small dataset? May 31, 2015 at 21:06
• @Thomas, when your dataset gets too small you end up nearly doing LOO-CV so the benefit of 10-fold CV diminishes as your dataset size decreases. Jun 4, 2015 at 19:03

Cross-validation gives a pessimistically biased estimate of performance because most statistical models will improve if the training set is made larger. This means that k-fold cross-validation estimates the performance of a model trained on a dataset $$100\times\frac{(k-1)}{k}\%$$ of the available data, rather than on 100% of it. So if you perform cross-validation to estimate performance, and then use a model trained on all of the data for operational use, it will perform slightly better than the cross-validation estimate suggests.

Leave-one-out cross-validation is approximately unbiased, because the difference in size between the training set used in each fold and the entire dataset is only a single pattern. There is a paper on this by Luntz and Brailovsky (in Russian).

Luntz, Aleksandr, and Viktor Brailovsky. "On estimation of characters obtained in statistical procedure of recognition." Technicheskaya Kibernetica 3.6 (1969): 6–12.

Lachenbruch,Peter A., and Mickey, M. Ray. "Estimation of Error Rates in Discriminant Analysis." Technometrics 10.1 (1968): 1–11.

However, while leave-one-out cross-validation is approximately unbiased, it tends to have a high variance (so you would get very different estimates if you repeated the estimate with different initial samples of data from the same distribution). As the error of the estimator is a combination of bias and variance, whether leave-one-out cross-validation is better than 10-fold cross-validation depends on both quantities.

Now the variance in fitting the model tends to be higher if it is fitted to a small dataset (as it is more sensitive to any noise/sampling artifacts in the particular training sample used). This means that 10-fold cross-validation is likely to have a high variance (as well as a higher bias) if you only have a limited amount of data, as the size of the training set will be smaller than for LOOCV. So k-fold cross-validation can have variance issues as well, but for a different reason. This is why LOOCV is often better when the size of the dataset is small.

However, the main reason for using LOOCV in my opinion is that it is computationally inexpensive for some models (such as linear regression, most kernel methods, nearest-neighbour classifiers, etc.), and unless the dataset were very small, I would use 10-fold cross-validation if it fitted in my computational budget, or better still, bootstrap estimation and bagging.

• +1 for the obscure 1969 Russian reference! Do you have a good reference for LOOCV having high variance? This is stated in Hastie et al but I am not sure I am 100% convinced by the argument and I haven't seen empirical demonstrations (simulations). Mar 1, 2017 at 8:43
• yes, I don't think I agree with it though, as it assumes that the model is stable under the perturbations caused by deleting the test samples, which is only likely to approach being true if you have a very large dataset (i.e. it is only asymptotically true, but if you had that much data, almost any sensible performance evaluation scheme would give you the same result). Mar 1, 2017 at 10:46
• +1 (both the post as well as the latest comment - great paper but not to be blinded followed (as any other paper)). May 20, 2017 at 11:11
• @Dikran This topic (of LOOCV having the largest variance) came up again in a separate and quite interesting question: stats.stackexchange.com/questions/280665, you might want to take a look. May 22, 2017 at 6:10
• Here is another simulation stats.stackexchange.com/a/357749 showing that the variance of CV estimator decreases with the number of folds and LOOCV has the same (or lower) variance as 10-fold. Another simulation linked in my comment above showed another example where variance was decreasing with $k$, and was the lowest for LOOCV. By now I am really curious to see any simulation where the variance would increase with the number of folds. I am also starting to be rather skeptical that it can happen in practice. Jul 19, 2018 at 20:35

In my opinion, leave one out cross validation is better when you have a small set of training data. In this case, you can't really make 10 folds to make predictions on using the rest of your data to train the model.

If you have a large amount of training data on the other hand, 10-fold cross validation would be a better bet, because there will be too many iterations for leave one out cross-validation, and considering these many results to tune your hyperparameters might not be such a good idea.

According to ISL, there is always a bias-variance trade-off between doing leave one out and k fold cross validation. In LOOCV(leave one out CV), you get estimates of test error with lower bias, and higher variance because each training set contains n-1 examples, which means that you are using almost the entire training set in each iteration. This leads to higher variance too, because there is a lot of overlap between training sets, and thus the test error estimates are highly correlated, which means that the mean value of the test error estimate will have higher variance.

The opposite is true with k-fold CV, because there is relatively less overlap between training sets, thus the test error estimates are less correlated, as a result of which the mean test error value won't have as much variance as LOOCV.

• "...you get estimates of test error with lower bias, and higher variance because each training set contains n-1 examples, which means that you are using almost the entire training set in each iteration. This leads to higher variance too, because there is a lot of overlap between training sets, and thus the test error estimates are highly correlated..." Are these two separate reasons or are you just saying the same thing twice? Nov 10, 2021 at 22:09

The existing answers focus on getting good estimates of the out of sample prediction error. This is not the only perspective on the LOOCV versus K-fold CV decision. In particular, some readers may believe there is a true model and may wish to recover it.

In this case, Shao 1993 famously showed that for linear models, LOOCV is inconsistent for recovering the true model and something resembling K-fold CV is consistent. (Shao considered a train-test split where the $$n_{test}/n_{train}$$ goes to 1, i.e. the test set dominates.)

Model selection is a deep and confusing topic, so let me add a couple of references to related discussions.

• A related discussion here on LASSO reinforces this dichotomy of prediction vs inference and has interesting, thought-provoking comments about LASSO and CV in specific.
• This site has maybe hundreds of related threads that describe AIC and BIC. In some cases, these are asymptotically equivalent to LOOCV (AIC) and K-fold CV (BIC) (More info and orignal sources in this answer: https://stats.stackexchange.com/a/414610/86176). These threads reinforce what I wrote above, because like LOOCV, AIC emphasizes prediction. Like K-fold, BIC emphasizes selecting the true model. Yuhong Yang showed that no procedure can be optimal for both purposes.
• Rob Hyndman has a useful cross-validation overview here mentioning AIC, BIC, LOO, and leave-more-out CV. His position is that consistency in model selection is irrelevant because the true model is rarely in the set under consideration.