20
$\begingroup$

I'm doing nested cross-validation. I have read that leave-one-out cross-validation can be biased (don't remember why).

Is it better to use 10-fold cross-validation or leave-one-out cross-validation apart from the longer runtime for leave-one-out cross-validation?

$\endgroup$
  • 1
    $\begingroup$ Do you remember where you read that? $\endgroup$ – Richard Hardy May 31 '15 at 12:41
  • 4
    $\begingroup$ Have you seen this post about bias? Also, this answer has a quote from a very good book that recommends 5-fold or 10-fold cross validation. $\endgroup$ – Eric Farng May 31 '15 at 12:46
  • 1
    $\begingroup$ This post is a little related. $\endgroup$ – Richard Hardy May 31 '15 at 12:49
  • 1
    $\begingroup$ Thank you. So all in all it can be said that I should go with 10-fold CV instead of leave-one-out CV? Does this also hold for a small dataset? $\endgroup$ – machinery May 31 '15 at 21:06
  • 1
    $\begingroup$ @Thomas, when your dataset gets too small you end up nearly doing LOO-CV so the benefit of 10-fold CV diminishes as your dataset size decreases. $\endgroup$ – cdeterman Jun 4 '15 at 19:03
20
$\begingroup$

Just to add slightly to the answer of @SubravetiSuraj (+1)

Cross-validation gives a pessimistically biased estimate of performance because most statistical models will improve if the training set is made larger. This means that k-fold cross-validation estimates the performance of a model trained on a dataset 100*(k-1)/k% of the available data, rather than on 100% of it. So if you perform cross-validation to estimate performance, and then use a model trained on all of the data for operational use, it will perform slightly better than the cross-validation estimate suggests.

Leave-one-out cross-validation is approximately unbiased, because the difference in size between the training set used in each fold and the entire dataset is only a single pattern. There is a paper on this by Luntz and Brailovsky (in Russian).

Luntz, Aleksandr, and Viktor Brailovsky. "On estimation of characters obtained in statistical procedure of recognition." Technicheskaya Kibernetica 3.6 (1969): 6-12.

see also

Estimation of Error Rates in Discriminant Analysis Peter A. Lachenbruch and M. Ray Mickey Technometrics Vol. 10 , Iss. 1,1968

However, while leave-one-out cross-validation is approximately unbiased, it tends to have a high variance (so you would get very different estimates if you repeated the estimate with different initial samples of data from the same distribution). As the error of the estimator is a combination of bias and variance, whether leave-one-out cross-validation is better than 10-fold cross-validation depends on both quantities.

Now the variance in fitting the model tends to be higher if it is fitted to a small dataset (as it is more sensitive to any noise/sampling artifacts in the particular training sample used). This means that 10-fold cross-validation is likely to have a high variance (as well as a higher bias) if you only have a limited amount of data, as the size of the training set will be smaller than for LOOCV. So k-fold cross-validation can have variance issues as well, but for a different reason. This is why LOOCV is often better when the size of the dataset is small.

However, the main reason for using LOOCV in my opinion is that it is computationally inexpensive for some models (such as linear regression, most kernel methods, nearest-neighbour classifiers, etc.), and unless the dataset were very small, I would use 10-fold cross-validation if it fitted in my computational budget, or better still, bootstrap estimation and bagging.

$\endgroup$
  • 2
    $\begingroup$ +1 for the obscure 1969 Russian reference! Do you have a good reference for LOOCV having high variance? This is stated in Hastie et al but I am not sure I am 100% convinced by the argument and I haven't seen empirical demonstrations (simulations). $\endgroup$ – amoeba Mar 1 '17 at 8:43
  • 3
    $\begingroup$ yes, I don't think I agree with it though, as it assumes that the model is stable under the perturbations caused by deleting the test samples, which is only likely to approach being true if you have a very large dataset (i.e. it is only asymptotically true, but if you had that much data, almost any sensible performance evaluation scheme would give you the same result). $\endgroup$ – Dikran Marsupial Mar 1 '17 at 10:46
  • 2
    $\begingroup$ +1 (both the post as well as the latest comment - great paper but not to be blinded followed (as any other paper)). $\endgroup$ – usεr11852 May 20 '17 at 11:11
  • 2
    $\begingroup$ @Dikran This topic (of LOOCV having the largest variance) came up again in a separate and quite interesting question: stats.stackexchange.com/questions/280665, you might want to take a look. $\endgroup$ – amoeba May 22 '17 at 6:10
  • 2
    $\begingroup$ Here is another simulation stats.stackexchange.com/a/357749 showing that the variance of CV estimator decreases with the number of folds and LOOCV has the same (or lower) variance as 10-fold. Another simulation linked in my comment above showed another example where variance was decreasing with $k$, and was the lowest for LOOCV. By now I am really curious to see any simulation where the variance would increase with the number of folds. I am also starting to be rather skeptical that it can happen in practice. $\endgroup$ – amoeba Jul 19 '18 at 20:35
16
$\begingroup$

In my opinion, leave one out cross validation is better when you have a small set of training data. In this case, you can't really make 10 folds to make predictions on using the rest of your data to train the model.

If you have a large amount of training data on the other hand, 10-fold cross validation would be a better bet, because there will be too many iterations for leave one out cross-validation, and considering these many results to tune your hyperparameters might not be such a good idea.

According to ISL, there is always a bias-variance trade-off between doing leave one out and k fold cross validation. In LOOCV(leave one out CV), you get estimates of test error with lower bias, and higher variance because each training set contains n-1 examples, which means that you are using almost the entire training set in each iteration. This leads to higher variance too, because there is a lot of overlap between training sets, and thus the test error estimates are highly correlated, which means that the mean value of the test error estimate will have higher variance.

The opposite is true with k-fold CV, because there is relatively less overlap between training sets, thus the test error estimates are less correlated, as a result of which the mean test error value won't have as much variance as LOOCV.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.