8
$\begingroup$

I've been doing some Machine Learning, and have been using k-fold cross-validation to assess the generalisation performance of the algorithm. I've tried k-fold cross-validation with k = 5 and k = 200 and get very different results for Support Vector Machine classification.

k    SVM accuracy
-----------------
5    75%
200  94%

This seems like a huge difference in accuracy caused by changing the number of splits we're doing for the k-fold cross-validation. Is there any reason for this? I can't seem to find any references on studies that have been done investigating the effects of using different k values. Obviously, which k value I decide to use in my report gives completely different impressions of the quality of my classifier!

Improve this question
$\endgroup$
1
  • $\begingroup$ This answer discusses the effect of K value, including simulations. $\endgroup$
    – Mark Loyman
    Commented Dec 14, 2020 at 8:01

1 Answer 1

Reset to default
7
$\begingroup$

Not much of a "proof" but when k is small, you are removing a much larger chunk of your data, so you model has a much smaller amount of data to "learn from". For k=5 you are removing 20% of the data each time, whereas for k=200 you are only removing 0.5%. You model has a much better chance of picking up all the relevant "structure" in the training part when k is large. When k is small, the is a larger chance that the "left out" part will contain a structure which is absent from the "left in" bit - a bit like an "un-representative" sub-sample.

Improve this answer
$\endgroup$
3
  • $\begingroup$ On the other hand, the additional information that is being picked up could be just a fluke of the data, and your accuracy estimator will be overly optimistic because of overfitting. $\endgroup$
    – Aniko
    Commented Apr 26, 2011 at 19:26
  • $\begingroup$ @aniko - while true cross validation doesn't fix this problem - it relies on the data being a "realistic" version of the data one is likely to predict from. No method which uses only the data can account for this (because by definition, it is based only on the data). $\endgroup$
    – probabilityislogic
    Commented Apr 27, 2011 at 7:09
  • 5
    $\begingroup$ The number of times you repeat the whole cross-validation process is also important. My machine learning colleagues feel that 10-fold cross-validation should be repeated 100 times to achieve good precision. A single 10-fold cross-validation yields an accuracy measure that has a high variance. The choice of accuracy measures is also important. You are using an improper scoring rule that is also discontinuous, so expect more problems. $\endgroup$
    – Frank Harrell
    Commented Apr 27, 2011 at 11:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.