I have a fairly small dataset (~500 records) which use to evaluate the predictive accuracy of classification models produced by various classifiers (knn, decision trees, svm etc) via 10-fold cross validation.

Out of curiosity I conducted several consecutive evaluations with the exact same parameters (but with subsets randomly chosen each time) only to see that the results vary significantly.

For example, I run 10-fold cross validation 100 times on my dataset and KNN reported accuracy in the range of 84%-89% with with some results being 78%, 96%.

If I repeat the same experiments, again 100 times with the same parameters but this time choose to use the same subsets each time (not randomly chosen splits), I get very coherent results: identical accuracy for some classifiers and accuracy that varies by +/- 0.4% for others.

First of all does this variation on the results sounds reasonable? What could be the causes of that? Secondly, what is the best practice followed in such cases?

  • Take the results of a single cross-validation (e.g., the first) and report.
  • Change the number of folds until the results are coherent.
  • Repeat the evaluation multiple times and report the best results.
  • Repeat the evaluation multiple times and report the worst results.
  • Repeat the evaluation multiple times and report the avg.

It seems obvious to me that the last approach provides more objective results yet, to the best of my knowledge the first approach is by far the most popular in academic papers. Why is that?

  • $\begingroup$ The question is off topic for this site. If you can edit it to make the question mainly about statistics or machine learning it may not be put on hold. $\endgroup$ Jan 25, 2017 at 20:50
  • 2
    $\begingroup$ Cross validation is on topic @MichaelChernick. $\endgroup$ Jan 25, 2017 at 22:21
  • $\begingroup$ @drdoom: It might help to spell out what the two procedures are in a little more detail, rather than relying on readers to share your interpretation of "run" & "experiment". $\endgroup$ Jan 25, 2017 at 22:21
  • $\begingroup$ @Scortchi edited to make the question hopefully more readable. I tried to keep it generic and theoretical but if asked I will edit again to provide code. $\endgroup$
    – dr.doom
    Jan 26, 2017 at 17:20

1 Answer 1

  1. Sounds reasonable because of the number of samples is small. You can also use randomization tests to access classifier performance as in this [article].

  2. Your 5th option (Repeat the evaluation multiple times and report the avg.) seems most reasonable. Please report both average and the variance (or standard deviation).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.