I have a fairly small dataset (~500 records) which use to evaluate the predictive accuracy of classification models produced by various classifiers (knn, decision trees, svm etc) via 10-fold cross validation.
Out of curiosity I conducted several consecutive evaluations with the exact same parameters (but with subsets randomly chosen each time) only to see that the results vary significantly.
For example, I run 10-fold cross validation 100 times on my dataset and KNN reported accuracy in the range of 84%-89% with with some results being 78%, 96%.
If I repeat the same experiments, again 100 times with the same parameters but this time choose to use the same subsets each time (not randomly chosen splits), I get very coherent results: identical accuracy for some classifiers and accuracy that varies by +/- 0.4% for others.
First of all does this variation on the results sounds reasonable? What could be the causes of that? Secondly, what is the best practice followed in such cases?
- Take the results of a single cross-validation (e.g., the first) and report.
- Change the number of folds until the results are coherent.
- Repeat the evaluation multiple times and report the best results.
- Repeat the evaluation multiple times and report the worst results.
- Repeat the evaluation multiple times and report the avg.
It seems obvious to me that the last approach provides more objective results yet, to the best of my knowledge the first approach is by far the most popular in academic papers. Why is that?