I have been doing a specific check of k-fold technique to see the difference using different number of folds and the corresponding result on the score obtained. To perform this test I have made several runs with all possible number of folds and several iterations with shuffle activated. The number of iterations have depended on the number of folds to be consistent on the number of possible combinations, as explained in Choice of K in K-fold cross-validation
numIters = int(maxIters / numFolds)+1, where maxIters is a number of the possibilities performed (1000 [=100x10] in the example above).
This procedure has given me several scores with a wide variety of results, so I decided to check the statistical scattering for mean and/or percentiles.
- X-axis is the number of folds (number of features ~130)
- Y-axis is the statistical value of all the scores (mean, median, ...)
From this kind of result, can it be concluded that with Leave-One-Out method the result will be the best on a test dataset if the dataset is large enough?
I know that Leave-One-Out is a specific case where the number of folds is equal to the number of features. But I would like to know looking to the results if the Leave-One-Out always gives the lowest mean score, and therefore, it should be used if computationally available as k-fold methodology is used to reduce the amount of computation time for long datasets.