0
$\begingroup$

I have been doing a specific check of k-fold technique to see the difference using different number of folds and the corresponding result on the score obtained. To perform this test I have made several runs with all possible number of folds and several iterations with shuffle activated. The number of iterations have depended on the number of folds to be consistent on the number of possible combinations, as explained in Choice of K in K-fold cross-validation

numIters = int(maxIters / numFolds)+1, where maxIters is a number of the possibilities performed (1000 [=100x10] in the example above).

This procedure has given me several scores with a wide variety of results, so I decided to check the statistical scattering for mean and/or percentiles.

enter image description here

  • X-axis is the number of folds (number of features ~130)
  • Y-axis is the statistical value of all the scores (mean, median, ...)

From this kind of result, can it be concluded that with Leave-One-Out method the result will be the best on a test dataset if the dataset is large enough?

I know that Leave-One-Out is a specific case where the number of folds is equal to the number of features. But I would like to know looking to the results if the Leave-One-Out always gives the lowest mean score, and therefore, it should be used if computationally available as k-fold methodology is used to reduce the amount of computation time for long datasets.

Improve this question
$\endgroup$
3
  • $\begingroup$ What does your "score" represent? error? accuracy? Looks like lower is better. LOO gives the best score because each model is trained with almost all the training data, and it's normal that the score should improve with the amount of data used to train the model. However, the purpose of cross-validation is to estimate how well your final model will perform on a future unknown dataset, not to construct this final model. The final model should be constructed from all the available training data. $\endgroup$
    – user3780968
    Commented Apr 29, 2015 at 19:16
  • $\begingroup$ @user3780968 I agree with you that CV is to check the final model expected result and that the final model should use all the training data. As you said, the lower score, the better. But I would like to know, if you have all the dataset available except one case, the LOO is the best k-fold method to perform a final model? Suppose that you can feed your training dataset with all new data and you have always just one test value. Does LOO is the best choice to parameterize your model looking for the lowest score? $\endgroup$
    – iblasi
    Commented Apr 30, 2015 at 6:23
  • $\begingroup$ So your question is, is LOO the best way to fine-tune your model parameters? Yeah, to me intuitively I think it is... However fine-tuning parameters is still part of training, so if after that you want to know how good your model is you will need another test set that you didn't use at all during training (often called "hold out set"). Otherwise you'd be testing on the training set. $\endgroup$
    – user3780968
    Commented May 4, 2015 at 19:49

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.