I'm running StratifiedKFold(n_splits=5, shuffle = True) on a binary classification problem.

I take the accuracy and recall from each testing fold and report their average.

I noticed that the averaged recall score can range from 0.60 to 0.85. I know this is due to the shuffle that occurs before the data is split into 5 folds for cross validation. (I guess the shuffle can be favourable or unfavourable)

If I want to publish my results and be ethical, which cross validation results do I select to report? The ones with good average recall scores? the ones with bad average recall scores? One in the middle? Take an average of many cross-validation results (how many cross validation results should I average?)

  • $\begingroup$ I think a better way to perform CV is to use repeated CV. Frank talks a little bit about it here $\endgroup$ – Demetri Pananos Sep 4 '20 at 17:59
  • $\begingroup$ @DemetriPananos Thank you, I found RepeatedStratifiedKFold in scikitlearn. However I dont see a shuffle parameter for RepeatedStratifiedKFold like I see one for StratifiedKFold. Is there a reason behind this? is the shuffle parameter not needed for RepeatedStratifiedKFold ? $\endgroup$ – link Sep 4 '20 at 18:19
  • 1
    $\begingroup$ Probably because scikitlearn is primarily a data science and machine learning library. Practitioners in those areas are typically not well versed in the "proper" way to do statistics. $\endgroup$ – Robert Long Sep 4 '20 at 18:32
  • 1
    $\begingroup$ "is the shuffle parameter not needed for RepeatedStratifiedKFold?" No: not shuffling is not sensible for repeated CV, so there should not be an option to turn it off. $\endgroup$ – cbeleites unhappy with SX Sep 5 '20 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.