I'm running StratifiedKFold(n_splits=5, shuffle = True) on a binary classification problem.
I take the accuracy and recall from each testing fold and report their average.
I noticed that the averaged recall score can range from 0.60 to 0.85. I know this is due to the shuffle that occurs before the data is split into 5 folds for cross validation. (I guess the shuffle can be favourable or unfavourable)
If I want to publish my results and be ethical, which cross validation results do I select to report? The ones with good average recall scores? the ones with bad average recall scores? One in the middle? Take an average of many cross-validation results (how many cross validation results should I average?)