# Reshuffle before k-fold cross-validation split when doing grid search

I want to find the best hyperparameters of a neural network by a grid search. Let's say I have:

• activation (ReLU or sigmoid)
• batch size (32, 64, 128 or 256)

so my space of hyper-parameters has 8 points.

My data is:

[1 2 3 4 5 6]


I have limited data, so I do k-fold cross validation for each hyperparameter choice. For example I pick ReLU/32 and do 3 trainings:

train = [1 2 3 4], validation = [5 6] => accuracy = 0.9
train = [1 2 5 6], validation = [3 4] => accuracy = 0.7
train = [3 4 5 6], validation = [1 2] => accuracy = 0.8


Now I calculate the average accuracy (0.9 + 0.7 + 0.8)/3 = 0.8 and move to another point of space of hyperparameters (e.g.: ReLU/64), nothing special. And now arises my question: should I reshuffle data before next k-fold split?

For example:

data = [1 3 2 4 5 6]


would result in slightly different train/validation splits:

train = [1 3 2 4], validation = [5 6] (the same as before)
train = [1 3 5 6], validation = [2 4] (different)
train = [2 4 5 6], validation = [1 3] (different)


Should I use the same split for all points or reshuffle?

• It's a little known or recognized fact that with k-fold CV, valid model comparison is meaningful only if the exact same data is used for each k-fold between each model. In other words, given two models and 5-folds, the same information should be used for each fold for each model. Juggling the information in the draws between models will bias the results. – Mike Hunter Nov 17 '17 at 20:32
• A little late, but thank you @DJohnson for your comment - it could be the answer. – hans Dec 5 '17 at 22:01

## 1 Answer

It's a little known or recognized fact that with k-fold CV, valid model comparison is meaningful only if the exact same data is used for each k-fold between each model. In other words, given two models and 5-folds, the same information should be used for each fold for each model. Juggling the information in the draws between models will bias the results.

I've copied @DJohnson's comment as an answer because the comment is, more or less, an answer to this question. We have a dramatic gap between answers and questions. At least part of the problem is that some questions are answered in comments: if comments which answered the question were answers instead, we would have fewer unanswered questions.