Reshuffle before k-fold cross-validation split when doing grid search

Question

I want to find the best hyperparameters of a neural network by a grid search. Let's say I have:

• activation (ReLU or sigmoid)
• batch size (32, 64, 128 or 256)

so my space of hyper-parameters has 8 points.

My data is:

[1 2 3 4 5 6]

I have limited data, so I do k-fold cross validation for each hyperparameter choice. For example I pick ReLU/32 and do 3 trainings:

train = [1 2 3 4], validation = [5 6] => accuracy = 0.9
train = [1 2 5 6], validation = [3 4] => accuracy = 0.7
train = [3 4 5 6], validation = [1 2] => accuracy = 0.8

Now I calculate the average accuracy (0.9 + 0.7 + 0.8)/3 = 0.8 and move to another point of space of hyperparameters (e.g.: ReLU/64), nothing special. And now arises my question: should I reshuffle data before next k-fold split?

For example:

data = [1 3 2 4 5 6] 

would result in slightly different train/validation splits:

train = [1 3 2 4], validation = [5 6] (the same as before)
train = [1 3 5 6], validation = [2 4] (different)
train = [2 4 5 6], validation = [1 3] (different)

Should I use the same split for all points or reshuffle?

Answer 1

It's a little known or recognized fact that with k-fold CV, valid model comparison is meaningful only if the exact same data is used for each k-fold between each model. In other words, given two models and 5-folds, the same information should be used for each fold for each model. Juggling the information in the draws between models will bias the results.

---

I've copied @DJohnson's comment as an answer because the comment is, more or less, an answer to this question. We have a dramatic gap between answers and questions. At least part of the problem is that some questions are answered in comments: if comments which answered the question were answers instead, we would have fewer unanswered questions.

• Are we seeing a dramatic drop in answers per question?

• Comments that are actually answers

• We have a very large & widening gap between questions and answers. How do we fix it?