2
$\begingroup$

I want to find the best hyperparameters of a neural network by a grid search. Let's say I have:

  • activation (ReLU or sigmoid)
  • batch size (32, 64, 128 or 256)

so my space of hyper-parameters has 8 points.

My data is:

[1 2 3 4 5 6]

I have limited data, so I do k-fold cross validation for each hyperparameter choice. For example I pick ReLU/32 and do 3 trainings:

train = [1 2 3 4], validation = [5 6] => accuracy = 0.9
train = [1 2 5 6], validation = [3 4] => accuracy = 0.7
train = [3 4 5 6], validation = [1 2] => accuracy = 0.8

Now I calculate the average accuracy (0.9 + 0.7 + 0.8)/3 = 0.8 and move to another point of space of hyperparameters (e.g.: ReLU/64), nothing special. And now arises my question: should I reshuffle data before next k-fold split?

For example:

data = [1 3 2 4 5 6] 

would result in slightly different train/validation splits:

train = [1 3 2 4], validation = [5 6] (the same as before)
train = [1 3 5 6], validation = [2 4] (different)
train = [2 4 5 6], validation = [1 3] (different)

Should I use the same split for all points or reshuffle?

$\endgroup$
2
  • 3
    $\begingroup$ It's a little known or recognized fact that with k-fold CV, valid model comparison is meaningful only if the exact same data is used for each k-fold between each model. In other words, given two models and 5-folds, the same information should be used for each fold for each model. Juggling the information in the draws between models will bias the results. $\endgroup$ – Mike Hunter Nov 17 '17 at 20:32
  • $\begingroup$ A little late, but thank you @DJohnson for your comment - it could be the answer. $\endgroup$ – hans Dec 5 '17 at 22:01
1
$\begingroup$

It's a little known or recognized fact that with k-fold CV, valid model comparison is meaningful only if the exact same data is used for each k-fold between each model. In other words, given two models and 5-folds, the same information should be used for each fold for each model. Juggling the information in the draws between models will bias the results.


I've copied @DJohnson's comment as an answer because the comment is, more or less, an answer to this question. We have a dramatic gap between answers and questions. At least part of the problem is that some questions are answered in comments: if comments which answered the question were answers instead, we would have fewer unanswered questions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.