I want to optimize the hyperparams for a CNN-architecture by using GridSearchCV. As hyperparameters to optimize, I would like to use the learning rate, dropout rate, number of neurons in den dense layer and the number of epochs. For the training of the final Model with the optimized hyperparams, I would like to use EarlyStopping to counteract overfitting. And here is my problem, I am not sure if it would still be advisable to use a search range for number of epochs in GridSearchCV at all, or whether I should simply use a fixed number of epochs to determine the remaining hyperparams with GridSearch?

What are your opinions and recommendations?

Thanks in advance

  • $\begingroup$ GridSearchCV from Scit-kit learn? $\endgroup$ – develarist Nov 29 '19 at 15:53
  • $\begingroup$ @develarist yes $\endgroup$ – Code Now Nov 29 '19 at 16:17
  • $\begingroup$ @develarist yes I'm using GridSearchCV from Scikit-learn. Which approach do you think is right? $\endgroup$ – Code Now Nov 29 '19 at 18:42

If the model you input to grid search has early stopping enabled (it should be by the way), you might end up selecting an irrelevant epoch number as the best one if the epoch numbers you try in your grid are greater than the one required by early stopping. Thus, you'll loop over number of epochs in vain. You can log a few trials and see if your models stops early in general. If they do, it might be more efficient to set a large number of epoch, and let your models stop early for some reason. This way, you can increase the grid search over other hyper-parameters.

  • $\begingroup$ Summarizing again, so I understood correctly: The conclusion would be to use EarlyStopping instead of a search range for number of epochs in GridSearch, right? If yes, this idea already came to my mind. Unfortunately I could not find a code example for such a approach, so I assumed so far that this is not a recommended approach. $\endgroup$ – Code Now Nov 29 '19 at 22:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.