4
$\begingroup$

Is there a risk of overfitting when hyperparameter tuning a model using Optuna (or another hyperparameter tuning method ), with evaluation on a validation set and a large number of trials?

While a smaller number of trials may not find the best combination of parameters, could increasing the number of trials lead to the model being overfitted to the validation set? In both cases, the final model is evaluated on a test set.

$\endgroup$
3
  • $\begingroup$ There is always a risk of overfitting when choosing a model and tuning hyperparameters, which is why you have the final test. $\endgroup$
    – Henry
    Apr 4, 2023 at 8:53
  • $\begingroup$ How can I ensure that choosing a smaller number of trials during the tuning process will not yield better results on the test set $\endgroup$
    – Amit S
    Apr 4, 2023 at 9:09
  • $\begingroup$ That is a convoluted way of describing things. A smaller data set used in training is less likely to give a better model (it has less information), and you should design your cross-validation steps to reduce overfitting. But nothing is ever guaranteed. $\endgroup$
    – Henry
    Apr 4, 2023 at 10:18

1 Answer 1

8
+100
$\begingroup$

Yes, choosing hyperparameters with a validation set (or similarly through cross-validation) can lead to overfitting to the validation set. This get worse, the smaller the validation set is, the more hyper-parameters to tune there are and the more different hyper-parameter you try (although there is a limit closely related to the previous point based on how flexibly the hyper-parameters can ever be made to overfit, if you tried to maximize overfitting). This tends to be less bad with cross-validation (or even repeated cross-validation) vs. with a single training-validation split, but to some extent cannot be totally avoided.

Exactly how you try hyperparameters (some kind of clever search like in optuna, grid-search, random-grid-search) does not really matter too much for this answer.

An example of this for ridge or LASSO regression is the 1SE rule, which suggest to look for the value of a single hyper-parameter that is the minimum cross-validation performance and then to make the penalty stronger until the CV-performance is still within 1 standard error (in order to pick something that will perform better on unseen new data). This is a decent rule of thumb that tries to account for the overfitting to the validation parts of the CV-fold splits. With models with many more hyperparameters, it is a lot harder to find such a simple rule.

An illustration of the issue can also be seen in the "Do ImageNet classifiers generalize to ImageNet" paper, if you look at the ImageNet test set as a validation set on which you try out completely different model architecture (a very high-dimensional hyperparameter space). As one can see in such a case with a very large test set, the test set performance overestimates the performance on a newly created test set, but at least the ordering of the models is roughly right.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.