Is there a risk of overfitting when hyperparameter tuning a model using Optuna (or another hyperparameter tuning method ), with evaluation on a validation set and a large number of trials?
While a smaller number of trials may not find the best combination of parameters, could increasing the number of trials lead to the model being overfitted to the validation set? In both cases, the final model is evaluated on a test set.
Yes, choosing hyperparameters with a validation set (or similarly through cross-validation) can lead to overfitting to the validation set. This get worse, the smaller the validation set is, the more hyper-parameters to tune there are and the more different hyper-parameter you try (although there is a limit closely related to the previous point based on how flexibly the hyper-parameters can ever be made to overfit, if you tried to maximize overfitting). This tends to be less bad with cross-validation (or even repeated cross-validation) vs. with a single training-validation split, but to some extent cannot be totally avoided.
Exactly how you try hyperparameters (some kind of clever search like in optuna, grid-search, random-grid-search) does not really matter too much for this answer.
An example of this for ridge or LASSO regression is the 1SE rule, which suggest to look for the value of a single hyper-parameter that is the minimum cross-validation performance and then to make the penalty stronger until the CV-performance is still within 1 standard error (in order to pick something that will perform better on unseen new data). This is a decent rule of thumb that tries to account for the overfitting to the validation parts of the CV-fold splits. With models with many more hyperparameters, it is a lot harder to find such a simple rule.
An illustration of the issue can also be seen in the "Do ImageNet classifiers generalize to ImageNet" paper, if you look at the ImageNet test set as a validation set on which you try out completely different model architecture (a very high-dimensional hyperparameter space). As one can see in such a case with a very large test set, the test set performance overestimates the performance on a newly created test set, but at least the ordering of the models is roughly right.
