I'm performing hyperparameter tuning for a classifier. After I finish, I'm updating the hyperparameter search space and re-tuning the hyperparameters again. I repeat this process a few times. In addition to the validation set used for hyperparameter optimization, I use a separate test set for final evaluation. Is it a good approach, or can it lead to overfitting?

I'm asking for a general case and not my private case, but my hyperparameter tuning method is Optuna, and the classifier is Catboost. In addition, in each Optuna iteration, when fitting the CatBoost, I use an early stop on the same validation set.

The relevant code is attached below.

 def objective(self,trial):
        params, p = get_catboost_params(trial, self.categorical_features)
        clf = CatBoostClassifier(verbose=200, random_seed=42, eval_metric='AUC', auto_class_weights='Balanced', **params)
        clf.fit(self.X_train, self.y_train, eval_set=(self.X_val, self.y_val), early_stopping_rounds=p, cat_features=self.categorical_features)
        y_val_prob = clf.predict_proba(self.X_val)[:, 1]
        auc_score= roc_auc_score(self.y_val,  y_val_prob)
        return auc_score

 def optimize_hyperparameters (self):

        study = optuna.create_study(direction="maximize")
        study.optimize(self.objective, n_trials=self.n_trials)
        self.best_params=  study.best_params
        trials_params = self.get_trial_params(study)

1 Answer 1


Sure there is always a risk of overfitting hyperparameters to the data. But for gbdt you can do this trick. In case your bagging and feature fractions are less than 1, change the random seed several times to see whether the validation error fluctuate wildly. If the error is stable, the chosen hyperparamters should be good. The same applies to the best number of iters. Also the error curve should have a blunt pit so that the test error will not be too sensitive to the choice of #iters.

Another issue is that you should keep your modeling process on validation and test sets exactly the same. For your pipeline, it's better to use cross-validation to select the best #iters instead of using the validation set, i.e. you optuna trial should like this:

  1. Select a set of hyperpars.
  2. Train model(**hyperpars) on training set with cv and early stopping. Record best #iters.
  3. Train another model(**hyperpars) on all training data with best #iters.
  4. Evaluate model from step 3 on validation data. Return accuracy.

After optimization, you just repeat the process on training + validation set, i.e.

  1. Train model(**best_hyperpars) on training+validation set with cv and early stopping. Record best #iters.
  2. Train your final model(**best_hyperpars) on training+validation set with best #iters.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.