I'm struggling with a couple of concepts related to hyperparameter tuning.

I'm developing a model (gradient boosted tree) in python using sklearn. Currently, I'm in the process of using sklearn's RandomizedSearchCV to find the optimal hyperparameters. In order to be thorough, I'm testing many different scoring functions (recall, accuracy, f1, roc_auc, etc.) to evaluate the performance of the parameters.

I'm running the random search with a designated validation data set that's separate from my training and test sets.

Now I want to take the optimal parameters from each scoring function and generate predictions for each. This way I can select parameters that make the model behave the way I desire.

My question is: What data should I use to generate the predictions from each set of hyperparameters?

I can't use the validation data, because that was used to recommend the hyperparameters. Do I need to create a specific holdout data set from within my validation dataset?

How do I evaluate the performance of the various scoring functions without data leakage?

I feel like I'm overcomplicating things and that there is a simple and intuitive approach that I'm missing.

Any advice/guidance/resources for this problem would be greatly appreciated.


1 Answer 1


This isn't the best way to go about validating models. If you're doing hyperparameter selection, you need to validate the process of selecting the hyperparameters vis a vis a nested cross validation as I explain here.

Sklearn has some tools to do nested cross validation. Additionally, you can evaluate all your scoring metrics in a single nested cross validation by passing a dictionary to the scoring argument.

  • $\begingroup$ Thanks! One thing that I couldn't understand from the sklearn documentation: does passing multiple metrics evaluate all of them individually, or does it take them all into consideration at once? $\endgroup$ Apr 29, 2022 at 17:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.