I'm performing model evaluation of my sklearn Pipeline, looking for:

  • optimal hyperparameters: the model is a random forest so in this case: number of trees and tree max depth (using grid search)

  • train size: size of the training set (using validation_curve)

  • train/test performance: to check for high bias / high variance (using learning_curve)

My question is: In which order should I perform these steps?

Let say I start with train size, which hyperparameters should I use? Or if I start with hyperparamter tuning, should I use the whole dataset (it's huge) even if it takes forever to train?


It would be better that you first randomize your data and do a train/test split before going ahead with hyperparameter tuning. Usually, an 80/20 split is the recommended with a 5-fold cross-validation on the training set and that usually works out for me.

I don't think you have to evaluate the size of the training set that best suits you unless there is some computational constraint. For hyperparameter tuning, for each training phase in CV, using a random search or a grid search to find the best parameters should work.

To plot out the train/test performance with the number of iterations, it would be best to first find the set of parameters that works for you and then see how they perform on your datasets over time and plot them out.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.