I see suggestions like this for tuning a GradientBoostingRegressor's params through multiple steps. In each step, the best params are selected via CV. For example (abbreviated steps):

  1. find some of the best tree-based params through grid search
  2. find the remaining best tree-based params through grid search, having fixed other params in step 1
  3. decrease the learning rate and increase n_estimators

My question is whether this method of choosing the best hyperparams through multiple passes leads to overfitting or a cross-validation score that is overoptimistic in tree-based models like RandomForestRegressor or GradientBoostingRegressor.


1 Answer 1


No, you don't over-fit

The method you chose to tune your parameters is independent of whether your performance estimation is biased (over-estimated) or not.

As long as you use CV for your parameter-tuning, you cannot "over-fit" in general (meaning to create a too complex model) but only be "over-precise" (like deciding whether using 319 or 318 trees is the best...) which, apart from consuming time, does not really harm you.

Edit: just to mention, if you do have small data-sets, really at the lower limit of what you feel comfortable, you can "overfit" to some extend. Simple tip: if you are using any boosting algorithms, don't go for the last 1 percent of the score; if you use any other algorithm, let the L2-regularization (or other, similar regularization) better be a little higher then the "optimum".

Beside: the site you mentioned has some good tips, mostly the one with the n_estimators/learning_rate ratio that can be increased in the end (only!).

(Remark: In very special cases, where you know that your training data is quite different from what you will predict on, pure CV-based optimization is not sufficient but there needs to be extended, problem specific tuning)

  • $\begingroup$ In addition to the question of overfitting, I was also curious about evaluating model performance - it appears that the entire model building process would need to be repeated to actually estimate performance accurately, which is not computationally practical using this procedure $\endgroup$
    – Brian Bien
    Feb 1, 2017 at 15:56
  • $\begingroup$ What exactly do you mean by "model building process"? And what do you mean by "estimate performance"? Do you want to use a holdout set to estimate the performance? There are several ways to "estimate performance";) $\endgroup$
    – Mayou36
    Feb 2, 2017 at 10:38
  • $\begingroup$ Suppose you then use params tuned using this approach to then estimate performance of a GBR using K-fold CV. The hyperparams selected have been chosen using multiple passes through the dataset, in contrast to the more common K-fold CV pipeline that selects hyperparams inside each fold. $\endgroup$
    – Brian Bien
    Feb 5, 2017 at 16:15
  • $\begingroup$ There is the possibility to make a holdout set: you don't use all of the data for your cv-optimization but instead predict in the end on that holdout set (which is somehow a more reliable measure for how well your algorithm does as this set has not yet been involved into the hyper-optimization). If you have small data-sets in general, you can use this approach (as CV on the whole data can slightly over-fit). But you best read about the holdout-set strategy. And yes, I agree, if you want to do this K-folded as well, you have a huge computation time... $\endgroup$
    – Mayou36
    Feb 5, 2017 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.