Do multiple rounds of tree (GBR) param tuning lead to overoptimistic CV scores? I see suggestions like this for tuning a GradientBoostingRegressor's params through multiple steps. In each step, the best params are selected via CV. For example (abbreviated steps):


*

*find some of the best tree-based params through grid search

*find the remaining best tree-based params through grid search, having fixed other params in step 1

*decrease the learning rate and increase n_estimators 


My question is whether this method of choosing the best hyperparams through multiple passes leads to overfitting or a cross-validation score that is overoptimistic in tree-based models like RandomForestRegressor or GradientBoostingRegressor.
 A: No, you don't over-fit
The method you chose to tune your parameters is independent of whether your performance estimation is biased (over-estimated) or not.
As long as you use CV for your parameter-tuning, you cannot "over-fit" in general (meaning to create a too complex model) but only be "over-precise" (like deciding whether using 319 or 318 trees is the best...) which, apart from consuming time, does not really harm you.
Edit: just to mention, if you do have small data-sets, really at the lower limit of what you feel comfortable, you can "overfit" to some extend. Simple tip: if you are using any boosting algorithms, don't go for the last 1 percent of the score; if you use any other algorithm, let the L2-regularization (or other, similar regularization) better be a little higher then the "optimum".
Beside: the site you mentioned has some good tips, mostly the one with the n_estimators/learning_rate ratio that can be increased in the end (only!).
(Remark: In very special cases, where you know that your training data is quite different from what you will predict on, pure CV-based optimization is not sufficient but there needs to be extended, problem specific tuning)
