I see suggestions like this for tuning a GradientBoostingRegressor's params through multiple steps. In each step, the best params are selected via CV. For example (abbreviated steps):
- find some of the best tree-based params through grid search
- find the remaining best tree-based params through grid search, having fixed other params in step 1
- decrease the learning rate and increase n_estimators
My question is whether this method of choosing the best hyperparams through multiple passes leads to overfitting or a cross-validation score that is overoptimistic in tree-based models like RandomForestRegressor or GradientBoostingRegressor.