Doing hyperparameter tuning in parts For a regression problem, where I'm doing hyperparameter tuning for LGBMRegressor, I was wondering whether I can first tune max_depth and n_estimators and then (in another GridSearchCV round) the regularisation parameters (reg_lambda, learning_rate, reg_alpha)? This will help me save time.
 A: You won't necessarily reach the optimal hyperparameters that way.
The n_estimators and learning_rate are maybe the easiest to see: as you reduce the learning rate, you need more trees to reach the same level of model capacity. In my experience, lowering the learning rate pretty far improves performance, but requires more trees.
Other hyperparameters will generally have the same issue: the optimum value of one depends on the values of others. I don't have enough experience playing with all of them to give confident advice, but max_depth is likely one of the most important hyperparameters, and since its optimal value is usually fairly small and it only takes integer values, optimizing it first might not hurt too much.  The regularization parameters tend to have less impact on performance, so tuning those last may be OK.
A: This is actually a common strategy implemented in some software (e.g. in optuna) and commonly used in practice. As indicated in the other answer, it may not lead to the best result given infinite training time, but may actually be better than alternatives if you only have a fixed amount of time available.
The argument for why it might work reasonably well to tune certain parameters in turn is that some of them depend not so much on the other parameters (e.g. in this order: 1. depth of trees 2. proportion of records you subsample per tree 3. minimum child weight 4. proportion to sub-sample predictor columns). Some examples discussing such strategies are this Kaggle forum post, this tutorial on Kaggle. Some people are less than convinced, as e.g. seen in this discussion.
