For a regression problem, where I'm doing hyperparameter tuning for LGBMRegressor, I was wondering whether I can first tune max_depth and n_estimators and then (in another GridSearchCV round) the regularisation parameters (reg_lambda, learning_rate, reg_alpha)? This will help me save time.
2
-
$\begingroup$ You can do this, but it is typically done with several iterations. See the Wikipedia article about coordinate descent. $\endgroup$– Sextus EmpiricusCommented Jan 8, 2023 at 10:30
-
$\begingroup$ This example has some ideas on the sequential order of xgboost parameter tuning (if you're forced to, rather then a parallel search): xgboosting.com/optimal-order-for-tuning-xgboost-hyperparameters $\endgroup$– jasonbCommented May 19 at 23:15
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
You won't necessarily reach the optimal hyperparameters that way.
The n_estimators and learning_rate are maybe the easiest to see: as you reduce the learning rate, you need more trees to reach the same level of model capacity. In my experience, lowering the learning rate pretty far improves performance, but requires more trees.
Other hyperparameters will generally have the same issue: the optimum value of one depends on the values of others. I don't have enough experience playing with all of them to give confident advice, but max_depth is likely one of the most important hyperparameters, and since its optimal value is usually fairly small and it only takes integer values, optimizing it first might not hurt too much. The regularization parameters tend to have less impact on performance, so tuning those last may be OK.
answered Jan 7, 2023 at 17:36
-
$\begingroup$ Thank you. And what about reg_lambda and reg_alpha? $\endgroup$– GoodQuesCommented Jan 7, 2023 at 17:41
-
$\begingroup$ I want to start with general values for the parameters and narrow down (to find the best ones), but if I do them all at once then each run takes about 40 minutes $\endgroup$– GoodQuesCommented Jan 7, 2023 at 17:44
-
$\begingroup$ There are other ways to try to speed up the search, but if all else fails then you might decide to trade some optimality for shorter training, sure. $\endgroup$ Commented Jan 8, 2023 at 1:25
$\begingroup$
$\endgroup$
This is actually a common strategy implemented in some software (e.g. in optuna) and commonly used in practice. As indicated in the other answer, it may not lead to the best result given infinite training time, but may actually be better than alternatives if you only have a fixed amount of time available.
The argument for why it might work reasonably well to tune certain parameters in turn is that some of them depend not so much on the other parameters (e.g. in this order: 1. depth of trees 2. proportion of records you subsample per tree 3. minimum child weight 4. proportion to sub-sample predictor columns). Some examples discussing such strategies are this Kaggle forum post, this tutorial on Kaggle. Some people are less than convinced, as e.g. seen in this discussion.
