2
$\begingroup$

I have read multiple tutorials that talk about tuning the number of trees (n_estimators or num_boosting_rounds) as a hyperparameter. However, when I train the tuned model, I can see that the loss curves do not fully plateau. My question is, why do we tune the number of trees if it is not going to minimize the loss during the actual training? It seems like we can and should just put a high number and use early stopping (much like the number of epochs in neural networks). Thank you.

$\endgroup$
1
  • $\begingroup$ Agreed, we should not be setting the number of boosting rounds, but instead have it determined automatically by performance on a validation set (e.g. early stopping). $\endgroup$
    – jasonb
    Commented May 16 at 0:42

1 Answer 1

3
$\begingroup$

The number of trees is a key hyperparameters. Keeping all other hyperparameters constant, you usually have that too many trees = overfitting, too few trees = underfitting, and just the right number of trees = just the right level of regularization.

We should not look at the training loss (which sounds like what you are doing), which will of course be able to decrease more and more as we overfit more and more. As long as no two records with different target values have the same exact features, you can eventually fit a XGBoost tree ensemble perfectly to the training data. However, that's not normally what we want to achieve, but we rather tend to be interested in the generalization ability of the model (i.e. prediction on new data). To assess something more like that loss on a validation set (or from cross-validation) tends to be a better thing to look at. For that, you should usually see a decreasing loss, then it should stabilize and eventually rise again (if you don't see that, yet, you just need to add even more trees).

The way one normally tends to tune two of the key hyperparameters, namely, learning rate (aka eta) and number of trees is to set the learning rate to a low value (as low as one can computationally afford, because low is always better, but requires more trees), then do hyperparameter search of some kind over other hyperparameters using cross-validation, while determining the number of trees using early stopping. I.e. as you train, you look at the cross-validation performance as you add more and more trees, but you keep going until loss very clearly is just going up. At that point, you can then go back and see at what number of trees the loss was best across CV samples and that's a sensible number of trees.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.