I understand that you can overfit a Gradient Boosting Machine (GBM) by using too many trees (unlike random forest), and also that you can overfit a GBM by using too high of a learning rate. My question is, if I simultaneously lower the learning rate and increase the number of trees in an inversely proportionate manner, could I expect to change the amount of overfitting in a predictable way? More precisely, could I expect this to decrease the extent of overfitting?
$\begingroup$
$\endgroup$
1
1 Answer
$\begingroup$
$\endgroup$
Just to note, a very large learning rate can result in underfitting too.
Returning to your main question: "yes". Lowering the learning rate while increasing the number of trees will likely result in decreasing the extent of overfitting, but that is not a given and it will be likely a dataset dependant phenomenon. In general, the decreased learning rate can potentially reduce overfitting, while the increased number of trees can compensate for the more conservative updates, by capturing more complex patterns (i.e. we explore our function space taking more, but smaller, steps). To be clear, though: The best approach depends on our specific dataset, using some a GBM implementation with in-build cross-validation for early stopping (e.g. LightGBM) is probably our best bet to get the balance between the two hyperparameters.
min_deltain LightGBM sklearn API). Learning-rate/number-of-trees scheme seems more complicated because the exact proportion is unclear until you run enough experiments, and it would certainly depend on the other hyperparams like tree depth. $\endgroup$