How to tune the weak learner in boosted algorithms

Question

It is commonly said that boosted algorithms (adaboost, gradient boosted trees) are composed of many "weak" learners. Let's stick to decision trees as the base learners. Some empirical studies recommended using trees with something like 5-10 terminal nodes (e.g. stochastic gradient boost paper). However, these results were obtained when computation was more expensive, and datasets were smaller/ had lower dimensionality than contemporary ones. Nowadays, people win kaggle competitions by using deep boosted trees (base learners with depth 6 or even 10 are not uncommon). These are arguably weak learners, which begs the question: Are weak learners necessary at all?

This answer from stats stackexchange, suggests that the reason for using weak learners was that they are faster to train. This seems to be only part of the picture: An early motivation of boosted models was that weak learners are less prone to overfitting. At the same time, some sources suggest that too strong a learner can lead to overfitting, while too weak a learner can lead to poor performance (see "Caveats" slide).

Putting all of this together, I am left with a huge parameter space to scan when training an algorithm (base learner depth can be 1-10, iterations can be 100-2000, learning rate can be 0.01 to 0.99). One simplification could be to say: I am going to vary the depth of a single decision tree until I get an AUC of between 0.5 and 0.55 on my dataset, and I will call this my "weak" learner. Then I will use trees of this depth in my boosted algorithm.

Is that the best I can do? Are there some other common rules of thumb that could help (e.g. keep iterations at around 500, and scan learning rate and depth until you get optimal performance, or something along these lines)?

Answer 1

Yes, weak learners are absolutely required for boosting to be really successful. That is because each boosting round for trees actually results in more splits and a more complicated model. This will overfit quite quick if we let it. On the other hand, if we pass a more complicated model such as a large polynomial linear regression then boosting will actually apply some regularization. Although, linear models won't be able to give you the best accuracy when boosting (typically).

You mention kaggle and deep trees. There is a significant difference between standard boosting trees and the base trees used for xgboost or lightgbm (what wins Kaggle). The trees in these algorithms are heavily regularized which allows us to use deeper trees and still keep them 'weak'.

In terms of param spaces to try, it depends on the algo. If you are using some basic boosted tree like scikit-learn's then you will typically be dealing with really short trees (so max depth around 1-3ish). Whereas with xgboost or lightgbm you will get good results typically going deeper to something like 8-16. Learning rates typically are lower around .01 - .1 although I have seen some go up to .5 for optimal solutions.

Overall though, the huge param space search is the price we pay for state-of-the-art results. As Tim mentioned in his answer we typically leverage hyperparameter packages to speed this up like hyperopt or optuna.

Answer 2

The definition of weak learner is:

• Something better than a random guess (1), (2), (3)

There is nothing in that about overfitting, though weakness means overfitting is highly likely. Being a very bad estimator and being a robust estimator tend to not coincide.

Whether or not your learners are weak, you can do a primitive statistical design of experiment on your grid searching to accelerate the overall testing by orders of magnitude.

Instead of uniform, try random sparse. Look at the functions that make autoplotting, how they bisect and look for changes, and use those. Use the multivariate versions.

These two things are not necessarily connected, and the second one has some very good solutions to be had.

Answer 3

One of the practical reasons why we use weak learners is that we don't need to care about issues like this. In many cases ensembling many weak learners is just enough to achieve good performance. Weak learners are simple by design, we don't usually tune them. You are correct, if you wanted to tune them, this becomes a complicated optimization problem. If you really want to do this, one thing that could make things easier is to use a more clever optimization algorithm than grid search, for example Bayesian optimization that "cleverly" explores the parameter space.