It is commonly said that boosted algorithms (adaboost, gradient boosted trees) are composed of many "weak" learners. Let's stick to decision trees as the base learners. Some empirical studies recommended using trees with something like 5-10 terminal nodes (e.g. stochastic gradient boost paper). However, these results were obtained when computation was more expensive, and datasets were smaller/ had lower dimensionality than contemporary ones. Nowadays, people win kaggle competitions by using deep boosted trees (base learners with depth 6 or even 10 are not uncommon). These are arguably weak learners, which begs the question: Are weak learners necessary at all?

This answer from stats stackexchange, suggests that the reason for using weak learners was that they are faster to train. This seems to be only part of the picture: An early motivation of boosted models was that weak learners are less prone to overfitting. At the same time, some sources suggest that too strong a learner can lead to overfitting, while too weak a learner can lead to poor performance (see "Caveats" slide).

Putting all of this together, I am left with a huge parameter space to scan when training an algorithm (base learner depth can be 1-10, iterations can be 100-2000, learning rate can be 0.01 to 0.99). One simplification could be to say: I am going to vary the depth of a single decision tree until I get an AUC of between 0.5 and 0.55 on my dataset, and I will call this my "weak" learner. Then I will use trees of this depth in my boosted algorithm.

Is that the best I can do? Are there some other common rules of thumb that could help (e.g. keep iterations at around 500, and scan learning rate and depth until you get optimal performance, or something along these lines)?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.