7
$\begingroup$

I've been learning about machine learning boosting methods (e.g., ADA boost, gradient boost) and the information sources mentioned that boosting tree methods are less likely to overfit than other machine learning methods. Why would that be the case?

Since boosting overweights inputs that were not predicted correctly, it seems like it could easily end up fitting the noise and overfitting the data, but I must be misunderstanding something.

$\endgroup$
3
  • $\begingroup$ would you please provide the source? What I learnt is that boosting is more likely to overfit, at least than method like random forest? $\endgroup$
    – user112758
    Jan 20, 2017 at 20:30
  • $\begingroup$ My source was the Coursera University of Washington machine learning sequence (classification course). $\endgroup$ Jan 21, 2017 at 20:33
  • $\begingroup$ Actually, gradient boosting does overfit (at least, much more than random forests) when you increase the number of trees. You may be interested in the following article does gradient boosting overfit where I run some simulations on various datasets $\endgroup$
    – RUser4512
    Dec 20, 2021 at 9:57

3 Answers 3

4
$\begingroup$

The general idea is that each individual tree will over fit some parts of the data, but therefor will under fit other parts of the data. But in boosting, you don't use the individual trees, but rather "average" them all together, so for a particular data point (or group of points) the trees that over fit that point (those points) will be average with the under fitting trees and the combined average should neither over or under fit, but should be about right.

As with all models, you should try this out on some simulated data to help yourself understand what is going on. Also, as with all models, you should look at diagnostics and use your knowledge of the science and common sense to make sure that the modeling represents your data reasonably.

$\endgroup$
4
  • 1
    $\begingroup$ Isn't that the reason why the random forest doesn't overfit? As far as I'm concerned is the principle of gradient boosting that each single trees use the output of the tree built before in a recursive manner, rather than beeing averaged altogether; compare (blog.kaggle.com/2017/01/23/…). Therefore there should be another reason why gradient boosting does not overfit that quickly. $\endgroup$
    – David Go
    Apr 10, 2017 at 10:27
  • $\begingroup$ @DavidGo, In the article you link to it talks about summing the predictions from each tree. Summing is related to averaging, so conceptually is the same. Other forms of boosting from what is mentioned reweight and fit rather than fit the residuals, then average rather than sum (a difference of a constant). the exact algorithm is different, but the concept is the same. $\endgroup$
    – Greg Snow
    Apr 11, 2017 at 15:40
  • $\begingroup$ you've got a point, summing the predictions from each tree is related to averaging. However, the trees are trained to fit the predicted residuals of their previously trained tree. Therefore, the residuals grow smaller by each tree and the following trees are trained to fit a preconditioned problem. Summing single tree results by itself might not by the only reason why the model is unlikely to overfit, as single trees depend on each other and could eventually overfit the training dataset when the residuals grow near zero. $\endgroup$
    – David Go
    Apr 24, 2017 at 11:36
  • $\begingroup$ @DavidGo, the link that you included was for gradient boosting, which is not the only style of boosting, other boosting methods refit to the original data with different weights based on the previous fit. My answer works more directly with these other forms. I have not done the math or other formal proof, but I expect that using the residuals in gradient boosting gives a similar effect to the other forms. $\endgroup$
    – Greg Snow
    Apr 24, 2017 at 19:24
2
$\begingroup$

This is one of those things that has been observed for a while but not necessarily theoretically explained. In one of the original random forest papers, Breiman hypothesized that adaboost functions as a kind of random forest in its latter stage as the weights are essentially drawn from a random distribution. His full supposition hasn't been proven but gives reasonable intuition.

In modern gradient boosting machines etc it is common to use the learning rate and sub-sampeling of the data features to make the tree growth explicitly randomized.

Its also notable that their are relatively few hyper-paramaters to tune and they function pretty directly to combat overfitting. So while it is possible to overfit with a boosted model its also easy to dial back the tree depth, leaf size, learning rate etc and/or add in randomization to combat this.

$\endgroup$
1
  • $\begingroup$ make the tree growth explicitly randomized, why would this randomized growth of trees make gbdt less prone to overfitting then? $\endgroup$
    – avocado
    Mar 30, 2021 at 8:52
0
$\begingroup$

This is not a very formal justification, but the discussion in this article provides some interesting perspective on this question. I would recommend reading the article itself (it is fairly short and not too technical), but here is an overview of the basic argument:

The way boosting selects trees is algorithmically similar to a technique for computing the "regularization path" of LASSO (i.e. the set of solutions to $\min_\beta ||y - X\beta||_2^2 + \lambda ||\beta||_1$ as $\lambda$ varies), called Least Angle Regression, which was first introduced by many of the same authors here. Moreover, the way boosting works (successively adding more trees from a model with no predictors) can be thought of as tracing out a path from having a lot of regularization (high $\lambda$, so small $||\beta||_1$ with many 0s) to having very little regularization (and therefore larger $||\beta||_1$) with more 0s). In this view then, boosting as an algorithm tends to guarantee that the solutions found are ones that are "sparse", while early stopping can be thought of as a numerically efficient way of doing cross-validation to tune an implicit $L^1$ regularization parameter.

The authors then discuss the virtues of $L^1$ regularization using what they call the "bet on sparsity" principle to justify why this algorithmic view of boosting might explain why it seems to work well.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.