Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I've been learning about machine learning boosting methods (e.g., ADA boost, gradient boost) and the information sources mentioned that boosting tree methods are less likely to overfit than other machine learning methods. Why would that be the case?
Since boosting overweights inputs that were not predicted correctly, it seems like it could easily end up fitting the noise and overfitting the data, but I must be misunderstanding something.
The general idea is that each individual tree will over fit some parts of the data, but therefor will under fit other parts of the data. But in boosting, you don't use the individual trees, but rather "average" them all together, so for a particular data point (or group of points) the trees that over fit that point (those points) will be average with the under fitting trees and the combined average should neither over or under fit, but should be about right.
As with all models, you should try this out on some simulated data to help yourself understand what is going on. Also, as with all models, you should look at diagnostics and use your knowledge of the science and common sense to make sure that the modeling represents your data reasonably.
This is one of those things that has been observed for a while but not necessarily theoretically explained. In one of the original random forest papers, Breiman hypothesized that adaboost functions as a kind of random forest in its latter stage as the weights are essentially drawn from a random distribution. His full supposition hasn't been proven but gives reasonable intuition.
In modern gradient boosting machines etc it is common to use the learning rate and sub-sampeling of the data features to make the tree growth explicitly randomized.
Its also notable that their are relatively few hyper-paramaters to tune and they function pretty directly to combat overfitting. So while it is possible to overfit with a boosted model its also easy to dial back the tree depth, leaf size, learning rate etc and/or add in randomization to combat this.
make the tree growth explicitly randomized, why would this randomized growth of trees make gbdt less prone to overfitting then?
$\endgroup$
This is not a very formal justification, but the discussion in this article provides some interesting perspective on this question. I would recommend reading the article itself (it is fairly short and not too technical), but here is an overview of the basic argument:
The way boosting selects trees is algorithmically similar to a technique for computing the "regularization path" of LASSO (i.e. the set of solutions to $\min_\beta ||y - X\beta||_2^2 + \lambda ||\beta||_1$ as $\lambda$ varies), called Least Angle Regression, which was first introduced by many of the same authors here. Moreover, the way boosting works (successively adding more trees from a model with no predictors) can be thought of as tracing out a path from having a lot of regularization (high $\lambda$, so small $||\beta||_1$ with many 0s) to having very little regularization (and therefore larger $||\beta||_1$) with more 0s). In this view then, boosting as an algorithm tends to guarantee that the solutions found are ones that are "sparse", while early stopping can be thought of as a numerically efficient way of doing cross-validation to tune an implicit $L^1$ regularization parameter.
The authors then discuss the virtues of $L^1$ regularization using what they call the "bet on sparsity" principle to justify why this algorithmic view of boosting might explain why it seems to work well.
