# dropout regularization in gbm

Background:
Dropout regularization reduces overfitting in Neural networks, especially deep belief networks (srivastava14a). It also has the opportunity to accelerate learning because individual learning iterations are on a reduced set of the model.

The gradient boosted tree (like those xgboost or gbm) is known for being an excellent ensemble learner, but one that suffers from over-fitting.

Question:
Is there an analog to dropout regression that is used in GBM learning?
Does "subsampling" or "stochastic gradient learning" relate to this - does it count as a version of dropout regression?

I think that the stochastic gradient learning is about "randomly disabling inputs" and not "randomly disabling individual learners".

• For me neural net dropout is analogue to random variable subspace aka. mtry of random forests. In the RF model each node can only be split by a random fraction of features, hereby achieving more a regularized ensemble. I see that the [xgboost package][1] has a colsample_bytree option which can reduce the number of variables available for each tree. This could be seen as an analouge to dropout. [1]: cran.r-project.org/web/packages/xgboost/xgboost.pdf – Soren Havelund Welling Mar 9 '16 at 13:26
• Does gradient boosting suffer from overfitting? With proper use of early stopping and learning rates, I have always been able to use it effectively. – Matthew Drury Sep 1 '16 at 14:56
• @MatthewDrury It can still overfit (easier than Random Forests for example) though. – Firebug May 8 '18 at 16:44
• @Firebug Huh. I wonder what I was thinking when I posted that... – Matthew Drury May 8 '18 at 16:55
• @MatthewDrury Haha happens to us all xD – Firebug May 8 '18 at 17:15

As Soren points out, colsample_bytree and colsample_bylevel are analogous to input-layer dropout.
DART is available in xgboost already by setting booster="dart"