5
$\begingroup$

Background:
Dropout regularization reduces overfitting in Neural networks, especially deep belief networks (srivastava14a). It also has the opportunity to accelerate learning because individual learning iterations are on a reduced set of the model.

The gradient boosted tree (like those xgboost or gbm) is known for being an excellent ensemble learner, but one that suffers from over-fitting.

Question:
Is there an analog to dropout regression that is used in GBM learning?
Does "subsampling" or "stochastic gradient learning" relate to this - does it count as a version of dropout regression?

I think that the stochastic gradient learning is about "randomly disabling inputs" and not "randomly disabling individual learners".

$\endgroup$
  • 3
    $\begingroup$ For me neural net dropout is analogue to random variable subspace aka. mtry of random forests. In the RF model each node can only be split by a random fraction of features, hereby achieving more a regularized ensemble. I see that the [xgboost package][1] has a colsample_bytree option which can reduce the number of variables available for each tree. This could be seen as an analouge to dropout. [1]: cran.r-project.org/web/packages/xgboost/xgboost.pdf $\endgroup$ – Soren Havelund Welling Mar 9 '16 at 13:26
  • $\begingroup$ Does gradient boosting suffer from overfitting? With proper use of early stopping and learning rates, I have always been able to use it effectively. $\endgroup$ – Matthew Drury Sep 1 '16 at 14:56
  • $\begingroup$ @MatthewDrury It can still overfit (easier than Random Forests for example) though. $\endgroup$ – Firebug May 8 '18 at 16:44
  • 1
    $\begingroup$ @Firebug Huh. I wonder what I was thinking when I posted that... $\endgroup$ – Matthew Drury May 8 '18 at 16:55
  • $\begingroup$ @MatthewDrury Haha happens to us all xD $\endgroup$ – Firebug May 8 '18 at 17:15
6
$\begingroup$

Check out this paper: DART: Dropouts meet Multiple Additive Regression Trees (Arxiv PDF).

Their interpertation of dropout is this: instead of developing the next tree from the residual of all previous trees, develop the next tree from the residual of a sample of previous trees. The effect on the model is similar in that individual components are forced to be more self-sufficient. They observe some reasonably significant gains.

As Soren points out, colsample_bytree and colsample_bylevel are analogous to input-layer dropout.

DART is available in xgboost already by setting booster="dart"

$\endgroup$
  • $\begingroup$ my version says "gblinear" or "gbtree" and does not give "dart". Is there another library that I need for xgboost in r to do this? $\endgroup$ – EngrStudent Sep 1 '16 at 16:15
  • 2
    $\begingroup$ It got added very recently, so your version may be slightly out of date or they may not have pushed it to the stable channels. You can get the most up-to-date version by installing from their github repository. github.com/dmlc/xgboost. For R: github.com/dmlc/xgboost/tree/master/R-package $\endgroup$ – Dex Groves Sep 1 '16 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.