6
$\begingroup$

Background:
Dropout regularization reduces overfitting in Neural networks, especially deep belief networks (srivastava14a). It also has the opportunity to accelerate learning because individual learning iterations are on a reduced set of the model.

The gradient boosted tree (like those xgboost or gbm) is known for being an excellent ensemble learner, but one that suffers from over-fitting.

Question:
Is there an analog to dropout regression that is used in GBM learning?
Does "subsampling" or "stochastic gradient learning" relate to this - does it count as a version of dropout regression?

I think that the stochastic gradient learning is about "randomly disabling inputs" and not "randomly disabling individual learners".

Improve this question
$\endgroup$
5
  • 3
    $\begingroup$ For me neural net dropout is analogue to random variable subspace aka. mtry of random forests. In the RF model each node can only be split by a random fraction of features, hereby achieving more a regularized ensemble. I see that the [xgboost package][1] has a colsample_bytree option which can reduce the number of variables available for each tree. This could be seen as an analouge to dropout. [1]: cran.r-project.org/web/packages/xgboost/xgboost.pdf $\endgroup$
    – Soren Havelund Welling
    Commented Mar 9, 2016 at 13:26
  • $\begingroup$ Does gradient boosting suffer from overfitting? With proper use of early stopping and learning rates, I have always been able to use it effectively. $\endgroup$
    – Matthew Drury
    Commented Sep 1, 2016 at 14:56
  • $\begingroup$ @MatthewDrury It can still overfit (easier than Random Forests for example) though. $\endgroup$
    – Firebug
    Commented May 8, 2018 at 16:44
  • 1
    $\begingroup$ @Firebug Huh. I wonder what I was thinking when I posted that... $\endgroup$
    – Matthew Drury
    Commented May 8, 2018 at 16:55
  • $\begingroup$ @MatthewDrury Haha happens to us all xD $\endgroup$
    – Firebug
    Commented May 8, 2018 at 17:15

1 Answer 1

Reset to default
8
$\begingroup$

Check out this paper: DART: Dropouts meet Multiple Additive Regression Trees (Arxiv PDF).

Their interpertation of dropout is this: instead of developing the next tree from the residual of all previous trees, develop the next tree from the residual of a sample of previous trees. The effect on the model is similar in that individual components are forced to be more self-sufficient. They observe some reasonably significant gains.

As Soren points out, colsample_bytree and colsample_bylevel are analogous to input-layer dropout.

DART is available in xgboost already by setting booster="dart"

Improve this answer
$\endgroup$
2
  • $\begingroup$ my version says "gblinear" or "gbtree" and does not give "dart". Is there another library that I need for xgboost in r to do this? $\endgroup$
    – EngrStudent
    Commented Sep 1, 2016 at 16:15
  • 2
    $\begingroup$ It got added very recently, so your version may be slightly out of date or they may not have pushed it to the stable channels. You can get the most up-to-date version by installing from their github repository. github.com/dmlc/xgboost. For R: github.com/dmlc/xgboost/tree/master/R-package $\endgroup$
    – Dex Groves
    Commented Sep 1, 2016 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.