The extreme-gradient boosting algorithm seems to be widely applied these days. I often have the feeling that boosted models tend to overfit. I know that there are parameters in the algorithm to prevent this. Sticking to the documentation here the parameters subsample and colsample_bytree could (among others) prevent overfitting. But they do not serve for the same purpose as bagging xgboosted models would - right?

My question: would you apply bagging on top of xgboost to reduce the variance of the fit?

So far the question is statistical and I dare to add a code detail: in case bagging makes sense I would be happy about example code using the R package caret.

EDIT after the remark: if we rely on the parameters only to control the overfit, then how can we design the cross-validation best? I have approx. 6000 data points and apply 5-fold x-validation. What could improve the out-of-sample performance: going to something like 10-fold x-validation or doing repeated 5-fold x-validation? Just to mention: I use the package cartet where such strategies are implemented.

  • 2
    $\begingroup$ Just a comment. You didn't mention the learning rate of boosted models explicitly, which is extremely important in preventing over-fitting. $\endgroup$ – Matthew Drury Mar 25 '16 at 16:29
  • $\begingroup$ Could work, but ensembles of ensembles can grow quite big. It may be more efficient to find a appropriate set of training parameters not leading to over fitting for a given data set. $\endgroup$ – Soren Havelund Welling Mar 27 '16 at 13:43
  • $\begingroup$ @SorenHavelundWelling please see my edit. $\endgroup$ – Ric Mar 29 '16 at 6:47
  • 1
    $\begingroup$ link.springer.com/article/10.1186/1758-2946-6-10 If I were to publish some A-grade ML model I would go for the proposed Algorithm 3: repeated grid-search cross-validation for variable selection and parameter tuning. I don't use caret that much (I should). As I remember caret do not provide a outer cross-validation for a grid-search. I would feel comfortable by wrapping a caret grid search in a outer 5 or 10fold-CV loop and check if each fold optimal paramters close to the same. For final model, pick the typical parameter set from folds and use outer CV as error estimation. $\endgroup$ – Soren Havelund Welling Mar 29 '16 at 20:42
  • $\begingroup$ @SorenHavelundWelling I opened up a discussion about overfitting here: stats.stackexchange.com/questions/204489/… in case you want to join. $\endgroup$ – Ric Mar 30 '16 at 8:03

The bag in bagging is about aggregation. If you have k CART models then for an input you get k candidate answers. How do you reduce that to a single value. The aggregation does that. It is often a measure of central tendency like the mean or mode.

In order to aggregate, you would need multiple outputs. The gradient boosted machine (gbm) as in XGboost, is a series ensemble, not parallel one. This means that it lines them all up in a bucket brigade, and all the learners (but front and back) take the output of one, and give it to the next one). The final output is the same structure as a CART model - a single output. There is no bootstrapping to be done on a single element.

  • 1
    $\begingroup$ Nice to see a good answer to a long-unanswered question (+1). The serial nature of gradient boosted machines is why slow learning at each step is so important to avoid the overfitting noted by the OP. $\endgroup$ – EdM Jan 4 at 20:07
  • $\begingroup$ Slow learning does several good things. There isn't momentum, so the best resolving power of the learner is governed by the single learning rate. $\endgroup$ – EngrStudent Jan 25 at 12:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.