The extreme-gradient boosting algorithm seems to be widely applied these days. I often have the feeling that boosted models tend to overfit. I know that there are parameters in the algorithm to prevent this. Sticking to the documentation here the parameters subsample and colsample_bytree could (among others) prevent overfitting. But they do not serve for the same purpose as bagging xgboosted models would - right?

My question: would you apply bagging on top of xgboost to reduce the variance of the fit?

So far the question is statistical and I dare to add a code detail: in case bagging makes sense I would be happy about example code using the R package caret.

EDIT after the remark: if we rely on the parameters only to control the overfit, then how can we design the cross-validation best? I have approx. 6000 data points and apply 5-fold x-validation. What could improve the out-of-sample performance: going to something like 10-fold x-validation or doing repeated 5-fold x-validation? Just to mention: I use the package cartet where such strategies are implemented.

  • 1
    $\begingroup$ Just a comment. You didn't mention the learning rate of boosted models explicitly, which is extremely important in preventing over-fitting. $\endgroup$ – Matthew Drury Mar 25 '16 at 16:29
  • $\begingroup$ Could work, but ensembles of ensembles can grow quite big. It may be more efficient to find a appropriate set of training parameters not leading to over fitting for a given data set. $\endgroup$ – Soren Havelund Welling Mar 27 '16 at 13:43
  • $\begingroup$ @SorenHavelundWelling please see my edit. $\endgroup$ – Ric Mar 29 '16 at 6:47
  • 1
    $\begingroup$ link.springer.com/article/10.1186/1758-2946-6-10 If I were to publish some A-grade ML model I would go for the proposed Algorithm 3: repeated grid-search cross-validation for variable selection and parameter tuning. I don't use caret that much (I should). As I remember caret do not provide a outer cross-validation for a grid-search. I would feel comfortable by wrapping a caret grid search in a outer 5 or 10fold-CV loop and check if each fold optimal paramters close to the same. For final model, pick the typical parameter set from folds and use outer CV as error estimation. $\endgroup$ – Soren Havelund Welling Mar 29 '16 at 20:42
  • $\begingroup$ @SorenHavelundWelling I opened up a discussion about overfitting here: stats.stackexchange.com/questions/204489/… in case you want to join. $\endgroup$ – Ric Mar 30 '16 at 8:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.