9
$\begingroup$

The extreme-gradient boosting algorithm seems to be widely applied these days. I often have the feeling that boosted models tend to overfit. I know that there are parameters in the algorithm to prevent this. Sticking to the documentation here the parameters subsample and colsample_bytree could (among others) prevent overfitting. But they do not serve for the same purpose as bagging xgboosted models would - right?

My question: would you apply bagging on top of xgboost to reduce the variance of the fit?

So far the question is statistical and I dare to add a code detail: in case bagging makes sense I would be happy about example code using the R package caret.

EDIT after the remark: if we rely on the parameters only to control the overfit, then how can we design the cross-validation best? I have approx. 6000 data points and apply 5-fold x-validation. What could improve the out-of-sample performance: going to something like 10-fold x-validation or doing repeated 5-fold x-validation? Just to mention: I use the package cartet where such strategies are implemented.

Improve this question
$\endgroup$
6
  • 3
    $\begingroup$ Just a comment. You didn't mention the learning rate of boosted models explicitly, which is extremely important in preventing over-fitting. $\endgroup$
    – Matthew Drury
    Commented Mar 25, 2016 at 16:29
  • $\begingroup$ Could work, but ensembles of ensembles can grow quite big. It may be more efficient to find a appropriate set of training parameters not leading to over fitting for a given data set. $\endgroup$
    – Soren Havelund Welling
    Commented Mar 27, 2016 at 13:43
  • $\begingroup$ @SorenHavelundWelling please see my edit. $\endgroup$
    – Richi W
    Commented Mar 29, 2016 at 6:47
  • 1
    $\begingroup$ link.springer.com/article/10.1186/1758-2946-6-10 If I were to publish some A-grade ML model I would go for the proposed Algorithm 3: repeated grid-search cross-validation for variable selection and parameter tuning. I don't use caret that much (I should). As I remember caret do not provide a outer cross-validation for a grid-search. I would feel comfortable by wrapping a caret grid search in a outer 5 or 10fold-CV loop and check if each fold optimal paramters close to the same. For final model, pick the typical parameter set from folds and use outer CV as error estimation. $\endgroup$
    – Soren Havelund Welling
    Commented Mar 29, 2016 at 20:42
  • $\begingroup$ @SorenHavelundWelling I opened up a discussion about overfitting here: stats.stackexchange.com/questions/204489/… in case you want to join. $\endgroup$
    – Richi W
    Commented Mar 30, 2016 at 8:03

1 Answer 1

Reset to default
3
$\begingroup$

The bag in bagging is about aggregation. If you have k CART models then for an input you get k candidate answers. How do you reduce that to a single value. The aggregation does that. It is often a measure of central tendency like the mean or mode.

In order to aggregate, you would need multiple outputs. The gradient boosted machine (gbm) as in XGboost, is a series ensemble, not parallel one. This means that it lines them all up in a bucket brigade, and all the learners (but front and back) take the output of one, and give it to the next one). The final output is the same structure as a CART model - a single output. There is no bootstrapping to be done on a single element.

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ Nice to see a good answer to a long-unanswered question (+1). The serial nature of gradient boosted machines is why slow learning at each step is so important to avoid the overfitting noted by the OP. $\endgroup$
    – EdM
    Commented Jan 4, 2021 at 20:07
  • $\begingroup$ Slow learning does several good things. There isn't momentum, so the best resolving power of the learner is governed by the single learning rate. $\endgroup$
    – EngrStudent
    Commented Jan 25, 2021 at 12:44
  • $\begingroup$ You can do bagging of several XGBoost/GBM models, each of which is considered a black box rather than an ensemble. The question is whether this is going to be efficient, and the answer is probably no (not enough randomisation for efficient variance reduction). $\endgroup$
    – paperskilltrees
    Commented Mar 31, 2023 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.