10
$\begingroup$

For a classification problem (assume that the loss function is the negative binomial likelihood), the gradient boosting (GBM) algorithm computes the residuals (negative gradient) and then fit them by using a regression tree with mean square error (MSE) as the splitting criterion. How is that different from the XGBoost algorithm?

Does XGBoost utilize regression trees to fit the negative gradient? Is the only difference between GBM and XGBoost the regularization terms or does XGBoost use another split criterion to determine the regions of the regression tree?

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ XGBoost is a particular implementation of GBM that has a few extensions to the core algorithm (as do many other implementations) that seem in many cases to improve performance slightly. You can specify your own loss function or use one of the off-the-shelf ones. $\endgroup$
    – jbowman
    Mar 1, 2018 at 14:29
  • $\begingroup$ I have modified slightly my question. My main question is whether XGBoost utilizes regression trees to fit the negative gradient with mse as the split criterion? $\endgroup$
    – gnikol
    Mar 1, 2018 at 15:00
  • 1
    $\begingroup$ You don't have to use mse, but you can. $\endgroup$
    – jbowman
    Mar 1, 2018 at 15:46

3 Answers 3

Reset to default
6
$\begingroup$

@jbowman has the right answer: XGBoost is a particular implementation of GBM.

GBM is an algorithm and you can find the details in Greedy Function Approximation: A Gradient Boosting Machine.

XGBoost is an implementation of the GBM, you can configure in the GBM for what base learner to be used. It can be a tree, or stump or other models, even linear model.

Here is an example of using a linear model as base learning in XGBoost.

How does linear base learner works in boosting? And how does it works in the xgboost library?

Improve this answer
$\endgroup$
9
  • $\begingroup$ Thank you for your answer but I still do not get it. I have read the paper you cite and in step 4 of Algorithm 1 it uses the square loss to fit the negative gradient and in step 5 uses the loss function to find the optimal step. I have also read "Higgs Boson Discovery with Boosted Trees" which explains XGBoost and if I understand it correctly in order to determine the best split uses the loss function which need to be optimized and computes the loss reduction. It doesn't say anything about the square loss. $\endgroup$
    – gnikol
    Mar 1, 2018 at 16:06
  • 1
    $\begingroup$ @gnikol then what's your question? you are not connecting gmb paper with xgboost implementation? have you read this one? xgboost.readthedocs.io/en/latest/model.html $\endgroup$
    – Haitao Du
    Mar 1, 2018 at 16:07
  • $\begingroup$ I was trying to understand how are the two algorithms connected. At first I though that the only difference was the regularization terms. But I got lost regarding how XGBoost determines the tree structure. I know that GBM uses regression tree to fit the residual. And my question was whether XGBoost uses the same process but adds a regularization component. $\endgroup$
    – gnikol
    Mar 1, 2018 at 16:13
  • $\begingroup$ @gnikol If I remember correctly, XGboost is also using regression tree to fit. Even it is a classification problem. $\endgroup$
    – Haitao Du
    Mar 1, 2018 at 17:06
  • 1
    $\begingroup$ @gnikol if you want to know the details, why no check the source code of xgboost? $\endgroup$
    – Haitao Du
    Mar 1, 2018 at 17:38
1
$\begingroup$

the gradient boosting (GBM) algorithm computes the residuals (negative gradient) and then fit them by using a regression tree with mean square error (MSE) as the splitting criterion. How is that different from the XGBoost algorithm?

Both indeed fit a regression tree to minimize MSE w.r.t. a pseudo-response variable in every boosting iteration. This pseudo response is computed based on the original binary response, and predictions from the regression trees of previous iterations.

The two methods differ in how the pseudo response is computed. GBM uses a first-order derivative of the loss function at the current boosting iteration, while XGBoost uses both the first- and second-order derivatives. The latter is also known as Newton boosting.

A.f.a.i.k. R package gbm uses gradient boosting, by default. At each boosting iteration, the regression tree minimizes the least squares approximation to the negative gradient.

Is the only difference between GBM and XGBoost the regularization terms or does XGBoost use another split criterion to determine the regions of the regression tree?

I do not fully understand what you mean by the regularization term. The split criterion for the regression tree indeed differs, because XGBoost computes the pseudo-response variable differently. At each boosting iteration, it fits a regression tree to minimize a weighted least squares approximation. The pseudo response is calculated so that the more extreme the predictions from previous trees (i.e., linear predictor far from zero; predicted probability close to either 0 or 1), the less influence the observation will receive in the current iteration. Thus, the gradient is weighted by the uncertainty of predictions from previous trees.

Sigrist (2021) provides a good discussion of the differences, both in terms of predictive performance and the functions being optimized.

Sigrist, F. (2021). Gradient and Newton boosting for classification and regression. Expert Systems With Applications, 167, 114080. https://arxiv.org/abs/1808.03064

Improve this answer
$\endgroup$
-2
$\begingroup$

Both are the same XG boost and GBM, both works on the same principle. In Xg boost parallel computation is possible, means in XG boost parallelly many GBM's are working. In Xgboost tunning parameters are more. Any of them can be used, I choose to go with XG boost due to some few more tuning parameters, giving slightly more accuracy.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.