For a classification problem(assume that the loss function is the negative binomial likehood) the gradient boosting (GBM) algorithm computes the residuals (negative gradient) and then fit them by using a regression tree with mean square error (mse) as split criterion. How is that compared to the XGBoost algorithm?

Does XGBoost utilizes regression trees to fit the negative gradient? Is the only difference between GBM and XGBoost the regularization terms or XGBoost uses other split criterion to determine the regions of the regression tree?

  • 3
    $\begingroup$ XGBoost is a particular implementation of GBM that has a few extensions to the core algorithm (as do many other implementations) that seem in many cases to improve performance slightly. You can specify your own loss function or use one of the off-the-shelf ones. $\endgroup$ – jbowman Mar 1 '18 at 14:29
  • $\begingroup$ I have modified slightly my question. My main question is whether XGBoost utilizes regression trees to fit the negative gradient with mse as the split criterion? $\endgroup$ – gnikol Mar 1 '18 at 15:00
  • 1
    $\begingroup$ You don't have to use mse, but you can. $\endgroup$ – jbowman Mar 1 '18 at 15:46

@jbowman has the right answer: XGBoost is a particular implementation of GBM.

GBM is an algorithm and you can find the details in Greedy Function Approximation: A Gradient Boosting Machine.

XGBoost is an implementation of the GBM, you can configure in the GBM for what base learner to be used. It can be a tree, or stump or other models, even linear model.

Here is an example of using a linear model as base learning in XGBoost.

How does linear base learner works in boosting? And how does it works in the xgboost library?

  • $\begingroup$ Thank you for your answer but I still do not get it. I have read the paper you cite and in step 4 of Algorithm 1 it uses the square loss to fit the negative gradient and in step 5 uses the loss function to find the optimal step. I have also read "Higgs Boson Discovery with Boosted Trees" which explains XGBoost and if I understand it correctly in order to determine the best split uses the loss function which need to be optimized and computes the loss reduction. It doesn't say anything about the square loss. $\endgroup$ – gnikol Mar 1 '18 at 16:06
  • 1
    $\begingroup$ @gnikol then what's your question? you are not connecting gmb paper with xgboost implementation? have you read this one? xgboost.readthedocs.io/en/latest/model.html $\endgroup$ – Haitao Du Mar 1 '18 at 16:07
  • $\begingroup$ I was trying to understand how are the two algorithms connected. At first I though that the only difference was the regularization terms. But I got lost regarding how XGBoost determines the tree structure. I know that GBM uses regression tree to fit the residual. And my question was whether XGBoost uses the same process but adds a regularization component. $\endgroup$ – gnikol Mar 1 '18 at 16:13
  • $\begingroup$ @gnikol If I remember correctly, XGboost is also using regression tree to fit. Even it is a classification problem. $\endgroup$ – Haitao Du Mar 1 '18 at 17:06
  • 1
    $\begingroup$ @gnikol if you want to know the details, why no check the source code of xgboost? $\endgroup$ – Haitao Du Mar 1 '18 at 17:38

Both are the same XG boost and GBM, both works on the same principle. In Xg boost parallel computation is possible, means in XG boost parallelly many GBM's are working. In Xgboost tunning parameters are more. Any of them can be used, I choose to go with XG boost due to some few more tuning parameters, giving slightly more accuracy.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.