This question already has an answer here:

I found this post : https://www.analyticsvidhya.com/blog/2017/06/which-algorithm-takes-the-crown-light-gbm-vs-xgboost/

However, they did not explain clearly in terms of mathematical model. Could anyone make it more clearly please ?


marked as duplicate by Ferdi, mkt, mdewey, kjetil b halvorsen, Peter Flom Feb 21 at 22:45

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ I your question about speed or mathematical model? Those are two completely different things. Speed may depend on many different things, e.g. quality of the implementation, or technology that was used. $\endgroup$ – Tim Nov 11 '17 at 16:12
  • $\begingroup$ I focus on the mathematical model. What are the differences between them ? $\endgroup$ – kgk Nov 11 '17 at 16:21
  • 1
    $\begingroup$ I edited your title to make it more clear. $\endgroup$ – Tim Nov 11 '17 at 16:28
  • $\begingroup$ I think the main difference is in how the algorithms explore the space of possible decision trees. Most tree growing algorithms use a greedy search strategy, split decisions are made in a locally optimal way, but that doesn't necessary lead to the optimal decision tree. XGBoost, like most other decision tree based algorithms I've seen, appears to use a breadth first greedy approach to grow a tree. LightGBM appears to use a depth first greedy search strategy. $\endgroup$ – Kyle Nov 11 '17 at 19:27

Both build ensembles of trees in an iterative way denoted as boosting. Mathematically, the main difference is the way how a new tree is added to the ensemble. XGBoost uses a functional version of Newton's method (https://en.wikipedia.org/wiki/Newton%27s_method_in_optimization) whereas LightGBM uses functional gradient descent. For gradient boosting, in every iteration, a tree is fitted to the negative gradient of the empirical risk using least squares. This is similar to classical gradient descent (https://en.wikipedia.org/wiki/Gradient_descent) with the difference that the descent is done in a function space and not a finite-dimensional space. For Newton boosting, a tree is found by minimizing a (functional) second order approximation to the empirical risk. The latter can be done using weighted least squares. Note that for standard regression with a squared error, the two versions of boosting are equivalent. Apart from this, these two packages also differ in the way they handle data with large sample sizes.

For more details, this preprint https://arxiv.org/abs/1808.03064 explains the difference between Newton and gradient booting.

Disclaimer: I am the author of the article.

  • $\begingroup$ Welcome to CV! As it currently stands, your answer looks more to a comment than a real answer. Please provide more details. Moreover, you should give the full reference for your article in addition to the link. Finally, you should disclose that you're the author of the article. Thank you! $\endgroup$ – Antoine Feb 21 at 8:00

Not the answer you're looking for? Browse other questions tagged or ask your own question.