The xgboost package and the algorithm behind it are often mentioned in data science competitions. The method is called extreme gradient boosting. I would like to learn how it differs from typical gradient boosting, but I can not find a rigorous reference for its theoretical basis on the Web. What is the right reference for xgboost?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ Please consider accepting the answer referring to the xgboost paper, since it is the authoritative source for xgboost information. $\endgroup$– Sycorax ♦Commented Feb 2, 2018 at 20:06
-
$\begingroup$ @Sycorax you mean the newer answer? $\endgroup$– Richi WCommented Feb 10, 2018 at 10:44
-
$\begingroup$ That's the one. $\endgroup$– Sycorax ♦Commented Feb 10, 2018 at 15:40
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
Source: Tianqi Chen's Quora answer
Both xgboost and gbm follows the principle of gradient boosting. There are however, the difference in modeling details. Specifically, xgboost used a more regularized model formalization to control over-fitting, which gives it better performance.
We have updated a comprehensive tutorial on introduction to the model, which you might want to take a look at. Introduction to Boosted Trees
The name xgboost, though, actually refers to the engineering goal to push the limit of computations resources for boosted tree algorithms. Which is the reason why many people use xgboost. For model, it might be more suitable to be called as regularized gradient boosting.
-
$\begingroup$ Right .. this sees to be it .. I though he had written a paper about it but the regularization seems to be the key ... at least I understaood that ;) Thanks! $\endgroup$– Richi WCommented Oct 21, 2015 at 10:39
-
-
1$\begingroup$ @Richard Actually we do have a paper: jmlr.org/proceedings/papers/v42/chen14.pdf $\endgroup$– Tong HeCommented Dec 15, 2015 at 23:28
$\begingroup$
$\endgroup$
There is now an arXiv article XGBoost: A Scalable Tree Boosting System that describes the algorithm. At the time of the original question and answer this had not yet been posted.
