34
$\begingroup$

I am trying to understand how XGBoost works. I already understand how gradient boosted trees work on Python sklearn. What is not clear to me is if XGBoost works the same way, but faster, or if there are fundamental differences between it and the python implementation.

When I read this paper

http://learningsys.org/papers/LearningSys_2015_paper_32.pdf

It looks to me like the end result coming out of XGboost is the same as in the Python implementation, however the main difference is how XGboost finds the best split to make in each regression tree.

Basically, XGBoost gives the same result, but it is faster.

Is this correct, or is there something else I am missing ?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
30
$\begingroup$

You are correct, XGBoost ('eXtreme Gradient Boosting') and sklearn's GradientBoost are fundamentally the same as they are both gradient boosting implementations.

However, there are very significant differences under the hood in a practical sense. XGBoost is a lot faster (see http://machinelearningmastery.com/gentle-introduction-xgboost-applied-machine-learning/) than sklearn's. XGBoost is quite memory-efficient and can be parallelized (I think sklearn's cannot do so by default, I don't know exactly about sklearn's memory-efficiency but I am pretty confident it is below XGBoost's).

Having used both, XGBoost's speed is quite impressive and its performance is superior to sklearn's GradientBoosting.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ There is also a performance difference. Xgboost used second derivatives to find the optimal constant in each terminal node. The standard implementation only uses the first derivative. $\endgroup$
    – Zelazny7
    Jun 10, 2017 at 0:58
  • $\begingroup$ @Zelazny7 Do you have any references for your statement ? One would expect that calculating 2nd derivatives would degrade performance. It also would mean that something besides (any flavor of) gradient descent was used. $\endgroup$
    – meh
    Jul 14, 2017 at 12:45
  • $\begingroup$ medium.com/towards-data-science/… $\endgroup$
    – Zelazny7
    Jul 14, 2017 at 12:52
  • 2
    $\begingroup$ Relevant Q&A on Datascience: Why is xgboost so much faster than sklearn and here os Stats: XGBoost Loss function Approximation With Taylor Expansion $\endgroup$
    – Winks
    Jul 14, 2017 at 13:19
  • 1
    $\begingroup$ Confirmation to @K88's post, sklearn's GradientBoostingClassifier does not have a n_jobs interface parameter available; parallel processing is not possible internally with sklearn's implementation of the algorithm. $\endgroup$
    – bmc
    Jan 24, 2019 at 19:30
8
$\begingroup$

Unlike the Sklearn's gradient boosting, Xgboost does regularization of the tree as well to avoid overfitting and it deals with the missing values efficiently as well. Following link might be helpful to learn xgboost precisely https://www.youtube.com/watch?v=Vly8xGnNiWs

Improve this answer
$\endgroup$
2
3
$\begingroup$

XGboost is implementation of GBDT with randmization(It uses coloumn sampling and row sampling).Row sampling is possible by not using all of the training data for each base model of the GBDT. Instead of using all of the training data for each base-model, we sample a subset of rows and use only those rows of data to build each of the base models. This ensures that there is a lesser chance of overfitting which is a major issue with simple GBDT which XGBoost tries to address using this randomization.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.