20
$\begingroup$

I am trying to understand how XGBoost works. I already understand how gradient boosted trees work on Python sklearn. What is not clear to me is if XGBoost works the same way, but faster, or if there are fundamental differences between it and the python implementation.

When I read this paper

http://learningsys.org/papers/LearningSys_2015_paper_32.pdf

It looks to me like the end result coming out of XGboost is the same as in the Python implementation, however the main difference is how XGboost finds the best split to make in each regression tree.

Basically, XGBoost gives the same result, but it is faster.

Is this correct, or is there something else I am missing ?

$\endgroup$
20
$\begingroup$

You are correct, XGBoost ('eXtreme Gradient Boosting') and sklearn's GradientBoost are fundamentally the same as they are both gradient boosting implementations.

However, there are very significant differences under the hood in a practical sense. XGBoost is a lot faster (see http://machinelearningmastery.com/gentle-introduction-xgboost-applied-machine-learning/) than sklearn's. XGBoost is quite memory-efficient and can be parallelized (I think sklearn's cannot do so by default, I don't know exactly about sklearn's memory-efficiency but I am pretty confident it is below XGBoost's).

Having used both, XGBoost's speed is quite impressive and its performance is superior to sklearn's GradientBoosting.

$\endgroup$
  • 1
    $\begingroup$ There is also a performance difference. Xgboost used second derivatives to find the optimal constant in each terminal node. The standard implementation only uses the first derivative. $\endgroup$ – Zelazny7 Jun 10 '17 at 0:58
  • $\begingroup$ @Zelazny7 Do you have any references for your statement ? One would expect that calculating 2nd derivatives would degrade performance. It also would mean that something besides (any flavor of) gradient descent was used. $\endgroup$ – meh Jul 14 '17 at 12:45
  • $\begingroup$ medium.com/towards-data-science/… $\endgroup$ – Zelazny7 Jul 14 '17 at 12:52
  • 2
    $\begingroup$ Relevant Q&A on Datascience: Why is xgboost so much faster than sklearn and here os Stats: XGBoost Loss function Approximation With Taylor Expansion $\endgroup$ – Winks Jul 14 '17 at 13:19
  • $\begingroup$ Confirmation to @K88's post, sklearn's GradientBoostingClassifier does not have a n_jobs interface parameter available; parallel processing is not possible internally with sklearn's implementation of the algorithm. $\endgroup$ – bmc Jan 24 '19 at 19:30
7
$\begingroup$

Unlike the Sklearn's gradient boosting, Xgboost does regularization of the tree as well to avoid overfitting and it deals with the missing values efficiently as well. Following link might be helpful to learn xgboost precisely https://www.youtube.com/watch?v=Vly8xGnNiWs

$\endgroup$
  • $\begingroup$ SkLearn's GBM does regularization via the parameter learning_rate. $\endgroup$ – Teja Chebrole Sep 5 '19 at 16:08
1
$\begingroup$

XGboost is implementation of GBDT with randmization(It uses coloumn sampling and row sampling).Row sampling is possible by not using all of the training data for each base model of the GBDT. Instead of using all of the training data for each base-model, we sample a subset of rows and use only those rows of data to build each of the base models. This ensures that there is a lesser chance of overfitting which is a major issue with simple GBDT which XGBoost tries to address using this randomization.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.