2
$\begingroup$

It is not fully clear to me if Gradient Boosted Trees are boosting on the residuals of each stage or on the complete training set?

From the Wikipedia page it seems to me that the Overal "Algorithm" as explained for Gradient Boosting is boosting on the residuals but when the article explains "Gradient tree boosting" - residuals are never mentioned - emphasis is given to gamma-i which weights each individual weak learner added to the model.

So what is true now?

  1. Is Gradient Boosted Tree boosting on the residuals or on the complete training set?

  2. Gradient Boosted Tree weighting individual training examples as for example explained in AdaBoost?

$\endgroup$
  • $\begingroup$ What do you mean by "boosting on" the residuals? $\endgroup$ – user20160 Oct 12 '19 at 14:44
2
$\begingroup$

The https://xgboost.readthedocs.io/en/latest/tutorials/model.html [documentation of XGBoost][1] provides a great introduction to the boosted trees algorithm.

Perhaps, I can help out answering your first question:

I think you state two false alternatives. The algorithm uses the entire training set (leaving aside the fact that you can sample observations in every iteration). Plus, every tree is fitted on the residuals of the previous trees.

Concerning the latter: In the gradient descent method, we try to reach the global minimum of an objective function (loss function) by walking through the solution space along the gradient. You can consider every added tree, a further step towads the greatest descending slope direction. Since the gradient of the loss functions logloss or mean square error happen to be the difference between prediction and observation, you can interpret every tree to be fitted to the previous residuals.

However, I'm not sure if you can use this interpretation for every loss function (beside from logloss and mean square error).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.