3
$\begingroup$

I have often read that gradient boosting algorithms fit sequential models to the overall model's residuals, but I can't make sense of this for classification problems (for instance, what is the "residual" here?). In investigating some documentation, it seems that perhaps this is because, even in classification tasks, a gradient boosted tree algorithm is actually using a regression-tree based approach.

Is this the case, and if so what is the classification tree "regressing"? I grant that the output of a ensemble model is a sort of "confidence" rather than a strict label, so one could calculate a numeric difference from either 1 or 0 (as in the binary classification case), but GBT models are built sequentially, so I wouldn't know what "number" it predicts.

As a follow-up, I can't imagine what the gradient is being used to update in classification problems? The parameters of any individual tree are already determined for prior steps, and for future steps are dictated by whatever splitting algorithm (not a gradient).

Thanks, and apologies for the ignorance. I've read a number of posts/blogs, but there always seems to be some sleight of hand when it comes to classification (or just my own ignorance!).

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

The easiest way to explain gradient boosting is to take the specific case of regression with MSE as the loss function, because there each tree is fitting to the prior trees' (in aggregate) residuals. But more generally, the trees are fitted to the gradient of the loss function. The gradient of MSE being (proportional to) the residual, this agrees with the easy-to-explain specific case.

In the classification context, this is usually the (negative) derivative of log-loss, which isn't really the residual, but it does "move" the ensemble in the direction of smaller loss.

The individual trees are indeed regression trees, and the target they are approximating is the (negative) gradient of the loss function (with respect to the aggregate-predictions-so-far). (Before the first tree, usually some baseline constant prediction is made.) Then, given that, the splitting criteria apply as in any regression tree.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Ah so given "the trees are fitted to the gradient of the loss function", then gradient descent isn't really used to "update" anything (like say in neural net optimization) -- it's more to provide our target at each step! What is that initial "baseline" though that allows the establishment of the problem as a regression one? Maybe just like the log odds based on the simple proportion of classes in the data or something? $\endgroup$
    – Josh
    May 28, 2021 at 18:57
  • 1
    $\begingroup$ Yes, class proportions is probably the most common first score. xgboost though sets that with the parameter base_score, which defaults to just 0.5, and sklearn allows an arbitrary estimator (with default a DummyEstimator which produces the class proportions). $\endgroup$
    – Ben Reiniger
    May 28, 2021 at 19:04
  • 1
    $\begingroup$ Thanks, very succinct answer! I'm amazed at how many blog posts I had to read when you summed it up so quick! $\endgroup$
    – Josh
    May 28, 2021 at 19:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.