I have often read that gradient boosting algorithms fit sequential models to the overall model's residuals, but I can't make sense of this for classification problems (for instance, what is the "residual" here?). In investigating some documentation, it seems that perhaps this is because, even in classification tasks, a gradient boosted tree algorithm is actually using a regression-tree based approach.
Is this the case, and if so what is the classification tree "regressing"? I grant that the output of a ensemble model is a sort of "confidence" rather than a strict label, so one could calculate a numeric difference from either 1 or 0 (as in the binary classification case), but GBT models are built sequentially, so I wouldn't know what "number" it predicts.
As a follow-up, I can't imagine what the gradient is being used to update in classification problems? The parameters of any individual tree are already determined for prior steps, and for future steps are dictated by whatever splitting algorithm (not a gradient).
Thanks, and apologies for the ignorance. I've read a number of posts/blogs, but there always seems to be some sleight of hand when it comes to classification (or just my own ignorance!).