My understanding is that a regressor has to be used to fit to the residual. Is it possible to directly apply a classifier? If so, what are the requirements/restrictions?


I also struggled to understand why gradient boosting uses a regressor to solve a classification problem.

It is possible to use classifiers, though. AdaBoost, which is the special case Gradient Boosting was derived from, uses $\{-1,1\}$ classifiers. Simplified, AdaBoost learns a simple classifier, then checks which points it got wrong, and learns a new classifier that puts more weight on the ones that were wrong before.

More generally, when you compute the residuals, you can consider a binary classification problem, where you try to learn the sign of the residual. This is kind of what Adaboost is doing.

The thing to see though is that even though you may be working on a binary classification problem, you are trying to minimize a loss function, let say the logistic loss, which takes as input the predicted probability of belonging to the positive class; a continuous values between 0 and 1. This is why regressors as weak learners work very well.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.