11
$\begingroup$

Adaboost is an ensemble method that combines many weak learners to form a strong one. All of the examples of adaboost that i have read use decision stumps/trees as weak learners. Can i use different weak learners in adaboost? For example, how to implement adaboost (generally boosting) to boost a logistic regression model?

One main difference of classification trees and logistic regression is that the former outputs classes (-1,1) while the logistic regression outputs probs. One idea is to choose the best feature X from a set of features and pick up a threshold (0.5?) to convert the probs to classes and then use a weighted logistic regression to find the next feature etc.

But i imagine that there exists a general algorithm to boost different weak learners different than decision stumps that outputs probabilities. I believed that Logitboost is the answer to my question but i tried to read the "Additive Logistic Regression" paper and got stuck in the middle.

$\endgroup$
7
$\begingroup$

Don't confuse the handling of the predictors (via base learners, e.g. stumps) and the handling of the loss function in boosting. Although AdaBoost can be thought of as finding combinations of base learners to minimize misclassification error, the "Additive Logistic Regression" paper you cite shows that it can also be formulated to minimize an exponential loss function. This insight opened up the boosting approach to a wide class of machine-learning problems that minimize differentiable loss functions, via gradient boosting. The residuals that are fit at each step are pseudo-residuals calculated from the gradient of the loss function. Even if the predictors are modeled as binary stumps, the output of the model thus need not be a binary choice.

As another answer states, linear base learners might not work for boosting, but linear base learners are not required for "boosted regression" in either the standard or the logistic sense. Decidedly non-linear stumps can be combined as slow base learners to minimize appropriate loss functions. It's still called "boosted regression" even though it is far from a standard regression model linear in the coefficients of the predictors. The loss function can be functionally the same for linear models and "boosted regression" models with stumps or trees as predictors. Chapter 8 of ISLR makes this pretty clear.

So if you want a logistic-regression equivalent to boosted regression, focus on the loss function rather than on the base learners. That's what the LogitBoost approach in the paper you cite does: minimize a log-loss rather than the exponential loss implicit in adaboost. The Wikipedia AdaBoost page describes this difference.

Many participants in this site would argue that a log-odds/probability based prediction is highly preferable to a strict yes/no classification prediction, as the former more generally allows for different tradeoffs between the extra costs of false-positive and false-negative predictions. As the answer to your related question indicates, it is possible to obtain estimated probabilities from the strong classifier derived from AdaBoost, but LogitBoost may well give better performance.

Implementations of gradient boosting for classification can provide information on the underlying probabilities. For example, this page on gradient boosting shows how sklearn code allows for a choice between deviance loss for logistic regression and exponential loss for AdaBoost, and documents functions to predict probabilities from the gradient-boosted model.

$\endgroup$
  • $\begingroup$ Thank you very much for your answer. If i understand it correctly in order to achieve the functionality of logistic regression in the context of boosting all I have to do is to use the gradient boosting algorithm with the logistic loss function and weak learners classification trees? But the classification trees outputs {-1,1} while the logistic regression which outputs probabilities. Moreover the classification trees tries to minimize the gini index instead of the logistic loss. I miss something fundamental here. Where to put the logistic loss? How to output probs from the model? $\endgroup$ – gnikol Feb 17 '18 at 9:30
  • $\begingroup$ I understand the concept where y is continuous because regression trees minimize the mse which is the same loss function with linear regression. Hence i repeatedly fit a regression tree to the residuals. But in the context of classification, the classification trees minimize the gini index or something similar. How is that connected with the logistic regression or the loss function of the logistic regression? $\endgroup$ – gnikol Feb 17 '18 at 11:15
  • $\begingroup$ @gnikol I've revised my answer in a way that I hope will make this clearer both to you and to other readers. The residuals that are fit in gradient boosting are pseudo-residuals calculated from the gradient of the loss function; the choice of loss function is what distinguishes AdaBoost from LogitBoost. Probabilities can be obtained from the strong learner in any case; I provided a link to one example of an implementation in the last paragraph. $\endgroup$ – EdM Feb 19 '18 at 17:27
5
$\begingroup$

In fact we have a very similar question here on regression case. And we had a very good answer by @Matthew Drury

Gradient Boosting for Linear Regression - why does it not work?

Linear model (such as logistic regression) is not good for boosting. The reason is if you add two linear models together, the result is another linear model. On the other hand, adding two decision stumps or trees, will have a more complicated and interesting model (not a tree any more.)

Details can be found in this post. In this link I derived why adding two linear models are not interesting. And I am showing the effect of boosting on decision stump iteration by iteration.

How does linear base learner works in boosting? And how does it works in the xgboost library?

Note that, decision tree / stump is not a "linear model" similar to logistic regression.

See this post for details

Is a decision stump a linear model?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.