8
$\begingroup$

I know that AdaBoost can be used for classification, but how about regression?

With classification, it is clear how to assign the "amount of say" (or weight) to the predictions of each model (stump) in the final ensemble of models. Each of the stumps will make different errors. Would it be reasonable to weight each of the model's prediction according to the mean (mean squared) error?

$\endgroup$

2 Answers 2

6
$\begingroup$

AdaBoost is a meta-algorithm, which means it can be used together with other algorithms for perfomance improvement. Indeed, the concept of boosting is a type of linear regression.

Now, specifically answering your question, AdaBoost is actually intented for classification and regression problems. Scikit-Learn, for example, has an implemetation of an Adaboost regressor:

An AdaBoost regressor is a meta-estimator that begins by fitting a regressor on the original dataset and then fits additional copies of the regressor on the same dataset but where the weights of instances are adjusted according to the error of the current prediction. As such, subsequent regressors focus more on difficult cases.

$\endgroup$
1
  • $\begingroup$ While this is a good answer, it leaves many points open: What is the exact formulation of the algorithm for the regression case? How are the weights of the instances and of the different models calculated, etc. $\endgroup$
    – PhoemueX
    Jun 26, 2023 at 9:10
1
$\begingroup$

Maybe I'm a bit late. As you can see here AdaBoost.RT: A boosting algorithm for regression problems, during last years there were tried different solutions. One of the most common path is to try to adapt the regression problem to a classification problem.

The main idea of AdaBoost.RT is to introduce a constant as a relative error threshold value, used to demarcate predictions as correct or incorrect. Then error rate is calculated by counting number of correct and incorrect predictions. Consequently, weight updating parameter is computed and the distribution of the training set is updated using the weight updating parameter.

In other words, as you can see in page 3, when you have a classification problem with two classes you can identify which observation was predicted in the right or wrong class. With regression you do the same, not in terms of the same exact value but considering a little range of error. If the output is inside this range of tolerance the observation is predicted right, otherwise it isn't and so the error is computed based on the number predicted wrong or right, as in classification.

This isn't the only solution and could not be the best, but anyway is one can be considered.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.