0
$\begingroup$

AdaBoost creates a large amount of weak learners, and gives each one prediction score/weight that will be used to combine the learners during evaluation/generalization.

But isn't the point to create different learners that are stronger at different examples? If so, weighting them basd on their overall error is counterproductive. There needs to be a dynamic, state-space framework to ensembling. Is this correct?

UPDATE:
I watched the suggested video, but I still do not feel like my question has been answered. If you combine a bunch of weak learners using unchanging coefficients, the error of each individually should be pretty high because they each only target specific aspects of the example distribution.

$\endgroup$
5
  • 1
    $\begingroup$ See this talk by Trevor Hastie. $\endgroup$
    – Will
    Commented May 7, 2017 at 11:30
  • $\begingroup$ Thank you for the suggestion. I watched the video, but I still do not feel like my question has been answered. If you combine a bunch of weak learners using unchanging coefficients, the error of each individually should be pretty high because they each only target specific aspects of the example distribution. $\endgroup$ Commented May 7, 2017 at 13:29
  • 1
    $\begingroup$ Do you know gradient boosting? I feel like that algorithm is much easier to understand than AdaBoost. It's unfortunate that AdaBoost is still the first boosting algorithm that many learners encounter, it's both worse performing and more difficult to understand than gradient boosting. $\endgroup$ Commented May 7, 2017 at 19:48
  • $\begingroup$ I am familiar with it. It models the residuals as opposed to modeling a re-weighted distribution like AdaBoost does. $\endgroup$ Commented May 7, 2017 at 20:16
  • $\begingroup$ Are you sure that it doesn't? The point of boosting is to push an underfitting model toward lower variance. The amount you push is in the number of iterations, which is a hyperparameter. I wouldn't be suprised if Adaboost overfits in the limit of iterations (but I don't know either way, maybe it converges to a Bayes optimal classifier). $\endgroup$ Commented Feb 16, 2021 at 13:38

2 Answers 2

2
$\begingroup$

In general, boosting has potentially a huge problem with overfitting. Nothing is easier than to end up with bad overfitting when using boosting.

However, by letting each model in the ensemble try to improve what the previous ones still get wrong, what boosting provides is a very flexible modeling approach that can fit quite complex data often pretty well. So, the main challenge is how to make this as flexible as needed, but to regularize the whole process enough to avoid overfitting. Depending on your boosting algorithm, the various regularization options provide a number of knobs one can turn to try to get that trade-off right (as assessed by e.g. some form of cross-validation).

What you say about more "clever" averaging of the different models in the ensemble may be something one could do, but that gets complicated fast (especially for boosting instead of parallel building of independent predictors like in random forest). Additionally, if there's lots of models in your ensemble that all contribute not that differently, simple averages/sums over lots and lots of them is often a pretty good idea. I guess the other thing to say is that many modern boosting algorithms like XGBoost and LightGBM have options for sampling a fraction of records and a fraction of predictors to use in building each model (or even for each split in a tree), these options are pretty helpful, because they address what you mention regarding addressing different training examples.

$\endgroup$
0
$\begingroup$

I think I figured it out. When reweighting the distribution, we multiply the example weights by a regularization parameter, which is the same coefficient that serves as the learner weight during model voting. It is computed as

$$ \lambda_j = \frac{1}{2}\ln\frac{\sum_{i: h_j(x_i) y_i = 1} w_j^i}{\sum_{i: h_j(x_i) y_i = -1} w_j^i} $$

in the case of binary classification. By multiplying the example weight vector by this, the algorithm prevents the next learner at iteration $j+1$ from overfitting to residuals, but still changes the distribution enough that many iterations, the individual learners will master the whole example distribution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.