Why doesn't boosting have a really high generalization error?

Question

AdaBoost creates a large amount of weak learners, and gives each one prediction score/weight that will be used to combine the learners during evaluation/generalization.

But isn't the point to create different learners that are stronger at different examples? If so, weighting them basd on their overall error is counterproductive. There needs to be a dynamic, state-space framework to ensembling. Is this correct?

UPDATE:
I watched the suggested video, but I still do not feel like my question has been answered. If you combine a bunch of weak learners using unchanging coefficients, the error of each individually should be pretty high because they each only target specific aspects of the example distribution.

Answer 1

In general, boosting has potentially a huge problem with overfitting. Nothing is easier than to end up with bad overfitting when using boosting.

However, by letting each model in the ensemble try to improve what the previous ones still get wrong, what boosting provides is a very flexible modeling approach that can fit quite complex data often pretty well. So, the main challenge is how to make this as flexible as needed, but to regularize the whole process enough to avoid overfitting. Depending on your boosting algorithm, the various regularization options provide a number of knobs one can turn to try to get that trade-off right (as assessed by e.g. some form of cross-validation).

What you say about more "clever" averaging of the different models in the ensemble may be something one could do, but that gets complicated fast (especially for boosting instead of parallel building of independent predictors like in random forest). Additionally, if there's lots of models in your ensemble that all contribute not that differently, simple averages/sums over lots and lots of them is often a pretty good idea. I guess the other thing to say is that many modern boosting algorithms like XGBoost and LightGBM have options for sampling a fraction of records and a fraction of predictors to use in building each model (or even for each split in a tree), these options are pretty helpful, because they address what you mention regarding addressing different training examples.

Answer 2

I think I figured it out. When reweighting the distribution, we multiply the example weights by a regularization parameter, which is the same coefficient that serves as the learner weight during model voting. It is computed as

$$ \lambda_j = \frac{1}{2}\ln\frac{\sum_{i: h_j(x_i) y_i = 1} w_j^i}{\sum_{i: h_j(x_i) y_i = -1} w_j^i} $$

in the case of binary classification. By multiplying the example weight vector by this, the algorithm prevents the next learner at iteration $j+1$ from overfitting to residuals, but still changes the distribution enough that many iterations, the individual learners will master the whole example distribution.