# Why doesn't boosting assign higher weight to the "good" (low residual) models?

Lets say we start out with a dumb weak learner. Since its the 0th model and hasnt learned anything yet, we have a high residual, lets say of 10,000.

We produce the next weak learner, which actually turns out to be pretty good. The residual is only 1000. And so we update the residual per the formula above to 9000 (10000-1000 = 9000).

However, when we produce the next weak learner, it turns out to be bad. The residual is 8000. And so we update the residual to 1000 (9000-8000 = 1000).

So I think you can see where my question is - how does this boosting system assign higher weight to the good weak learner of only 1000 residual when its only barely moving the "Total Residual" (outcome variable)? Even if lambda parameter scales it so that the Total Residual wont go negative, we still get a higher weighting for the bad models and a lower weighting for the good models?

• I don't think your calculations make sense. When a weak learner $b$ is good, it means that $\hat{f}^b(x_i)\approx r_i$, so it has a large impact on the next residual. Jan 23, 2023 at 14:43

3. If you did fit the $$\lambda$$ for a new tree to the data, or re-weighted all existing trees, you should note that fixing $$\lambda$$ to one fixed value is a form of regularization. The more flexible approach you suggest has the potentially to fit the training data more closely (and also to overfit), so would require additional regularization. For example, consider that what you describe would be a much more complex form of model stacking than traditional model stacking. You already tend to do model stacking on out-of-fold predictions from cross-validation to mitigate overfitting. When doing it on the training data overfitting could potentially get way worse with what you describe and doing it on the training data would make it hard to avoid overfitting. I guess you could introduce extra hyperparameters that penalize this process and push weights towards $$\lambda$$ (not quite sure what's the obvious way to do that) based on what choices of hyperparameters lead to good out-of-fold performance in cross-validation.
4. The alternative to what you describe is the "vanilla"-algorithm, which is a lot simpler. What I think it relies on in practice is that if you set $$\lambda$$ low enough, you thereby allow for a large $$B$$, which has the effect that any "bad luck" in constructing one of the trees (e.g. when some of the random elements of the algorithm such as sub-sampling predictors or observations - as one tends to use in e.g. XGBoost or LightGBM - turned out to make it hard to learn much with the tree) doesn't have too much of an effect and gets balanced out by the many other trees.