Extremely confused about the following:

enter image description here

Lets say we start out with a dumb weak learner. Since its the 0th model and hasnt learned anything yet, we have a high residual, lets say of 10,000.

We produce the next weak learner, which actually turns out to be pretty good. The residual is only 1000. And so we update the residual per the formula above to 9000 (10000-1000 = 9000).

However, when we produce the next weak learner, it turns out to be bad. The residual is 8000. And so we update the residual to 1000 (9000-8000 = 1000).

So I think you can see where my question is - how does this boosting system assign higher weight to the good weak learner of only 1000 residual when its only barely moving the "Total Residual" (outcome variable)? Even if lambda parameter scales it so that the Total Residual wont go negative, we still get a higher weighting for the bad models and a lower weighting for the good models?

  • 2
    $\begingroup$ I don't think your calculations make sense. When a weak learner $b$ is good, it means that $\hat{f}^b(x_i)\approx r_i$, so it has a large impact on the next residual. $\endgroup$ Jan 23, 2023 at 14:43

1 Answer 1


There's a number of perspectives:

  1. Every step in the boosting algorithm will (with enough data and predictors) improve the fit to the training data. However, initially while you are very underfit (because there's initially no model, then initially a too simple one), it's easier to make gains in terms of fitting the training, while eventually gains in terms of fitting even to the training data will become smaller (note: we also care about what happens on the validation data, where while you still (over-)fit to the training data more closely, you may already be performing worse). So, the fact that we keep improving less from step to step is not necessarily "bad" or unexpected.
  2. To some degree the boosting algorithm above is meant to be a simple and easy to penalize algorithm that can be executed fast, so complicated re-weighting steps may be non-ideal. When you re-weight, because all the algorithms affect the ones that come after them (because the ones after work on the residuals of the preceding ones), it's much more time-consuming/complex to re-weight the algorithms than e.g. with model stacking and/or random forest. Additionally, once you re-weight what comes before, the splits in the newly constructed tree would no longer be optimal. I guess you could argue that you would only determine a weight for each new tree that is added, which would avoid a lot of that complexity.
  3. If you did fit the $\lambda$ for a new tree to the data, or re-weighted all existing trees, you should note that fixing $\lambda$ to one fixed value is a form of regularization. The more flexible approach you suggest has the potentially to fit the training data more closely (and also to overfit), so would require additional regularization. For example, consider that what you describe would be a much more complex form of model stacking than traditional model stacking. You already tend to do model stacking on out-of-fold predictions from cross-validation to mitigate overfitting. When doing it on the training data overfitting could potentially get way worse with what you describe and doing it on the training data would make it hard to avoid overfitting. I guess you could introduce extra hyperparameters that penalize this process and push weights towards $\lambda$ (not quite sure what's the obvious way to do that) based on what choices of hyperparameters lead to good out-of-fold performance in cross-validation.
  4. The alternative to what you describe is the "vanilla"-algorithm, which is a lot simpler. What I think it relies on in practice is that if you set $\lambda$ low enough, you thereby allow for a large $B$, which has the effect that any "bad luck" in constructing one of the trees (e.g. when some of the random elements of the algorithm such as sub-sampling predictors or observations - as one tends to use in e.g. XGBoost or LightGBM - turned out to make it hard to learn much with the tree) doesn't have too much of an effect and gets balanced out by the many other trees.

So, my bottom line would be that I'm not sure how well the alternative(s) along the lines you suggest would work and I think it would need regularization for how weights of trees are determined to be able to control overfitting. If you could implement such a thing (with a robust and efficient implementation), it may well be worth trying it and comparing it to traditional boosted trees to compare the performance. However, it's entirely possible that it would not result in any performance gains once you evaluate the algorithms appropriately (i.e. suitable leak-free cross-validation or large leak-free hold-out test set).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.