Overfit in aggregated models: boosting versus simple bagging

Question

Let's fix a bagging setup, where several models are build independently and than somehow aggregated. It is intuitive that increasing the number of weak learners ( N ) does not lead to overfit ( in the sense that the overfitting properties do not worsen adding an arbitrary number of trees ) . This is also discussed here for random forest:

https://datascience.stackexchange.com/questions/1028/do-random-forest-overfit

I was wondering if the situation is completely the opposite when we aggregate through boosting. In the AdaBoost algorithm, for example https://en.wikipedia.org/wiki/AdaBoost , the parameters of the next weak learner are chosen so that it improves the prediction of the previous step. Does it mean that, given enough weak learners, one would (over)fit perfectly the training data-set and, a fortiori, cause bad generaliazion ?

The question refers to the (theoretical) asymtptotic behavior for large N (the number of weak learners).

Answer 1

Yes, if you allow it to perfectly learn from the previous model. But, for example with gradient boosting we utilize HEAVY regularization such as a learning rate and subsampling procedures. For something like trees the depth of each tree is usually fairly shallow (at least it used to, now we are able to build larger trees due to other regularization advances) so it is still a pretty high bias model. So we try to add bias to each weak learner so that the next iteration doesn't add too-too much value to our ensemble. Also it is typical to stop iterating when your train/test set begins displaying signs of overfitting.

Even with this, we can use parameters which overfit badly, just like with a random forest or bagged tree.