2
$\begingroup$

Let's fix a bagging setup, where several models are build independently and than somehow aggregated. It is intuitive that increasing the number of weak learners ( N ) does not lead to overfit ( in the sense that the overfitting properties do not worsen adding an arbitrary number of trees ) . This is also discussed here for random forest:

https://datascience.stackexchange.com/questions/1028/do-random-forest-overfit

I was wondering if the situation is completely the opposite when we aggregate through boosting. In the AdaBoost algorithm, for example https://en.wikipedia.org/wiki/AdaBoost , the parameters of the next weak learner are chosen so that it improves the prediction of the previous step. Does it mean that, given enough weak learners, one would (over)fit perfectly the training data-set and, a fortiori, cause bad generaliazion ?

The question refers to the (theoretical) asymtptotic behavior for large N (the number of weak learners).

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ There's one misunderstanding (which is not central to your question): in bagging, we tend to employ highly flexible (strong) learners, and try to wash away their variance by averaging them all $\endgroup$
    – Firebug
    Sep 10, 2020 at 13:06
  • $\begingroup$ Thank you for this remark. I guess you want to say that bagging is in general used to average low bias, high variance models, whereas boosting is thought to combine models with high bias and low variance. Is this correct ? As you noticed, this is not central to the question but thanks for the remark ! $\endgroup$
    – Thomas
    Sep 10, 2020 at 13:50
  • $\begingroup$ Yes, that would be correct for most applications. Of course, deviations must exist, but I'm unaware of them. $\endgroup$
    – Firebug
    Sep 10, 2020 at 14:45

1 Answer 1

Reset to default
1
$\begingroup$

Yes, if you allow it to perfectly learn from the previous model. But, for example with gradient boosting we utilize HEAVY regularization such as a learning rate and subsampling procedures. For something like trees the depth of each tree is usually fairly shallow (at least it used to, now we are able to build larger trees due to other regularization advances) so it is still a pretty high bias model. So we try to add bias to each weak learner so that the next iteration doesn't add too-too much value to our ensemble. Also it is typical to stop iterating when your train/test set begins displaying signs of overfitting.

Even with this, we can use parameters which overfit badly, just like with a random forest or bagged tree.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.