4
$\begingroup$

I've seen two conflicting arguments:

  1. In a Stanford cs229 note, the author claims that boosting will increase variance (see section 2.5): http://cs229.stanford.edu/notes/cs229-notes-ensemble.pdf
  2. Prof. Yoav Freund said that boosting will reduce both variance and bias in his lecture.

From my understanding, since we are averaging over weak rules, the variance should decrease. But I'm not completely sure. Could someone provide some insights? Thank you!

$\endgroup$

1 Answer 1

7
$\begingroup$

1) and 2) use different models as reference.

1) Compared to the simple base learner (e.g. a shallow tree), boosting increases variance and reduces bias.

2) If you boost a simple base learner, the resulting model will have lower variance compared to some high variance reference like a too deep decision tree.

$\endgroup$
2
  • $\begingroup$ Let's say the reference is the base learner. 1. In that course note, equation 4 is used to explain why aggregation in bagging will reduce variance. This is also true in boosting just in a different way (weighted aggregation). 2. The upper bound of the generalization error of boosting is related to number of training samples and VC dimension of weak learners while has no dependence on the number of weak rules combined. This theorem is described in page 47 in : drive.google.com/open?id=1PSK7IPNGEhlI6x0GgTtDp9woNejexhYW Therefore, I'm not convinced that boosting will lead to overfitting. $\endgroup$
    – Kunyu Shi
    Commented Sep 24, 2019 at 21:11
  • $\begingroup$ @KunyuShi your first point is not true because in bagging we average models, fitted on bootstraped training sets from the original training set $(x_i,y_i)_{i=1}^N$. In gradient (not stochastic) boosting every base model is fitted on the same $(x_i)_{i=1}^N$ but on completely different targets $(r_i)_{i=1}^N$ which are defined as antigradients of loss function (so called pseudoresiduals). So we can't treat boosting as weighted average of models, fitted independently (while bagging is simple averaging of models, fitted independently of each other). $\endgroup$
    – Rodvi
    Commented Oct 22, 2020 at 14:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.