I am reading about the idea of bagging (boostrap aggregating). I have no trouble understanding how the variance of prediction can be reduced. The simple picture is if you have prediction Z_1 through Z_n for each boostrapped sample, the standard error of the mean will be the reduced by sqrt(n).

However, if this picture is true, Mean of boostrapped sample mean should be unbiased estimate of mean of each Z. Then how can the accuracy be improved as well?

  • $\begingroup$ If by "accuracy" you mean MSE, then since $\text{MSE}=\text{variance} + \text{bias}^2$, if the variance goes down (and the estimator is unbiased) the MSE will also go down. $\endgroup$ – Hong Ooi Sep 16 '14 at 1:34
  • $\begingroup$ Thank you @HongOoi for your comment. I agree MSE goes down. But bias won't get improved, right? $\endgroup$ – K.Chen Sep 16 '14 at 5:36

Because bagging equalizes influence. This essentially means that the influence of so-called leverage points (points which have a large impact on the overall model) decreases compared to non-bagged models. This is good if the leverage points are bad for the model's performance, which is not always the case. For an example of a leverage point, consider an outlier in least-squares regression.

The variance-reduction argument which most people know about does not explain everything as nicely (e.g. in some cases bagging does increase variance).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.