11
$\begingroup$

I am able to understand the intution behind saying that "Bagging reduces the variance while retaining the bias".

What is the mathematically principle behind this intution? I checked with few experts and they are not able to explain the math well.

$\endgroup$

1 Answer 1

18
$\begingroup$

Quite surprising that the experts couldn't help you out, the chapter on random forests in "The Elements of Statistical Learning" explains it very well.

Basically, given n i.d.d. random variables each with variance sigma², the variance of the mean of this variables will be sigma²/n.

Since the random forest is build on bootstrap samples of the data, the outputs of the individual trees can be viewed as identically distributed random variables.

Thus, by averaging the outputs of B trees, the variance of the final prediction is given by p*sigma² + (1 - p)sigma² / B, where p is the pairwise correlation between trees. For large B the right term vanishes and the variance is reduced to p*sigma².

This works not only for decision trees but every model that's baggable. The reason why it works particularly well for decision trees is that they inherently have a low bias (no assumptions are made, such as e.g linear relation between features and response) but a very high variance.

Since only the variance can be reduced, decision trees are build to node purity in context of random forest and tree bagging. (Building to node purity maximizes the variance of the individual trees, i.e. they fit the data perfectly, while minimizing the bias.)

$\endgroup$
2
  • 2
    $\begingroup$ You can use mathjax to do math typeseting. math.meta.stackexchange.com/questions/5020/… $\endgroup$
    – Sycorax
    Sep 12, 2020 at 23:11
  • $\begingroup$ Bagging does not work if the cross-correlation (p in the answer) is close to 1. I think the answer is incomplete without an explanation of how random forests keep the cross-correlation low. It does so by sampling the features also, not only the data set. $\endgroup$ Oct 7, 2022 at 16:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.