I am reading hands on machine learning .

Bootstrapping introduces a bit more diversity in the subsets that each predictor is trained on, so bagging ends up with a slightly higher bias than pasting, but this also means that predictors end up being less correlated so the ensemble’s variance is reduced.

I think in this situation model do not have many unique instances and it cause for raising bias because it can not predict best parammetr but I dont have any Idea why variance decreases.
I know about bias-variance trade off but I specially want to know what is happening in this situation.


1 Answer 1


Let $X_1,...,X_n$ be identically distributed random variables with variance $\sigma^2$.

If they are independent:

$$Var(\frac{1}{n}\sum\limits_{i=1}^nX_i) = \frac{1}{n}\sigma^2$$

If they are correlated with positive pairwise correlation $\rho > 0$, we have:

$$\frac{Cov(X_i,X_j)}{\sigma^2}=\rho \iff Cov(X_i,X_j) = \rho\sigma^2$$

$$\begin{aligned} Var(\frac{1}{n}\sum\limits_{i=1}^nX_i) &= \frac{1}{n^2} \times \left[\sum\limits_{i=1}^nVar(X_i) + 2\sum\limits_{1\leq i<j\leq n}^nCov(X_i,X_j)\right] \\ &= \frac{\sigma^2}{n}+\frac{n-1}{n}\rho\sigma^2 \end{aligned}$$

As you can see, as $\rho$ decreases (less correlation) variance decreases, and vice versa. The variance reduction effect is largest when individual trees are uncorrelated ($\rho=0$).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.