Elements of Statistical Learning: Variance Reduction via Bagging for Random Forests

Question

While Reading chapter 15 of ESL about Random Forests I had some confusion with this phrase:

The idea in random forests (Algorithm 15.1) is to improve the variance reduction of bagging by reducing the correlation between the trees, without increasing the variance too much.

If I interpret this sentence correctly it means that although the variance of the decision trees could be increased due to decorrelation the variance of the ensemble will decrease. Is this interpretation correct or am I missing something ?

Answer 1

Yes, the variance in the individual trees can increase since they use a bootstrap sample of the training dataset, and a subset of the columns. However, using a multiple number of trees will decrease the overall variance of the algorithm.

Answer 2

Say you have true labels

TL : 1 0 0 1 0

Say you have 3 trees predicting

T1 : 1 1 0 1 0
T2 : 1 0 1 1 1
T3 : 1 0 0 1 0

Bagging is computing the average from the upper tree threes.

AV : 1 0 0 1 0

This reduces the overall variance, thus single trees may have bigger variance comparing to the true labels.