Why does a bagged tree / random forest tree have higher bias than a single decision tree?

Question

If we consider a full grown decision tree (i.e. an unpruned decision tree) it has high variance and low bias.

Bagging and Random Forests use these high variance models and aggregate them in order to reduce variance and thus enhance prediction accuracy. Both Bagging and Random Forests use Bootstrap sampling, and as described in "Elements of Statistical Learning", this increases bias in the single tree.

Furthermore, as the Random Forest method limits the allowed variables to split on in each node, the bias for a single random forest tree is increased even more.

Thus, the prediction accuracy is only increased, if the increase in bias of the single trees in Bagging and Random Forests is not "overshining" the variance reduction.

This leads me to the two following questions: 1) I know that with bootstrap sampling, we will (almost always) have some of the same observations in the bootstrap sample. But why does this lead to an increase in bias of the individual trees in Bagging / Random Forests? 2) Furthermore, why does the limit on available variables to split on in each split lead to higher bias in the individual trees in Random Forests?

Answer 1

I will accept the answer on 1) from Kunlun, but just to close this case, I will here give the conclusions on the two questions that I reached in my thesis (which were both accepted by my Supervisor):

1) More data produces better models, and since we only use part of the whole training data to train the model (bootstrap), higher bias occurs in each tree (Copy from the answer by Kunlun)

2) In the Random Forests algorithm, we limit the number of variables to split on in each split - i.e. we limit the number of variables to explain our data with. Again, higher bias occurs in each tree.

Conclusion: Both situations are a matter of limiting our ability to explain the population: First we limit the number of observations, then we limit the number of variables to split on in each split. Both limitations leads to higher bias in each tree, but often the variance reduction in the model overshines the bias increase in each tree, and thus Bagging and Random Forests tend to produce a better model than just a single decision tree.

Answer 2

According to the authors of "Elements of Statistical Learning" (see proof below):

As in bagging, the bias of a random forest is the same as the bias of any of the individual sampled trees.

Taken from 2008. Elements of Statistical Learning 2nd Ed, Chapter 9.2.3. Hastie, Tibshirani, Friedman:

Your answer however seems to make sense, and in the right plot of Fig 15.10 we can see that the green horizontal line, which is the squared bias of a single tree, is way below the bias of a random forest. Seems to be a contradiction which I have not sorted out yet.

EDIT:

The above is clarified right below the proof (same source): a tree within the random forest has the same bias as a random forest, where the single tree is restricted by bootstrap and no# of regressors randomly selected at each split (m). A fully grown, unpruned tree outside the random forest on the other hand (not bootstrapped and restricted by m) has lower bias. Hence random forests / bagging improve through variance reduction only, not bias reduction.

Quote:

Answer 3

Your questions are pretty straightforward. 1) More data produces better model, since you only use part of the whole training data to train your model (bootstrap), higher bias is reasonable. 2) More splits means deeper trees, or purer nodes. This typically leads to high variance and low bias. If you limit the split, lower variance and higher bias.