If we consider a full grown decision tree (i.e. an unpruned decision tree) it has high variance and low bias.

Bagging and Random Forests use these high variance models and aggregate them in order to reduce variance and thus enhance prediction accuracy. Both Bagging and Random Forests use Bootstrap sampling, and as described in "Elements of Statistical Learning", this increases bias in the single tree.

Furthermore, as the Random Forest method limits the allowed variables to split on in each node, the bias for a single random forest tree is increased even more.

Thus, the prediction accuracy is only increased, if the increase in bias of the single trees in Bagging and Random Forests is not "overshining" the variance reduction.

This leads me to the two following questions: 1) I know that with bootstrap sampling, we will (almost always) have some of the same observations in the bootstrap sample. But why does this lead to an increase in bias of the individual trees in Bagging / Random Forests? 2) Furthermore, why does the limit on available variables to split on in each split lead to higher bias in the individual trees in Random Forests?


3 Answers 3


I will accept the answer on 1) from Kunlun, but just to close this case, I will here give the conclusions on the two questions that I reached in my thesis (which were both accepted by my Supervisor):

1) More data produces better models, and since we only use part of the whole training data to train the model (bootstrap), higher bias occurs in each tree (Copy from the answer by Kunlun)

2) In the Random Forests algorithm, we limit the number of variables to split on in each split - i.e. we limit the number of variables to explain our data with. Again, higher bias occurs in each tree.

Conclusion: Both situations are a matter of limiting our ability to explain the population: First we limit the number of observations, then we limit the number of variables to split on in each split. Both limitations leads to higher bias in each tree, but often the variance reduction in the model overshines the bias increase in each tree, and thus Bagging and Random Forests tend to produce a better model than just a single decision tree.

  • $\begingroup$ Is the bias of the ensemble really increased? Doesn't the ensemble of high-variance low-bias decorrelated base learnes do also have lower bias in addition to lower variance? Because the different directions of biased individual trees balance each other out on average? $\endgroup$
    – Thomas
    Jun 27, 2022 at 10:17
  • $\begingroup$ Naive example: Assume one base learner is biased 'left', the other 'right' - thus, the bias of the ensemble if closer to the ground truth in the 'middle'? $\endgroup$
    – Thomas
    Jun 27, 2022 at 10:23
  • $\begingroup$ @C. Refsgaard Doesn't less data also increase variance? $\endgroup$
    – ado sar
    Sep 23, 2023 at 19:58

According to the authors of "Elements of Statistical Learning" (see proof below):

As in bagging, the bias of a random forest is the same as the bias of any of the individual sampled trees.

Taken from 2008. Elements of Statistical Learning 2nd Ed, Chapter 9.2.3. Hastie, Tibshirani, Friedman:

enter image description here enter image description here

Your answer however seems to make sense, and in the right plot of Fig 15.10 we can see that the green horizontal line, which is the squared bias of a single tree, is way below the bias of a random forest. Seems to be a contradiction which I have not sorted out yet.


The above is clarified right below the proof (same source): a tree within the random forest has the same bias as a random forest, where the single tree is restricted by bootstrap and no# of regressors randomly selected at each split (m). A fully grown, unpruned tree outside the random forest on the other hand (not bootstrapped and restricted by m) has lower bias. Hence random forests / bagging improve through variance reduction only, not bias reduction.

Quote: enter image description here

  • $\begingroup$ Can you explain why bootstrap sampling (the "reduced sample space" if I get it right) increases bias? I have seen the same statement in the extra trees paper, but I can't get why this is the case. $\endgroup$
    – ado sar
    Jan 21 at 18:37
  • $\begingroup$ Not bootstrap sampling, but sampling of regressors. See the 2nd point of the accepted answer by @C. Refsgaard above: for each tree in the random forest you only use a number of regressors, not all (randomly sampled). Hence your model will be less flexible, so it will have lower variance, higher bias. So, for the same reason why $y_t=\beta_1 X_t + \varepsilon_t$ has more bias than $y_t=\beta_1 X_t + \beta_2 X_t^2 + \varepsilon_t$ (assuming we don't have an unbiased estimator). $\endgroup$
    – PaulG
    Jan 25 at 10:42
  • $\begingroup$ Thanks for the answer! This makes sense. The paper of Extra Trees states the following: "The usage of the full original learning sample rather than bootstrap replicas is motivated in order to minimize bias". This is what I don't understand. $\endgroup$
    – ado sar
    Jan 29 at 10:40

Your questions are pretty straightforward. 1) More data produces better model, since you only use part of the whole training data to train your model (bootstrap), higher bias is reasonable. 2) More splits means deeper trees, or purer nodes. This typically leads to high variance and low bias. If you limit the split, lower variance and higher bias.

  • 6
    $\begingroup$ I don't quite buy the argument for 1), since each bootstrap sample is equally likely, and bias is about the behaviour of the average model. It seems like it must be more subtle than that. I also do not think 2) addresses the question asked. The poster does not mean "limit splits" as in "grow shallower trees". $\endgroup$ Jan 22, 2018 at 5:19
  • $\begingroup$ @Kunlun Can you elaborate on the first point? In general the higher the number of samples, the lower the variance, but I can't understand its effect on bias. $\endgroup$
    – ado sar
    Jan 21 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.