20
$\begingroup$

I've read some literature that random forests can't overfit. While this sounds great, it seems too good to be true. Is it possible for rf's to overfit?

$\endgroup$
4
  • 7
    $\begingroup$ If it can fit, it can overfit. In terms of RF, think about what happens if your forest doesn't contain enough trees (say your forest is a single tree to make the effect obvious). There are more issues than this one, but this is the most obvious. $\endgroup$ Commented Aug 25, 2014 at 16:54
  • $\begingroup$ I've just responded to another thread on RF that it could easily overfit if the number of predictors is large. $\endgroup$
    – horaceT
    Commented Jun 11, 2016 at 4:07
  • $\begingroup$ "Can't" is a very dangerous word. It takes a lot of abuse or a bit of unluck to make it happen but it absolutely can happen. The RF is somewhat more abuse-resistant than other methods, but no method is perfect. Too short, too tall, to fat, too skinny, ... it feels like a zefrankism. $\endgroup$ Commented Jul 16, 2020 at 15:09
  • $\begingroup$ This question would benefit from some context. Where did you find the claim that random forest cannot overfit? Can you edit to include a quotation that makes the claim & its citation? $\endgroup$
    – Sycorax
    Commented Feb 22, 2022 at 3:48

3 Answers 3

12
$\begingroup$

I will try to give a more thorough answer building on Donbeo's answer and Itachi's comment.

Can Random Forests overfit?
In short, yes, they can.

Why is there a common misconception that Random Forests cannot overfit?
The reason is that, from the outside, the training of Random Forests looks similar to the ones of other iterative methods such as Gradient Boosted Machines, or Neural Networks.
Most of these other iterative methods, however, reduce the model's bias over the iterations, as they make the model more complex (GBM) or more suited to the training data (NN). It is therefore common knowledge that these methods suffer from overtraining, and will overfit the training data if trained for too long since bias reduction involves an increase in variance.
Random Forests, on the other hand, simply average trees over the iterations, reducing the model's variance instead, while leaving the bias unchanged. This means that they do not suffer from overtraining, and indeed adding more trees (therefore training longer) cannot be source of overfitting. This is where they get their non-overfitting reputation from!

Then how can they overfit?
Random Forests are usually built of high-variance, low-bias fully grown decision trees, and their strength comes from the variance reduction that comes from the averaging of these trees. However, if the predictions of the trees are too close to each other then the variance reduction effect is limited, and they might end up overfitting.
This can happen for example if the dataset is relatively simple, and therefore the fully grown trees perfectly learn its patterns and predict very similarly. Also having a high value for mtry, the number of features considered at every split, causes the trees to be more correlated, and therefore limits the variance reduction and might cause some overfitting
(it is important to know that a high value of mtry can still be very useful in many situations, as it makes the model more robust to noisy features)

Can I fix this overfitting?
Like always, more data helps.
Limiting the depth of the trees has also been shown to help in this situation, and reducing the number of selected features to make the trees as uncorrelated as possible.

For reference, I really suggest reading the relative chapter of Elements of Statistical Learning, which I think gives a very detailed analysis, and dives deeper into the math behind it.

$\endgroup$
10
$\begingroup$

Random forest can overfit. I am sure of this. What is usually meant is that the model would not overfit if you use more trees.

Try for example to estimate the model $y = log(x) + \epsilon$ with a random forest. You will get an almost zero training error but a bad prediction error

$\endgroup$
19
  • $\begingroup$ Random Forest principally reduces variance, how can it overfit? @Donbeo could it be perhaps because, decision tree models do not perform well on extrapolation. Let's say, for anomalous predictor variable, DT could give bad prediction. $\endgroup$
    – Itachi
    Commented Jun 22, 2017 at 18:34
  • $\begingroup$ One clear indication of overfitting is that the residual variance is reduced too much. What, then, are you trying to imply with your first remark? $\endgroup$
    – whuber
    Commented Jun 22, 2017 at 18:47
  • 1
    $\begingroup$ In bias-variance trade off, when we try to reduce bias, we compensate for variance. Such that, if x = 80 gives y = 100, but x = 81 gives y = -100. This would be overfitting. Isn't Ovefitting similar to for having high variance. @whuber i assumed ovefitting is only because of high variance. I do not understand how reducing residual variance results in overfitting. Can you please share some paper for me to read on. $\endgroup$
    – Itachi
    Commented Jun 23, 2017 at 11:23
  • 3
    $\begingroup$ This doesn't require any paper! You can try it yourself. Take a small simple bivariate dataset, such as $x_i=1,2,\ldots,10$ and any collection of corresponding $y_i$ you care to produce. Using least squares (because this aims to reduce the variance of the residuals), fit the series of models $y=\beta_0+\beta_1 x+\beta_2 x^2 + \cdots + \beta_k x^k$ for $k=0, 1, \ldots, 9$. Each step will reduce the variance until at the last step the variance is zero. At some point, almost anyone will agree, the models have begun to overfit the data. $\endgroup$
    – whuber
    Commented Jun 23, 2017 at 13:11
  • 2
    $\begingroup$ @Davide Your remark shows I should have explicitly stated I was offering my example not as a statement about random forests, but about the underlying concepts of variance reduction and overfitting. But your first comment is opaque because it is irrelevant (and, as I read it, is incorrect): the residual variance matters in this sequence of OLS models, not the prediction variance. Indeed--returning to the general question of fitting models--if reducing variance of the predictions were the objective, then any model that always predicts zero would be optimal! $\endgroup$
    – whuber
    Commented Jan 30, 2020 at 13:26
3
$\begingroup$

Hastie et al. address this question very briefly in Elements of Statistical Learning (page 596).

Another claim is that random forests “cannot overfit” the data. It is certainly true that increasing $\mathcal{B}$ [the number of trees in the ensemble] does not cause the random forest sequence to overfit... However, this limit can overfit the data; the average of fully grown trees can result in too rich a model, and incur unnecessary variance. Segal (2004) demonstrates small gains in performance by controlling the depths of the individual trees grown in random forests. Our experience is that using full-grown trees seldom costs much, and results in one less tuning parameter.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.