I've read some literature that random forests can't overfit. While this sounds great, it seems too good to be true. Is it possible for rf's to overfit?

  • 5
    $\begingroup$ If it can fit, it can overfit. In terms of RF, think about what happens if your forest doesn't contain enough trees (say your forest is a single tree to make the effect obvious). There are more issues than this one, but this is the most obvious. $\endgroup$ – Marc Claesen Aug 25 '14 at 16:54
  • $\begingroup$ I've just responded to another thread on RF that it could easily overfit if the number of predictors is large. $\endgroup$ – horaceT Jun 11 '16 at 4:07

Random forest can overfit. I am sure of this. What is usually meant is that the model would not overfit if you use more trees.

Try for example to estimate the model $y = log(x) + \epsilon$ with a random forest. You will get an almost zero training error but a bad prediction error

  • $\begingroup$ Random Forest principally reduces variance, how can it overfit? @Donbeo could it be perhaps because, decision tree models do not perform well on extrapolation. Let's say, for anomalous predictor variable, DT could give bad prediction. $\endgroup$ – Itachi Jun 22 '17 at 18:34
  • $\begingroup$ One clear indication of overfitting is that the residual variance is reduced too much. What, then, are you trying to imply with your first remark? $\endgroup$ – whuber Jun 22 '17 at 18:47
  • $\begingroup$ In bias-variance trade off, when we try to reduce bias, we compensate for variance. Such that, if x = 80 gives y = 100, but x = 81 gives y = -100. This would be overfitting. Isn't Ovefitting similar to for having high variance. @whuber i assumed ovefitting is only because of high variance. I do not understand how reducing residual variance results in overfitting. Can you please share some paper for me to read on. $\endgroup$ – Itachi Jun 23 '17 at 11:23
  • 2
    $\begingroup$ This doesn't require any paper! You can try it yourself. Take a small simple bivariate dataset, such as $x_i=1,2,\ldots,10$ and any collection of corresponding $y_i$ you care to produce. Using least squares (because this aims to reduce the variance of the residuals), fit the series of models $y=\beta_0+\beta_1 x+\beta_2 x^2 + \cdots + \beta_k x^k$ for $k=0, 1, \ldots, 9$. Each step will reduce the variance until at the last step the variance is zero. At some point, almost anyone will agree, the models have begun to overfit the data. $\endgroup$ – whuber Jun 23 '17 at 13:11
  • $\begingroup$ @whuber I think you're missing the point on what "variance reduction" is. Random Forest (and bagging in general) do not reduce the variance of the residuals, but the variance of your predictions. So in your example, each step you talk about INCREASES variance :) $\endgroup$ – Davide ND Jan 30 at 10:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.