80
$\begingroup$

I have a computer science background but am trying to teach myself data science by solving problems on the internet.

I have been working on this problem for the last couple of weeks (approx 900 rows and 10 features). I was initially using logistic regression but now I have switched to random forests. When I run my random forest model on my training data I get really high values for auc (> 99%). However when I run the same model on the test data the results are not so good (Accuracy of approx 77%). This leads me to believe that I am over fitting the training data.

What are the best practices regarding preventing over fitting in random forests?

I am using r and rstudio as my development environment. I am using the randomForest package and have accepted defaults for all parameters

Improve this question
$\endgroup$
7
  • $\begingroup$ You may wish to use cross validation methods, such as K fold cross validation. $\endgroup$
    – Coloane
    Commented Aug 15, 2014 at 5:38
  • 9
    $\begingroup$ Breiman claims that RF does not overfit. stat.berkeley.edu/~breiman/RandomForests/cc_home.htm On the assumption that he is correct, perhaps there is some sort of inconsistency between your training and test set? $\endgroup$
    – Sycorax ♦
    Commented Aug 15, 2014 at 15:55
  • 1
    $\begingroup$ RF does AVOIDS overfit to TEST DATA if you optimize the tuning parameter $\endgroup$
    – Brash Equilibrium
    Commented Aug 15, 2014 at 16:26
  • $\begingroup$ I find sampsize a complety overlooked tuning parameter. stackoverflow.com/questions/34997134/… $\endgroup$
    – Soren Havelund Welling
    Commented Jul 26, 2016 at 11:32
  • $\begingroup$ You can also use the max_depth parameter as the regularization parameter to reduce the overfiting $\endgroup$
    – Faizan Shakeel
    Commented Jun 14, 2020 at 14:08

4 Answers 4

Reset to default
54
$\begingroup$

To avoid over-fitting in random forest, the main thing you need to do is optimize a tuning parameter that governs the number of features that are randomly chosen to grow each tree from the bootstrapped data. Typically, you do this via $k$-fold cross-validation, where $k \in \{5, 10\}$, and choose the tuning parameter that minimizes test sample prediction error. In addition, growing a larger forest will improve predictive accuracy, although there are usually diminishing returns once you get up to several hundreds of trees.

Improve this answer
$\endgroup$
11
  • 2
    $\begingroup$ Thank you. Is there some tutorial which shows how to optimize these parameters? $\endgroup$
    – Abhi
    Commented Aug 20, 2014 at 2:14
  • 6
    $\begingroup$ I would argue against this answer: two of the appealing features of RFs are that it is difficult to overfit them and the default parameters are usually fairly good. This answer seem to imply that the RF are sensitive to the defaults which is rarely the case $\endgroup$
    – charles
    Commented Feb 3, 2015 at 1:49
  • 2
    $\begingroup$ Difficult but not impossible to overfit. My post may imply to you that RFs are sensitive to the defaults but you're putting words in my mouth. Read the comment again. It suggests, with reference to a fairly authoritative text, that mtry is an important tuning parameter. It goes on to explain that the number of trees you grow is not an important tuning parameter, but just to make sure you grow a fairly large one. $\endgroup$
    – Brash Equilibrium
    Commented Feb 3, 2015 at 17:12
  • 16
    $\begingroup$ Empirically, I have not found it difficult at all to overfit random forest, guided random forest, regularized random forest, or guided regularized random forest. They regularly perform very well in cross validation, but poorly when used with new data due to over fitting. I believe it has to do with the type of phenomena being modeled. It's not much of a problem when modeling a mechanical process, but with something like a behavioral model I get much more stable results with a well-specified regression. $\endgroup$
    – Hack-R
    Commented Apr 14, 2015 at 16:08
  • 2
    $\begingroup$ Overfitting is when you have your train << oob/cv score. This is often the case for the RFs I have used. People keep repeating that Brieman thinks there is no overfitting in RF. He means that you increase n_estimators or increase max_sample_size and you wont overfit as a result of increasing this parameter. But A simplest example of overfitting is when you have min_sample_leafs=1. This means the tree in the end could potentially have one leaf for each sample. This 100% will overfit. In this case you will have 100% training score and a not so 100% oob/cv score. $\endgroup$
    – Pandian Le
    Commented Jan 10, 2021 at 8:38
46
$\begingroup$

How are you getting that 99% AUC on your training data? Be aware that there's a difference between

predict(model)

and

predict(model, newdata=train)

when getting predictions for the training dataset. The first option gets the out-of-bag predictions from the random forest. This is generally what you want, when comparing predicted values to actuals on the training data.

The second treats your training data as if it was a new dataset, and runs the observations down each tree. This will result in an artificially close correlation between the predictions and the actuals, since the RF algorithm generally doesn't prune the individual trees, relying instead on the ensemble of trees to control overfitting. So don't do this if you want to get predictions on the training data.

Improve this answer
$\endgroup$
4
  • 6
    $\begingroup$ I was using predict(model, data=train). I have now switched to predict(model) and my auc has dropped to 87%. Is this a good thing or a bad thing? $\endgroup$
    – Abhi
    Commented Aug 18, 2014 at 19:29
  • 3
    $\begingroup$ Thank you! I found that this was the issue for me as well. I posted a follow-up question on what measure to use as 'training error' for RF models here: stats.stackexchange.com/questions/162353/… $\endgroup$
    – Berk U.
    Commented Jul 20, 2015 at 21:00
  • 2
    $\begingroup$ Great, thank you!! I was doing this mistake too! To @Abhi: it's a good thing, because the previous AUC was nonsensically high. This one is more realistic. Try cross-validation and measure AUC on that and you will probably see similar value. $\endgroup$
    – Tomas
    Commented Nov 4, 2019 at 11:11
  • 1
    $\begingroup$ Imagine that we train a RF with a training dataset and we saved the model, and we forgot to save the training dataset. So we lost the training dataset.... But one day we meet it again without knowing that it is the training dataset, what should we do to evaluate the this dataset? $\endgroup$
    – John Smith
    Commented Nov 1, 2020 at 18:16
11
$\begingroup$

For decision trees there are two ways of handling overfitting: (a) don't grow the trees to their entirety (b) prune

The same applies to a forest of trees - don't grow them too much and prune.

I don't use randomForest much, but to my knowledge, there are several parameters that you can use to tune your forests:

  • nodesize - minimum size of terminal nodes
  • maxnodes - maximum number of terminal nodes
  • mtry - number of variables used to build each tree (thanks @user777)
Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ And mtry, the number of variables the algorithm draws to build each tree, by default the square root of the number of features total. $\endgroup$
    – Sycorax ♦
    Commented Aug 15, 2014 at 15:56
  • $\begingroup$ I would leave maxnodes and lower sampsize instead. Both decresing maxnodes and sampsize give trees with less depth and a more robust forest, sampsize however lower tree correlation also, and the forest will likely converge to lower cross-validated prediction error, see stackoverflow.com/questions/34997134/… $\endgroup$
    – Soren Havelund Welling
    Commented Jul 26, 2016 at 11:29
3
$\begingroup$

Try to tune max_depth parameter in ranges of [5, 15] but not more than this because if you take large depth there is a high chance of overfitting.

Improve this answer
$\endgroup$
1
  • 11
    $\begingroup$ There's no particular reason to believe that every problem will have the optimal depth in the range [5,15]. Some problems are extremely complex, so a deep tree is required to achieve the desired result. $\endgroup$
    – Sycorax ♦
    Commented Jun 14, 2020 at 15:13

Not the answer you're looking for? Browse other questions tagged or ask your own question.