41
$\begingroup$

I have a computer science background but am trying to teach myself data science by solving problems on the internet.

I have been working on this problem for the last couple of weeks (approx 900 rows and 10 features). I was initially using logistic regression but now I have switched to random forests. When I run my random forest model on my training data I get really high values for auc (> 99%). However when I run the same model on the test data the results are not so good (Accuracy of approx 77%). This leads me to believe that I am over fitting the training data.

What are the best practices regarding preventing over fitting in random forests?

I am using r and rstudio as my development environment. I am using the randomForest package and have accepted defaults for all parameters

$\endgroup$
31
$\begingroup$

To avoid over-fitting in random forest, the main thing you need to do is optimize a tuning parameter that governs the number of features that are randomly chosen to grow each tree from the bootstrapped data. Typically, you do this via $k$-fold cross-validation, where $k \in \{5, 10\}$, and choose the tuning parameter that minimizes test sample prediction error. In addition, growing a larger forest will improve predictive accuracy, although there are usually diminishing returns once you get up to several hundreds of trees.

$\endgroup$
  • 1
    $\begingroup$ Thank you. Is there some tutorial which shows how to optimize these parameters? $\endgroup$ – Abhi Aug 20 '14 at 2:14
  • $\begingroup$ You will need to register for Stanford online courses, which is pretty simple, but here is a video tutorial for doing it in R: class.stanford.edu/courses/HumanitiesScience/StatLearning/… $\endgroup$ – Brash Equilibrium Aug 20 '14 at 18:03
  • $\begingroup$ If I am understanding this correctly we use the cross validation to ascertain the number of features that go into the random forest model as opposed to the number of features that the model will try at every step. Correct ? $\endgroup$ – Abhi Aug 27 '14 at 2:42
  • 4
    $\begingroup$ I would argue against this answer: two of the appealing features of RFs are that it is difficult to overfit them and the default parameters are usually fairly good. This answer seem to imply that the RF are sensitive to the defaults which is rarely the case $\endgroup$ – charles Feb 3 '15 at 1:49
  • 8
    $\begingroup$ Empirically, I have not found it difficult at all to overfit random forest, guided random forest, regularized random forest, or guided regularized random forest. They regularly perform very well in cross validation, but poorly when used with new data due to over fitting. I believe it has to do with the type of phenomena being modeled. It's not much of a problem when modeling a mechanical process, but with something like a behavioral model I get much more stable results with a well-specified regression. $\endgroup$ – Hack-R Apr 14 '15 at 16:08
33
$\begingroup$

How are you getting that 99% AUC on your training data? Be aware that there's a difference between

predict(model)

and

predict(model, newdata=train)

when getting predictions for the training dataset. The first option gets the out-of-bag predictions from the random forest. This is generally what you want, when comparing predicted values to actuals on the training data.

The second treats your training data as if it was a new dataset, and runs the observations down each tree. This will result in an artificially close correlation between the predictions and the actuals, since the RF algorithm generally doesn't prune the individual trees, relying instead on the ensemble of trees to control overfitting. So don't do this if you want to get predictions on the training data.

$\endgroup$
  • 3
    $\begingroup$ I was using predict(model, data=train). I have now switched to predict(model) and my auc has dropped to 87%. Is this a good thing or a bad thing? $\endgroup$ – Abhi Aug 18 '14 at 19:29
  • 2
    $\begingroup$ Thank you! I found that this was the issue for me as well. I posted a follow-up question on what measure to use as 'training error' for RF models here: stats.stackexchange.com/questions/162353/… $\endgroup$ – Berk U. Jul 20 '15 at 21:00
  • $\begingroup$ Great, thank you!! I was doing this mistake too! To @Abhi: it's a good thing, because the previous AUC was nonsensically high. This one is more realistic. Try cross-validation and measure AUC on that and you will probably see similar value. $\endgroup$ – Curious Nov 4 at 11:11
7
$\begingroup$

For decision trees there are two ways of handling overfitting: (a) don't grow the trees to their entirety (b) prune

The same applies to a forest of trees - don't grow them too much and prune.

I don't use randomForest much, but to my knowledge, there are several parameters that you can use to tune your forests:

  • nodesize - minimum size of terminal nodes
  • maxnodes - maximum number of terminal nodes
  • mtry - number of variables used to build each tree (thanks @user777)
$\endgroup$
  • 1
    $\begingroup$ And mtry, the number of variables the algorithm draws to build each tree, by default the square root of the number of features total. $\endgroup$ – Sycorax says Reinstate Monica Aug 15 '14 at 15:56
  • $\begingroup$ I would leave maxnodes and lower sampsize instead. Both decresing maxnodes and sampsize give trees with less depth and a more robust forest, sampsize however lower tree correlation also, and the forest will likely converge to lower cross-validated prediction error, see stackoverflow.com/questions/34997134/… $\endgroup$ – Soren Havelund Welling Jul 26 '16 at 11:29
2
$\begingroup$

You may wish to use cross validation methods, such as K fold cross validation.

$\endgroup$
  • $\begingroup$ you need to normalize/scale the features? $\endgroup$ – charles Aug 15 '14 at 14:07
  • 4
    $\begingroup$ @charles my apologies. It is indeed not necessary to scale the features in using random forests. See: stackoverflow.com/questions/8961586/… $\endgroup$ – Fre Aug 15 '14 at 14:36
  • $\begingroup$ I do think cross-validation would be useful. This is a relatively small dataset with split sample validation potentially producing unstable estimates of error (though admittedly I get the sense this isn't the issue here) $\endgroup$ – charles Aug 15 '14 at 18:10
1
$\begingroup$

you can tune your parameters using gridsearch

from sklearn.ensemble import RandomForestClassifier

from sklearn.grid_search import GridSearchCV

random_classifier = RandomForestClassifier()

parameters = { 'max_features':np.arange(5,10),'n_estimators':[500],'min_samples_leaf': [10,50,100,200,500]}

random_grid = GridSearchCV(random_classifier, parameters, cv = 5)
$\endgroup$
  • $\begingroup$ An attempted editor suggests that the module GridSearchCV is in is called model_selection, & thus that the second line of code should be from sklearn.model_selection import GridSearchCV. $\endgroup$ – gung - Reinstate Monica Jul 25 at 15:07
0
$\begingroup$

Try to tune max_depth parameter in ranges of [5, 15] but not more than this because if you take large depth there is a high chance of overfitting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.