0
$\begingroup$

I am running a random forest classifier in R and during 10-fold cross-validation, I discovered that the model is overfitting. I am using a grid search to find the best hyperparameters and used the best model to predict the same training set again. The random Forest model classified every data point of the train set perfectly. I ran the same model on the test set and got an accuracy of 0.6976. Does this mean that the model is overfitting? I thought random forest models don't overfit? My data is slightly imbalanced with 110 positive samples and 95 negative samples.

$\endgroup$

2 Answers 2

2
$\begingroup$

How does the 70 % accuracy of your CV compare to the rF's oob estimate?


The behaviour you observe is to be expected for random forests, see also my old answer: https://stats.stackexchange.com/a/208018/4598

The submodels of a random forest are unpruned decision trees. Training data for such an unpruned tree will basically be recognized as known and assigned to the correct class. Since each case in the training set for the random forest is training case for ca. 68 % of the trees, these correct predictions will also dominate majority vote.

Training error is therefore usually not even reported for random forests -- and it cannot tell you anything about the generalization error of the forest, nor whether the forest is overfit.

This is not a flaw in the algorithm, it's by design. Usually, when worrying about overfitting the idea is that the model may be too complex and thus unstable. Random forest by design uses overfit trees as submodels. The ensemble prediction then takes care of the overfitting.


The out-of-bag estimate of generalization error that is usually computed alongside random Forest training, however, in the usual implementation depends crucially on independence between the data rows. If rows are dependent (clustered/hierarchical/nested data structure), the oob esimate will have an optimistic bias.
In that case, a cross-validation whose splits achieve independence will help to get a more realistic estimate of the generalization performance. An off-the-shelf cross validation randomly assigning rows won't - it has the same problem the oob estimate has with such data.


I am using a grid search to find the best hyperparameters

When evaluating the grid search results, you do take into account that some of the rF hyperparameters behave differently from how most hyperparameters for other models behave, do you? In particular, don't expect a classical optimum for the number of trees.

$\endgroup$
1
$\begingroup$

Random Forest Classifiers (RFC) with 100% training accuracy are not necessarily problematic.

Make sure you are optimizing your hyperparameters on a separate validation set, this is especially important with RFC because overfitting is more difficult to identify. In the end, you should have a training, validation and a testing set. Other than that, you may be in good shape.

$\endgroup$
2
  • 2
    $\begingroup$ The linked blog post does not say RF is known to overfit: "The post explains why 100% train accuracy with Random Forest has nothing to do with overfitting." $\endgroup$
    – cbeleites
    Commented Aug 2, 2022 at 13:09
  • $\begingroup$ You are correct, I should have taken more effort to make sure the ideas communicated in my answer were reflected in the linked blog post. $\endgroup$
    – SeanBrooks
    Commented Aug 2, 2022 at 17:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.