overfitting of random forest in r

Question

I am running a random forest classifier in R and during 10-fold cross-validation, I discovered that the model is overfitting. I am using a grid search to find the best hyperparameters and used the best model to predict the same training set again. The random Forest model classified every data point of the train set perfectly. I ran the same model on the test set and got an accuracy of 0.6976. Does this mean that the model is overfitting? I thought random forest models don't overfit? My data is slightly imbalanced with 110 positive samples and 95 negative samples.

Answer 1

How does the 70 % accuracy of your CV compare to the rF's oob estimate?

---

The behaviour you observe is to be expected for random forests, see also my old answer: https://stats.stackexchange.com/a/208018/4598

The submodels of a random forest are unpruned decision trees. Training data for such an unpruned tree will basically be recognized as known and assigned to the correct class. Since each case in the training set for the random forest is training case for ca. 68 % of the trees, these correct predictions will also dominate majority vote.

Training error is therefore usually not even reported for random forests -- and it cannot tell you anything about the generalization error of the forest, nor whether the forest is overfit.

This is not a flaw in the algorithm, it's by design. Usually, when worrying about overfitting the idea is that the model may be too complex and thus unstable. Random forest by design uses overfit trees as submodels. The ensemble prediction then takes care of the overfitting.

---

The out-of-bag estimate of generalization error that is usually computed alongside random Forest training, however, in the usual implementation depends crucially on independence between the data rows. If rows are dependent (clustered/hierarchical/nested data structure), the oob esimate will have an optimistic bias.
In that case, a cross-validation whose splits achieve independence will help to get a more realistic estimate of the generalization performance. An off-the-shelf cross validation randomly assigning rows won't - it has the same problem the oob estimate has with such data.

---

I am using a grid search to find the best hyperparameters

When evaluating the grid search results, you do take into account that some of the rF hyperparameters behave differently from how most hyperparameters for other models behave, do you? In particular, don't expect a classical optimum for the number of trees.

Answer 2

Random Forest Classifiers (RFC) with 100% training accuracy are not necessarily problematic.

Make sure you are optimizing your hyperparameters on a separate validation set, this is especially important with RFC because overfitting is more difficult to identify. In the end, you should have a training, validation and a testing set. Other than that, you may be in good shape.