Problem with overfitting of RandomForestClassifier (Sklearn)

Question

I have the following problem and I'm desperately looking for a solution since several weeks, so I'm hoping to find some help here. In this post, I'm always refering to Sklearn.

My goal: I want to classify a dataset with 4 classes which is imbalanced. The ratio of datapoints is approximately 1:1:1:5. I have a total number of 5165 datapoints and 632 features. The final metric I want to optimize is the balanced accuracy score (= average recall rate over all classes).

My current status: I tried using the calculation pipeline below to solve the problem. With a SVC using an RBF kernel and doing a gridsearch for hyperparameter optimization, I reached a sufficient result of approximately 0.85 for my dataset.

My problem: If I use the RandomForestClassifier for prediction, the result is very bad (bac is around 0.45 or even lower, using the same pipeline as before just another classifier). What I see is that during training I reach a bac of 1 (perfect), but the bac of the testset is as bad as already mentioned. If we consider the fact that random guessing would give a bac of 0.25, the result looks even worse. So my problem might be overfitting. But how to solve this problem? I already checked several resources like:

https://towardsdatascience.com/feature-selection-using-random-forest-26d7b747597f https://medium.com/all-things-ai/in-depth-parameter-tuning-for-random-forest-d67bb7e920d

The result didn't get any better. I tried: - using a different amount of n_estimators: 100, 1000, 5000, ... - using undersampling using Kmeans or oversampling using SMOTE from imblearn - criterion 'gini' or 'entropy' - different values for max_features - oob_score True / False

I have to admit that I don't really understand the properties like max_leaf_nodes mean, so I just used the default values there. I'm not looking for the perfect result, all I'm looking for is a more or less reasonable result and a start point to go on. For me, it seems that the RandomForestClassifier doesn't work here at all, like there is a bug or so (which is most probably not the case).

Do you have any suggestion what I could try to do, what parameters are the ones I have to focus on?

Below I inserted my calculation steps using a dataset from Sklearn. Please note that this is not my dataset I want to do my prediction. On this dataset, the algorithm seems to work ok.

I'm really grateful for every helpful reply.

from sklearn.datasets import load_breast_cancer
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.preprocessing import RobustScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import make_scorer, balanced_accuracy_score

y = load_breast_cancer(return_X_y=True)
X = y[0]
y = y[1]
X_1, X_2, y_1, y_2 = train_test_split(X, y, test_size=0.2)

scaling = RobustScaler()
feature_selection = SelectKBest(score_func=f_classif, k=25)
estimator = RandomForestClassifier(class_weight='balanced')

X_1 = scaling.fit_transform(X_1, y_1)
X_2 = scaling.transform(X_2)
X_1 = feature_selection.fit_transform(X_1, y_1)
X_2 = feature_selection.transform(X_2)
estimator = estimator.fit(X_1, y_1)
prediction = estimator.predict(X_2)

print(balanced_accuracy_score(y_2, prediction))

Answer 1

First up, some random forest background to address the questions about what the parameters do:

1. Forests are "trained" one tree at a time. Each tree starts as a single node with all the training data. For each node, a random subset of the features is selected, and the best (feature, value) pair for splitting the data in a "left" and "right" half is determined. "Best" is defined by the criterion. If that split is good enough and allowed by the parameters, then a left and right child are created with the left and right partitions of the data. Then the process continues with the children. If not good enough or allowed, then the node remains a leaf forever.
2. Forests are an ensemble model, and the theory behind ensembles shows you want each estimator in the ensemble to have very low bias and low correlation with the other estimators (implies high bias). Low bias is good in general, while the low correlation means the ensemble average will be much lower variance than the individual trees.

Most of the parameters you mention control the creation of the trees - min_samples_split and min_samples_leaf only allow a split if the current node or resulting nodes, respectively have at least that number of samples. The criterion is how the tree determines which split is best. And so on. Per (2), you usually want the trees to grow very deep (low bias). So focus on min_samples_* on the order of 1-10, have low or nonexistent criteria for the minimum required improvement, that sort of thing.

So with that in mind, here are a couple of things to try:

• try many fewer features per split, order 3-10, much less than the minimum of 100 you've tried thus far. This is because of (2) - you want each tree to be uncorrelated. With 600 features though, my guess is that many of them are effectively duplicates, or at least highly correlated. Some of those features will be better than others at various stages of tree growth. So by taking 100+, the decisions made by each tree may be highly correlated. Many fewer features though will promote diversity of the trees. Don't worry about the top-level decisions being poor - per (2) again, you'll be training deep. So there will be many more opportunities for the tree to make good splits later.
• try ExtraTreesClassifier instead. It's similar to RandomForestClassifier, but the training process in (1) is kicked up a notch in terms of variance: only a single randomly selected pivot is considered for each feature. This adds extra randomness (hence the Extra in ExtraTrees!), which in turn generally improves performance a bit. As a bonus, only considering one pivot per feature makes training the model on continuous-valued features dramatically faster. Win-win.
• per (2) above, in your other features, always favor parameter values that will make the trees more random and let them train deeper.
• if the most important features in your model are axis aligned, be careful of what @Sycorax points out: trees make splits on a single feature at a time, and thus are axis aligned. So making e.g. 45 degree cuts or circles in GPS lat/lon data is impossible - only squares. Deep trees still do okay on this because they can split recursively down to only 1-2 samples per leaf which basically aliases the angled/round boundary into a stairstep pattern. If your data is 2D like lat/lon GPS coordinates, you can mitigate this by adding two new features: a 45 degree rotation of your original coordinates. This lets trees make octagonal boundaries which should be close enough since they're trained deep.
• feature selection: forests are generally robust to overfitting thanks to the randomness among trees and ensemble averaging, but not immune. You can run your favorite feature selection algorithm to cut down on the number of features

I expect the original asker has long since moved on, but other people might find this helpful.

Answer 2

Perhaps K_best with f_classif is removing features that RandomForest could leverage. Try Selecting Kbest features from RF.feature_importances_ then rerun RF with that feature set.