I have the following problem and I'm desperately looking for a solution since several weeks, so I'm hoping to find some help here. In this post, I'm always refering to Sklearn.
My goal: I want to classify a dataset with 4 classes which is imbalanced. The ratio of datapoints is approximately 1:1:1:5. I have a total number of 5165 datapoints and 632 features. The final metric I want to optimize is the balanced accuracy score (= average recall rate over all classes).
My current status: I tried using the calculation pipeline below to solve the problem. With a SVC using an RBF kernel and doing a gridsearch for hyperparameter optimization, I reached a sufficient result of approximately 0.85 for my dataset.
My problem: If I use the RandomForestClassifier for prediction, the result is very bad (bac is around 0.45 or even lower, using the same pipeline as before just another classifier). What I see is that during training I reach a bac of 1 (perfect), but the bac of the testset is as bad as already mentioned. If we consider the fact that random guessing would give a bac of 0.25, the result looks even worse. So my problem might be overfitting. But how to solve this problem? I already checked several resources like:
The result didn't get any better. I tried: - using a different amount of n_estimators: 100, 1000, 5000, ... - using undersampling using Kmeans or oversampling using SMOTE from imblearn - criterion 'gini' or 'entropy' - different values for max_features - oob_score True / False
I have to admit that I don't really understand the properties like max_leaf_nodes mean, so I just used the default values there. I'm not looking for the perfect result, all I'm looking for is a more or less reasonable result and a start point to go on. For me, it seems that the RandomForestClassifier doesn't work here at all, like there is a bug or so (which is most probably not the case).
Do you have any suggestion what I could try to do, what parameters are the ones I have to focus on?
Below I inserted my calculation steps using a dataset from Sklearn. Please note that this is not my dataset I want to do my prediction. On this dataset, the algorithm seems to work ok.
I'm really grateful for every helpful reply.
from sklearn.datasets import load_breast_cancer from sklearn.feature_selection import SelectKBest, f_classif from sklearn.preprocessing import RobustScaler from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import train_test_split from sklearn.metrics import make_scorer, balanced_accuracy_score y = load_breast_cancer(return_X_y=True) X = y y = y X_1, X_2, y_1, y_2 = train_test_split(X, y, test_size=0.2) scaling = RobustScaler() feature_selection = SelectKBest(score_func=f_classif, k=25) estimator = RandomForestClassifier(class_weight='balanced') X_1 = scaling.fit_transform(X_1, y_1) X_2 = scaling.transform(X_2) X_1 = feature_selection.fit_transform(X_1, y_1) X_2 = feature_selection.transform(X_2) estimator = estimator.fit(X_1, y_1) prediction = estimator.predict(X_2) print(balanced_accuracy_score(y_2, prediction))