I am working with a very imbalanced dataset (16k lines, 4% in the minority class), using random forest to for a binary classification. I’m using the Python Sklearn implementation of RandomForestClassifier and have tried using class_weights to balance the classes . I’ve also tried upsampling the minority class using both simple upsampling and SMOTE. When upsampling the training set I get great results on the training set - around .80 accuracy and F1, which are my metrics of interest. I then get horrendous results on my non-upsampled test set (F1 plummets to well below .20).
These results imply overfitting, and there seems to be some debate over whether RF is even capable of overfitting.
Any suggestions as far as resampling or modifying the optimization process to reduce the gap between training and test data? What am I missing?
A few of the details: Using RandomizedGridSearchCV to refit the RandomForestClassifier on the f1_score I am upsampling before the train_test_split and specifying a stratified split Data is 16k lines with 5 features and binary target variable I’ve also tried doing nothing to balance the classes and get poor results on both training and test sets I’ve tried optimizing via Gridsearch on a number of different metrics - F1 gives me most desirable results, but nothing resolves the “overfitting” when upsampling
Any ideas would be appreciated!