My dataset has about ~75,000 records with 39 features. Most of the features are categorical, so I have one-hot encoded them. About 14% are minority with label 0 and the rest 86% with label 1. I have used XGBoost, Logistic Regression, and Random Forest using scikit-learn. I have tried everything from upsampling, downsampling, and SMOTE. I've also used GridSearchCV for my models and set class_weights = 'balanced' and tested custom ones as well. Weighting {0:2, 1:1} actually works best.

It seems like the optimal evaluation metric is the f1 score. I've also looked at precision, recall, auc. The best I can do is a Random Forest with an f1 score for my minority class of .48 and for my majority class of .92.

Is this good enough? I've read everything I feel like I can about imbalanced dataset and resampling. Is it ultimately my features that don't allow for a good prediction? I don't really have any more features to add.


  • $\begingroup$ You should be evaluating the quality of your model based on probabilities. All of the models you have listed view the world probabilistically, it is only humans that need to enforce decisions, and it's good to keep these two issues separate in your analysis. So work with your raw data: is your model learning to distinguish from positive and negative probabilistically? Are the output probabilities for your positive classes on average higher than for your negative classes? That's the first question to answer. $\endgroup$ – Matthew Drury Mar 22 at 18:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.