I have a very imbalanced dataset (9:1), for which I have performed under-sampling and achieved a balanced training set (~130k samples total post balancing).

I am performing classification using RandomForest. My test set comprises of unbalanced data (since this is how it is expected to be in the real world), and my ROC curve is pretty good, with AUC=0.873. However, my P-R curve suffers, with AUC=0.547.

P-R curve with imbalanced test data

If I balance the test data, I see a better AUC; however, this is not how the ground truth has behaved in the past, so quoting performance on balanced test data doesn't seem too rational.

P-R curve with balanced test data

What am I doing wrong? Should I train on imbalanced data? Or are there any ways where I can improve the P-R curve?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.