I'm working with imbalanced data, where there are about 40 class=0 cases for every class=1. I can reasonably discriminate between the classes using individual features, and training a naive Bayes and SVM classifier on 6 features and balanced data yielded better discrimination (ROC curves below).
That's fine, and I thought I was doing well. However, the convention for this particular problem is to predict hits at a precision level, usually between 50% and 90%. e.g. "We detected some-number of hits at 90% precision." When I tried this, the maximum precision I could get from the classifiers was about 25% (black line, PR curve below).
I could understand this as a class imbalance problem, since PR curves are sensitive to imbalance and ROC curves aren't. However, the imbalance doesn't seem to affect the individual features: I can get pretty high precision using the individual features (blue and cyan).
I don't understand what's going on. I could understand it if everything performed badly in PR space, since, after all, the data is very imbalanced. I could also understand it if the classifiers looked bad in ROC and PR space - maybe they're just bad classifiers. But what is going on to make the classifiers better as judged by ROC, but worse as judged by Precision-Recall?
Edit: I noticed that in the low TPR/Recall areas (TPR between 0 and 0.35), the individual features consistently outperform the classifiers in both ROC and PR curves. Maybe my confusion is because the ROC curve "emphasizes" the high TPR areas (where the classifiers do well) and the PR curve emphasizes the low TPR (where the classifiers are worse).
Edit 2: Training on nonbalanced data, i.e. with the same imbalance as the raw data, brought the PR curve back to life (see below). I'd guess my problem was improperly training the classifiers, but I don't totally understand what happened.