What does it mean when two methods have same ROC AUC but different Precision/Recall AUC?

Question

I'm trying to evaluate my different feature scoring methods with the same data by cross-validation.

I used svm for both approaches, the only difference is the scoring method for each feature. e.g. feature1 scored 0.8 in method 1 and maybe 0.9 in method 2, etc. I want to know which scoring scheme is better by looking at the performance.

I did some resampling in the process and the cross-validation for multiple times for each method, so I got more than one result for each method. I calculated the mean value of AUCs under the curves.

The two approaches basically gave me same ROC curve AUCs. But the AUCs under Precision/Recall curve were different. How can I interpret this?

Thanks!

Answer 1

The ROC and PR curves are related, and there is a mapping between points on the ROC and PR curves (see, for example, the paper "The Relationship Between Precision-Recall and ROC Curves" by Jesse Davis and Mark Goadrich). They note that

a curve dominates in ROC space if and only if it dominates in PR space

and show that

an algorithm that optimizes the area under the ROC curve is not guaranteed to optimize the area under the PR curve.

So what you see is totally normal. In short, AUC of the PR curve is more informative than AUC of the ROC curve, especially when there is class imbalance.