0
$\begingroup$

I'm trying to evaluate my different feature scoring methods with the same data by cross-validation.

I used svm for both approaches, the only difference is the scoring method for each feature. e.g. feature1 scored 0.8 in method 1 and maybe 0.9 in method 2, etc. I want to know which scoring scheme is better by looking at the performance.

I did some resampling in the process and the cross-validation for multiple times for each method, so I got more than one result for each method. I calculated the mean value of AUCs under the curves.

The two approaches basically gave me same ROC curve AUCs. But the AUCs under Precision/Recall curve were different. How can I interpret this?

Thanks!

$\endgroup$
0
$\begingroup$

The ROC and PR curves are related, and there is a mapping between points on the ROC and PR curves (see, for example, the paper "The Relationship Between Precision-Recall and ROC Curves" by Jesse Davis and Mark Goadrich). They note that

a curve dominates in ROC space if and only if it dominates in PR space

and show that

an algorithm that optimizes the area under the ROC curve is not guaranteed to optimize the area under the PR curve.

So what you see is totally normal. In short, AUC of the PR curve is more informative than AUC of the ROC curve, especially when there is class imbalance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.