0
$\begingroup$

So I know that the area under the precision-recall curve is often a more useful metric than AUROC when dealing with highly imbalanced datasets. However, while AUROC can easily be used to compare classifier performance between different groups (since values are on the same scale), AUC-PR has the "problem" that baselines may differ between groups: a random classifier will have an AUC-PR that is equal to the baseline incidence in that group, i.e., the fraction of positive outcomes.

Given this knowledge, how can one compare AUC-PR values between groups with different baselines? Obviously, one cannot simply compare the absolute values since they are on different scales: a value of 0.5 might indicate a fully random classifier in one group (with incidence 50%) and a pretty good classifier in another group (with incidence of, say, 10%).

One obvious idea would be to normalize AUC-PR values to the range [0,1] by subtracting the baseline and dividing by (1-baseline) in each group. Would it be a good idea to use such normalized AUC-PR values for comparing classifier performance between groups? Are there drawbacks? Do people do this? Are there other, better ways?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Have a look at AUPR gain curves. papers.nips.cc/paper/2015/hash/… $\endgroup$
    – Georg M. Goerg
    Jan 1 at 23:40
  • $\begingroup$ @GeorgM.Goerg Excellent suggestion, thanks! I should have guessed that Peter had written something good on the subject. ;-) Will try to write up a short answer based on that in the next days (unless you want to do it!). $\endgroup$
    – Eike P.
    Jan 3 at 15:50

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.