10
$\begingroup$

I'm currently comparing three methods and I have the Accuracy, auROC and auPR as metrics. And I have the following results :

Method A - acc: 0.75, auROC: 0.75, auPR: 0.45

Method B - acc: 0.65, auROC: 0.55, auPR: 0.40

Method C - acc: 0.55, auROC: 0.70, auPR: 0.65

I have a good understanding of accuracy and auROC (to remember well i often try to come up with a sentence like "auROC = characterize the ability to predict the positive class well", while not exactly correct it helps me remember). I have never had auPR data before and while I understand how it is built I can't get the "feeling" behind it.

In fact I fail to understand why the method C has an incredibly high score for auPR while being bad/average for the accuracy and auPR.

If someone could help me understand it a little better with a simple explanation that would be really great. Thank you.

$\endgroup$
10
+50
$\begingroup$

One axis of ROC and PR curves is the same, that is TPR: how many positive cases have been classified correctly out of all positive cases in the data.

The other axis is different. ROC uses FPR, which is how many mistakenly declared positives out of all negatives in the data. PR curve uses precision: how many true positives out of all that have been predicted as positives. So the base of the second axis is different. ROC uses what's in the data, PR uses what's in the prediction as a basis.

PR curve is thought to be more informative when there is a high class imbalance in the data, see this paper http://pages.cs.wisc.edu/~jdavis/davisgoadrichcamera2.pdf .

$\endgroup$
  • $\begingroup$ For the auROC 0.5 is the minimum (because less would be better by inverting the predicition). Is there some similar rules with the auPR? Also concerning my measurements : what could I assert by looking at the scores of the Method C? Because I'm working with the same dataset in the 3 cases and from my point of view for a dataset with more or less even distribution among the classes it wouldn't make sense that the auROC and auPR do not follow the same ranking for my methods. $\endgroup$ – AdrienNK May 28 '14 at 15:25
  • $\begingroup$ what is the random classifier score in auPR? I know it's 0.5 in auROC but I am unable to know in auPR. $\endgroup$ – Jack Twain Oct 28 '14 at 19:56
  • 8
    $\begingroup$ The expected auPR score for a random classifier is just the proportion of true positive cases in the dataset. That is the precision you would expect if you were to guess the class, and you would get that precision for all levels of recall. So the expected PR curve for a random classifier is just a rectangle with side lengths "proportion of true positives" x 1. For example, if your dataset contains 10% positive cases and 90% negative cases, the expected auPR under chance is 0.1. $\endgroup$ – Lizzie Silver May 9 '16 at 21:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.