2
$\begingroup$

I am trying to interpret the AUCPR. Say I have the following Precision-Recall curve.

Firstly: It ends at 0.38 on the y-axis because this particular plot has slightly imbalanced data, so am I correct in thinking its the sum of the positive class / the sum of the negative class.

If I had a negative class of 26300 and a positive class of 10000 then the PR curve would stop at 10000 / 26300 = 0.38.

Secondly: In regards to the AUCPR. I understand that perfect accuracy would be a straight line going along the top of the plot but if this curve had a AUCPR of 0.90 then it would be considered good, however how can I interpret it when the value is perhaps 0.50, this couldn´t be a random guess like in AUROC since the curve ends at 0.38. Is there a way to know when the model becomes worse than flipping a coin?

enter image description here

$\endgroup$
  • $\begingroup$ But isn't the AUCPR in that image closer to 0.98? $\endgroup$ – Dan Jun 21 '18 at 14:46
  • $\begingroup$ Yes probably, I just took a PR curve from the internet and made up a AUCPR $\endgroup$ – user113156 Jun 21 '18 at 14:58
  • $\begingroup$ It’s not total positive over total negative. It’s total positive over total. That’s also the PR curve of the random classifier (the line y = positive prevalence). $\endgroup$ – kbrose Jun 21 '18 at 17:17
0
$\begingroup$

The PR curve has a simple geometric interpretation of the expected precision when uniformly varying the recall. It does have a 1-1 correspondence to the ROC curve as shown by Davis & Goadrich (2006) "The relation between precision-recall and ROC curves".

A more informative interpretation of AUCPR can be found in Flach & Kill (2015) "Precision-Recall-Gain Curves: PR Analysis Done Right". There, the authors present the concept of Precision-Recall-Gain curves, a reformulation of the usual PR curve where the "always positive" classifier serves as the baseline. Following that the AUPRG can be interpreted as the expected $F_1$ score.

Regarding the specific side-questions raised:

  1. $0.38$ reflects the proportion of positive examples within the training sample. So in the particular case mentioned having $26300$ negative and $10000$ positive examples leads to a base-line of $~0.275$. A more extensive discussion on the baseline of a PR curve can be found in the following CV.SE tread: What is "baseline" in precision recall curve
  2. The AUPR in itself is not very informative. As mentioned that related to the expected precision when uniformly varying the recall. The model quickly becomes "worse than flipping a coin" as we move to more imbalanced datasets; if we know for example that we have a 90-10 imbalance a "fair coin" is bad and it is to our advantage to use a "loaded coin". This is where the work of F&K cited above comes into play; it directly models the gain in terms of P-R assuming a "recall-aware" baseline. To that extent, one might want to look at Cohen's $\kappa$ as a quick measurement as it directly accounts for the expected accuracy of a classifier; personally I use almost always when first looking a binary classifier's results. CV.SE has an excellent thread explaining it, in more detail: Cohen's kappa in plain English.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.