3
$\begingroup$

In considering ROC AUC, there is a sense in which $0.5$ is the performance of a random model. Conveniently, this is true, no matter the data or the prior probability of class membership; the ROC AUC of a random model is $0.5$ whether the binary classes are balanced or not.

Does PR AUC have a similar notion that is data-independent? (Since precision depends on the prior probability, I would imagine not.) If not, is there a notion that is a function of the prior probability?

EDIT

"ROC" means receiver-operator characteristic.

"PR" means precision-recall.

"AUC" means area under the curve.

Thus, "PR AUC" is the area under the curve that plots precision as a function of recall.

$\endgroup$
2
  • $\begingroup$ What does PR AUC mean exactly? I’m aware that PR stands for “precision” and AUC stands for “area under curve”, but I’m not sure what they mean together $\endgroup$
    – mhdadk
    Mar 13 at 20:21
  • $\begingroup$ @mhdadk Edited to clarify $\endgroup$
    – Dave
    Mar 13 at 20:24

1 Answer 1

2
$\begingroup$

Firstly, you will want to have a look at precision-recall-gain curves, which enable comparison of classifier performance across datasets with different base rates. It's basically just a clever (and theoretically justified) nonlinear rescaling such that PRG curves cover [0,1]x[0,1] and are baseline-independent.

Snippet from Flach and Kull, NeurIPS 2015, see link below. Left: normal PR curve. Right: rescaled PRG curve.

enter image description here

Secondly, in PR(G) space, the baseline to beat is the always-positive classifier, not the random classifier. That model has precision=baseline and recall=1.

Thirdly, in their NeurIPS paper on PRG curves, Flach and Kull show that classifiers with the same $F_1$ score as the always-positive classifier lie on the (0,1)--(1,0) diagonal in PRG space. Any model operating point below that line thus has worse $F_1$ score than the always-positive classifier.

The only aspect that is still unclear to me (and they also don't write this explicitly in the paper) is whether that also implies a meaningful baseline of AUPRG=0.5 to beat, i.e., whether that baseline is also meaningful for the area under the curve, and not just pointwise. It seems to me that you could have AUPRG < 0.5, but as long as you have an operating point above the diagonal, the model could still be very useful if employed in that operating point? I think the AUROC=0.5 baseline relies crucially on the fact that any point below the diagonal can be "mirrored" by taking 1-the score instead of the actual score. It's not clear to me whether something like that can also be done in PR(G) space.

Lastly, if you really want to see what the random classifier is doing in PR(G) space, Flach and Kull also have an expression for its $F_1$ score as a function of the baseline. You may also be interested in this earlier paper on the relationship between PR and ROC curves.

$\endgroup$
2
  • $\begingroup$ I have upvoted this because it brings to my attention a related and interesting idea. However, this does not appear to answer the question about if there is a "baseline" value of PRAUC analogous to $0.5$ for ROCAUC, even if it depends on the prior probability. This may be implicit in the discussion about the precision-recall gain curve, but I am not seeing it. If that was your intention, could you please make explicit how the PR gain relates to a baseline PR AUC? $\endgroup$
    – Dave
    Mar 15 at 15:03
  • $\begingroup$ @Dave that would be $\text{baseline incidence} \times 1$ (always-predict-positive classifier) $\endgroup$
    – Firebug
    yesterday

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.