3
$\begingroup$

In considering ROC AUC, there is a sense in which $0.5$ is the performance of a random model. Conveniently, this is true, no matter the data or the prior probability of class membership; the ROC AUC of a random model is $0.5$ whether the binary classes are balanced or not.

Does PR AUC have a similar notion that is data-independent? (Since precision depends on the prior probability, I would imagine not.) If not, is there a notion that is a function of the prior probability?

EDIT

"ROC" means receiver-operator characteristic.

"PR" means precision-recall.

"AUC" means area under the curve.

Thus, "PR AUC" is the area under the curve that plots precision as a function of recall.

Improve this question
$\endgroup$
5
  • $\begingroup$ What does PR AUC mean exactly? I’m aware that PR stands for “precision” and AUC stands for “area under curve”, but I’m not sure what they mean together $\endgroup$
    – mhdadk
    Mar 13 at 20:21
  • $\begingroup$ @mhdadk Edited to clarify $\endgroup$
    – Dave
    Mar 13 at 20:24
  • $\begingroup$ In the risk of making a circular argument: if we have a prior probability doesn't that imply we have some notion of incidence rate and therefore we have a prior baseline? (We know that the baseline of the PR curve is the proportion of positive examples in our data; I have expanded on why this is the case on the thread What is "baseline" in precision recall curve and I particularly cover the case of a "random classifier".) $\endgroup$
    – usεr11852
    Mar 30 at 0:55
  • $\begingroup$ @usεr11852 Should this be considered a duplicate of the linked question? I need to read it in detail to understand it, but it sure seems to address this. $\endgroup$
    – Dave
    Mar 30 at 2:55
  • $\begingroup$ I think there is a substantial overlap indeed. That said, it doesn't touch upon the notion of a "prior probability there but as I comment here, it seems a bit circular to me... (Also one more partial duplicate won't kill the site so it's OK :D ) $\endgroup$
    – usεr11852
    Mar 30 at 16:20

1 Answer 1

Reset to default
2
$\begingroup$

Firstly, you will want to have a look at precision-recall-gain curves, which enable comparison of classifier performance across datasets with different base rates. It's basically just a clever (and theoretically justified) nonlinear rescaling such that PRG curves cover [0,1]x[0,1] and are baseline-independent.

Snippet from Flach and Kull, NeurIPS 2015, see link below. Left: normal PR curve. Right: rescaled PRG curve.

enter image description here

Secondly, in PR(G) space, the baseline to beat is the always-positive classifier, not the random classifier. That model has precision=baseline and recall=1.

Thirdly, in their NeurIPS paper on PRG curves, Flach and Kull show that classifiers with the same $F_1$ score as the always-positive classifier lie on the (0,1)--(1,0) diagonal in PRG space. Any model operating point below that line thus has worse $F_1$ score than the always-positive classifier.

The only aspect that is still unclear to me (and they also don't write this explicitly in the paper) is whether that also implies a meaningful baseline of AUPRG=0.5 to beat, i.e., whether that baseline is also meaningful for the area under the curve, and not just pointwise. It seems to me that you could have AUPRG < 0.5, but as long as you have an operating point above the diagonal, the model could still be very useful if employed in that operating point? I think the AUROC=0.5 baseline relies crucially on the fact that any point below the diagonal can be "mirrored" by taking 1-the score instead of the actual score. It's not clear to me whether something like that can also be done in PR(G) space.

Lastly, if you really want to see what the random classifier is doing in PR(G) space, Flach and Kull also have an expression for its $F_1$ score as a function of the baseline. You may also be interested in this earlier paper on the relationship between PR and ROC curves.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I have upvoted this because it brings to my attention a related and interesting idea. However, this does not appear to answer the question about if there is a "baseline" value of PRAUC analogous to $0.5$ for ROCAUC, even if it depends on the prior probability. This may be implicit in the discussion about the precision-recall gain curve, but I am not seeing it. If that was your intention, could you please make explicit how the PR gain relates to a baseline PR AUC? $\endgroup$
    – Dave
    Mar 15 at 15:03
  • $\begingroup$ @Dave that would be $\text{baseline incidence} \times 1$ (always-predict-positive classifier) $\endgroup$
    – Firebug
    Mar 23 at 10:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.