5
$\begingroup$

I understand for highly imbalanced dataset - we need to look for precision-recall vs ROC AUC to better judge the model.

My question is what is the range for PR AUC below which the model is bad? My current model has an ROC AUC of >90% while PR AUC is only 40%. Is the model bad due to low PR AUC or range for PR AUC is different than ROC AUC?

$\endgroup$
3
  • $\begingroup$ A bit more explanation of why it "looks" bad. Would be helpful. $\endgroup$
    – Khungry
    Sep 2, 2019 at 23:37
  • $\begingroup$ @user2974951 aren't they different metrics? $\endgroup$
    – Khungry
    Sep 4, 2019 at 3:24
  • $\begingroup$ Don't think that answers my question or you are getting what I am asking. Respectfully I will wait for someone's else response. $\endgroup$
    – Khungry
    Sep 4, 2019 at 7:26

1 Answer 1

6
$\begingroup$

You are correct to be suspicious of your results. While indeed it is relatively easy to somewhat simplistically dismiss an AUCROC as "bad" if it is close to $0.50$ (roughly speaking the probability that the model ranks a random positive example more highly than a random negative example), the same rationale is not relevant to the case of AUCPR. That is because the baseline of an AUCPR is not $0.50$ but rather it is dictated by the proportion of positives in our sample. That means that when dealing with an imbalanced sample our actual base-line might extremely low; one can read a more detail exposition on this matter on the CV.SE thread here: What is "baseline" in precision recall curve.

If we want a more informative interpretation of the P-R analysis, we can use what is knows as Precision-Recall Gain curves; these allows us to view the AUCPRG as the expected $F_1$ score. Details on the CV.SE thread gere: Area Under the Precision Recall curve -similar interpretation to AUROC?.

So to recap, a model with AUCROC ~ $90\%$ and AUCPR ~ $40\%$ is not bad, or good for that matter. Without a reference point for performance these numbers do not much match and especially the AUCPR does not lent itself to simple direct interpretations either.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.