2
$\begingroup$

I understand for highly imbalanced dataset - we need to look for precision-recall vs ROC AUC to better judge the model.

My question is what is the range for PR AUC below which the model is bad? My current model has an ROC AUC of >90% while PR AUC is only 40%. Is the model bad due to low PR AUC or range for PR AUC is different than ROC AUC?

$\endgroup$
  • $\begingroup$ There is no pre-defined range. Having said that, your model, given your results, looks pretty bad. $\endgroup$ – user2974951 Sep 2 at 12:50
  • $\begingroup$ A bit more explanation of why it "looks" bad. Would be helpful. $\endgroup$ – Khungry Sep 2 at 23:37
  • $\begingroup$ The difference between 90 and 40 is significant. $\endgroup$ – user2974951 Sep 3 at 12:57
  • $\begingroup$ @user2974951 aren't they different metrics? $\endgroup$ – Khungry Sep 4 at 3:24
  • $\begingroup$ Yes, but they are both bounded between 0 and 1 (or 0 and 100), so they are comparable. $\endgroup$ – user2974951 Sep 4 at 7:13
3
$\begingroup$

You are correct to be suspicious of your results. While indeed it is relatively easy to somewhat simplistically dismiss an AUCROC as "bad" if it is close to $0.50$ (roughly speaking the probability that the model ranks a random positive example more highly than a random negative example), the same rationale is not relevant to the case of AUCPR. That is because the baseline of an AUCPR is not $0.50$ but rather it is dictated by the proportion of positives in our sample. That means that when dealing with an imbalanced sample our actual base-line might extremely low; one can read a more detail exposition on this matter on the CV.SE thread here: What is "baseline" in precision recall curve.

If we want a more informative interpretation of the P-R analysis, we can use what is knows as Precision-Recall Gain curves; these allows us to view the AUCPRG as the expected $F_1$ score. Details on the CV.SE thread gere: Area Under the Precision Recall curve -similar interpretation to AUROC?.

So to recap, a model with AUCROC ~ $90\%$ and AUCPR ~ $40\%$ is not bad, or good for that matter. Without a reference point for performance these numbers do not much match and especially the AUCPR does not lent itself to simple direct interpretations either.

$\endgroup$
0
$\begingroup$

There is no pre-defined range or difference which tells you whether the two values are quite different. You have to use your own judgement and knowledge of the data to decide.

Having said that, your AUROC is relatively good (>90), however your PRAUC is not (<50), which tells you that you did not effectively model the imbalance in your data. In other words, most of the good predictions were in the majority class.

If you had dealt with the imbalance in your data, your model would return similar AUROC and PRAUC values. The difference between 90 and 50 is significant.

For a clearer picture of what is going on try using a more easily interpretable measure such as accuracy, F1, and so on (despite their shortcomings), this will tell you quickly where the issues are.

$\endgroup$
  • $\begingroup$ When dealing with imbalanced data, using accuracy can be misleading. You might want to look into the thread: Why is accuracy not the best measure for assessing classification models? for more details. In general, I would suggest to avoid the use of metrics that do hard-class assignments. (I did not down-vote this post but I try to explain why someone might find this advice not useful.) $\endgroup$ – usεr11852 says Reinstate Monic Sep 5 at 18:28
  • $\begingroup$ @usεr11852 Yes I am aware of the shortcomings of accuracy and other such metrics, I have seen Stephan's post. I suggested estimating them because they are much easier to interpret than something like PRAUC, that is you can quickly see if something is wrong by comparing accuracy and F1 scores. Also, it is true that you cannot conclude anything from a PRAUC score by itself since it does not have a fixed baseline, however you can draw some conclusions by comparing it with an AUROC score. In general, we should expect similar scores if we dealt with the imbalance. $\endgroup$ – user2974951 Sep 6 at 6:54
  • $\begingroup$ I am not fully supportive of the statement that "we should expect similar (AUCROC & AUCPR) scores if we dealt with the imbalance." Standard references on the relation of the two metrics (e.g. Davis & Goadric (2006), Flach & Kull (2015)) do not suggest anything of this kind; to me, it also seems somewhat unlikely given the variable baseline of AUCPR. Could you please provide some reference regarding this statement? $\endgroup$ – usεr11852 says Reinstate Monic Sep 6 at 10:21
  • $\begingroup$ @usεr11852 I do not have any references for that statement, only my (limited) experience. $\endgroup$ – user2974951 Sep 6 at 10:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.