9
$\begingroup$

I understand for highly imbalanced dataset - we need to look for precision-recall vs ROC AUC to better judge the model.

My question is what is the range for PR AUC below which the model is bad? My current model has an ROC AUC of >90% while PR AUC is only 40%. Is the model bad due to low PR AUC or range for PR AUC is different than ROC AUC?

$\endgroup$
3
  • $\begingroup$ A bit more explanation of why it "looks" bad. Would be helpful. $\endgroup$ Commented Sep 2, 2019 at 23:37
  • $\begingroup$ @user2974951 aren't they different metrics? $\endgroup$ Commented Sep 4, 2019 at 3:24
  • $\begingroup$ Don't think that answers my question or you are getting what I am asking. Respectfully I will wait for someone's else response. $\endgroup$ Commented Sep 4, 2019 at 7:26

2 Answers 2

8
$\begingroup$

You are correct to be suspicious of your results. While indeed it is relatively easy to somewhat simplistically dismiss an AUCROC as "bad" if it is close to $0.50$ (roughly speaking the probability that the model ranks a random positive example more highly than a random negative example), the same rationale is not relevant to the case of AUCPR. That is because the baseline of an AUCPR is not $0.50$ but rather it is dictated by the proportion of positives in our sample. That means that when dealing with an imbalanced sample our actual baseline might extremely low; one can read a more detailed exposition on this matter on the CV.SE thread here: What is "baseline" in precision recall curve.

If we want a more informative interpretation of the P-R analysis, we can use what is knows as Precision-Recall Gain curves; these allow us to view the AUCPRG as the expected $F_1$ score. Details on the CV.SE thread here: Area Under the Precision Recall curve -similar interpretation to AUROC?.

So to recap, a model with AUCROC ~ $90\%$ and AUCPR ~ $40\%$ is not bad or good for that matter. Without a reference point for performance, these numbers do not matter much and especially the AUCPR does not lend itself to simple direct interpretations either.

$\endgroup$
0
$\begingroup$

My question is what is the range for PR AUC below which the model is bad?

The PRAUC is bad when it is less than the prior probability of membership in the minority category, for a similar reason to why ROCAUC < 0.5 is bad or the argument I make here about classification accuracy (related to this). To see why, let's revisit our old friend, $R^2$ from linear regression.

We can look at $R^2$ as a comparison between the square loss of your model vs the square loss of a benchmark model that predicts $\bar y$ every time.

$$ R^2=1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right) =\dfrac{ \left( \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 \right) - \left( \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 \right)}{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 } $$

That right-most fraction is of the form of percent change. We can think of it as the percentage of garbage taken out. If we take out 70% of the garbage (square loss of the benchmark), that is better than taking out 40% of the garbage.

We can view ROCAUC in this same context, and that is kind of how we wind up with $0.5$ as the must-beat level of performance. Somers' $D$ takes this same idea by considering the area above the ROC curve as a measure of error and compares it to the benchmark area above the curve of $0.5$.

$$ D = 1 - \left(\dfrac{ 1 - ROCAUC } { 1 - 0.5 }\right) = 2ROCAUC - 1 $$

I say that you can do the same with the PRAUC, once you determine a benchmark, which is typically taken to be a horizontal line at the prior probability of the minority category (the proportion of overall events that belong to the minority category). Call that prior probability $p$. The area under this horizontal line (on the interval $(0, 1)$) is $p$ itself, so the area above the curve is $1 - p$.

$$ 1 - \left(\dfrac{ 1 - PRAUC } { 1 - p }\right) $$

In order to get a positive value that indicates better PRAUC than the benchmark, all you have to do is have a PRAUC greater than the prior probability $p$. You say your PRAUC is below $0.5$. However, consider what your $p$ is. If you have $p = 0.1$, then you get an $R^2$-inspired calculation, assuming $PRAUC = 0.4$ (say), of $1 - \frac{0.6}{0.9} = 33\%$.

You may feel a bit better about your perceived low PRAUC by putting it in this context and seeing that it makes a real improvement over a reasonable benchmark, where you can quantify that improvement.

This idea of comparing to a benchmark is totally aligned with how Gneiting and Resin (2023) develop the $R^*$ in their equations (31) and (32) that they call a universal coefficient of determination.

There is more to the model evaluation than looking at ROC and PR curves. For instance, neither of these consider model that might matter for your task the way that strictly proper do, but I do think this way of thinking gives important context to your PR and ROC curves.

(I've posted about this Gneiting/Resin paper a number of times in the past few weeks. I've been making an argument on her for years to do $R^2$-inspired comparisons to benchmark model performance, so now that I've found a reference, I am including it whenever I make such an argument.)

REFERENCE

Gneiting, Tilmann, and Johannes Resin. "Regression diagnostics meets forecast evaluation: Conditional calibration, reliability diagrams, and coefficient of determination." Electronic Journal of Statistics 17.2 (2023): 3226-3286.

$\endgroup$
1
  • $\begingroup$ This post is completely consistent with the other answer. Do you see why? $\endgroup$
    – Dave
    Commented 2 days ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.