Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I understand for highly imbalanced dataset - we need to look for precision-recall vs ROC AUC to better judge the model.
My question is what is the range for PR AUC below which the model is bad? My current model has an ROC AUC of >90% while PR AUC is only 40%. Is the model bad due to low PR AUC or range for PR AUC is different than ROC AUC?
You are correct to be suspicious of your results. While indeed it is relatively easy to somewhat simplistically dismiss an AUCROC as "bad" if it is close to $0.50$ (roughly speaking the probability that the model ranks a random positive example more highly than a random negative example), the same rationale is not relevant to the case of AUCPR. That is because the baseline of an AUCPR is not $0.50$ but rather it is dictated by the proportion of positives in our sample. That means that when dealing with an imbalanced sample our actual baseline might extremely low; one can read a more detailed exposition on this matter on the CV.SE thread here: What is "baseline" in precision recall curve.
If we want a more informative interpretation of the P-R analysis, we can use what is knows as Precision-Recall Gain curves; these allow us to view the AUCPRG as the expected $F_1$ score. Details on the CV.SE thread here: Area Under the Precision Recall curve -similar interpretation to AUROC?.
So to recap, a model with AUCROC ~ $90\%$ and AUCPR ~ $40\%$ is not bad or good for that matter. Without a reference point for performance, these numbers do not matter much and especially the AUCPR does not lend itself to simple direct interpretations either.
