Better in ROC AUC vs. better in PR AUC [duplicate]

Question

I'm comparing two classification models by computing the area under ROC and Precision-Recall curves.

However sometimes one model is better with AU-ROC but worse in AU-PR, and other times it's better in AU-PR but worse in AU-ROC. (Outdated sentence, please see edit)

Based on that I made a conclusion based on my understanding on the AU-ROC and AU-PR, and I just want to make sure that my understanding/conclusions are corrects.

If one method is better in AU-ROC but worse in AU-PR, then the method is better in Recall but worse in Precision. So you should use this method when you want high recall.

If one method is better in AU-PR but worse in AU-ROC, then the method is better in Precision but worse in Recall. So you should use this method when you want high precision.

Are my understanding/conclusions correct? Precision-Recall tradeoff.

Edit:

The case is the following: one model is always better in AU-ROC. While the other is always better in AU-PR.

So if my conclusions are incorrect, then what conclusion/s can one make from such a result? I'm interested in the conclusions of this because it allows me to make a rule to select each method. So if one method is always better in AU-ROC, then I would like to say that one should select the method if his priority is a high recall for example.

Answer 1

ROC and PR curves are primarily visual tools to select a given operating point (cutoff) of existing models for the task at hand (high recall, precision, ...). Area under either curve is a commonly used, though not very intuitive, summary statistic of the performance of models over their entire operating range.

In my opinion, AUC is useful as a scoring function in hyperparameter search, or for studies that compare the performance of different learning methods (e.g. where you don't want to examine a specific operating point). That said, AUC is far less practical to finally select a model to be used for a specific task and it is entirely irrelevant to select the model's operating point. It is perfectly possible for a suitable model for a given task to have terrible AUC under either curve.

---

Example

Suppose I want very high precision but don't need a lot of recall and I have two models $A$ and $B$ with corresponding PR curves (precision as function of recall): $$ \begin{align} PR_A(r) &= e^{-2r} \quad \rightarrow \quad \text{PR-AUC} \approx 0.43\\ PR_B(r) &= 0.80 \quad \rightarrow \quad \text{PR-AUC} = 0.80 \end{align} $$ The precision of model $A$ is higher than $B$ up to about $11\%$ recall. Its AUC is far worse, though. Given the application I presented, model $A$ is probably the best choice. Examples of such applications:

• gene prioritization: rank genes based on association with diseases; the top ranked genes are likely to become targets for (expensive) biological analyses.
• fraud detection: rank transactions based on potential of fraudulence; top ranked transactions may lead to lawsuits, false positives lead to counterclaims.

Answer 2

Err, not at all; you could at best judge this from a shape of those curves.

Anyway, when you see a big variation is which model wins it is probably due to a fact they are equivalent (i.e. none is significantly better than the other).