Ranking two models based on ROC-AUC and PR-AUC I have two methods/classifiers (completely different models) that I need to decide which one is better. The dataset is imbalanced. I trained both classifiers on the same dataset and then I computed the ROC-AUC and the Precision-Recall-AUC (PR-AUC). Then the surprise came!

*

*Method 1 is better than method 2 when I compare them using ROC-AUC.


*Method 2 is better than method 1 when I compare them using PR-AUC.
Now I'm so confused! How to say which method is better? As far as I know from this paper that if the ROC-AUC is high, then PR-AUC is also high. So if the ROC curve of method-1 dominates, so should method-1's PR curve. Is my understanding incorrect? Or am I missing something? Because I'm really going crazy!
 A: Usually you would obtain the same conclusion based on both measures. It is possible to get conflicting conclusions if the performance curves (both PR and ROC) of the models cross, e.g. one model is better at low recall while the other is better at high recall. Relying on summaries like AUC is good, but don't neglect the actual curves.
Your result implies that neither model is better than the other over the full operating range. If you still want to make a statement about which is better, you will need to be more specific about your priorities: do you want high recall, high precision, high specificity? (instead of asking which is best in any setting, e.g. the full operating range)

ROC-AUC is high, then PR-AUC is also high. 

Yes, but note that high is relative. Depending on the class balance, a PR-AUC of $20\%$ can already be excellent.

So if the ROC curve of method-1 dominates, so should method-1's PR curve. 

To quote the paper of Davis and Goadrich "a curve dominates in ROC space if and only if it dominates in PR space". This means that if you have one model A whose PR/ROC curve is entirely above another model B's PR/ROC curve, the ROC/PR curve for A will also be above that of B in the entire range.
A: ROC-AUC and PR-AUC are all AUC, confined by two axes with all thresholds.

*

*ROC-AUC:


*

*one axis is True Positive Rate (TPR), i.e., true positives / all positives

*the other axis is False Positive Rate (FPR), i.e., false positives / all negatives



*PR-AUC:


*

*one axis is Recall (which is another name for TPR)

*the other axis is Precision, i.e., true positives / (true positives + false positives)

We can see they both have TRP as one axis, their difference comes from the other axis.
Basically, PR-AUC misses the part true negatives, and focuses on true positives (both axes have true positives in the numerator, divided by different denominators).
In contrast, ROC-AUC has taken true negatives into consideration, the other axis FPR = false positives / (false positives + true negatives).
Thus, when we observe ROC-AUC increases while PR-AUC decreases, the model improves in FPR while deteriorating in Precision.
