Ranking two models based on ROC-AUC and PR-AUC

I have two methods/classifiers (completely different models) that I need to decide which one is better. The dataset is imbalanced. I trained both classifiers on the same dataset and then I computed the ROC-AUC and the Precision-Recall-AUC (PR-AUC). Then the surprise came!

• Method 1 is better than method 2 when I compare them using ROC-AUC.

• Method 2 is better than method 1 when I compare them using PR-AUC.

Now I'm so confused! How to say which method is better? As far as I know from this paper that if the ROC-AUC is high, then PR-AUC is also high. So if the ROC curve of method-1 dominates, so should method-1's PR curve. Is my understanding incorrect? Or am I missing something? Because I'm really going crazy!

Usually you would obtain the same conclusion based on both measures. It is possible to get conflicting conclusions if the performance curves (both PR and ROC) of the models cross, e.g. one model is better at low recall while the other is better at high recall. Relying on summaries like AUC is good, but don't neglect the actual curves.

Your result implies that neither model is better than the other over the full operating range. If you still want to make a statement about which is better, you will need to be more specific about your priorities: do you want high recall, high precision, high specificity? (instead of asking which is best in any setting, e.g. the full operating range)

ROC-AUC is high, then PR-AUC is also high.

Yes, but note that high is relative. Depending on the class balance, a PR-AUC of $20\%$ can already be excellent.

So if the ROC curve of method-1 dominates, so should method-1's PR curve.

To quote the paper of Davis and Goadrich "a curve dominates in ROC space if and only if it dominates in PR space". This means that if you have one model A whose PR/ROC curve is entirely above another model B's PR/ROC curve, the ROC/PR curve for A will also be above that of B in the entire range.

• Note that ROC area is too insensitive a measure to be used to compare two models. It is useful for summarizing predictive discrimination for a single model, in my opinion. – Frank Harrell Oct 28 '14 at 21:53
• @FrankHarrell this is a problem indeed. Proper score functions are better options if all models yield probabilities. What measure would you suggest to compare models which only rank instances? (e.g. predictions are in $\mathbb{R}$ rather than probabilities in $[0,1]$ -- an example of such models would be SVM classifiers). In such cases, I am unaware of better measures than ROC area and the like. – Marc Claesen Oct 28 '14 at 22:01
• It's best if the two model outputs can be transformed to the same scale; then there are more options. Otherwise consider the "is one model concordant for a pair of observations when the other model isn't" approach of the R Hmisc package rcorrp.cens function. – Frank Harrell Oct 28 '14 at 22:04
• Aha, so what you mean here is that if the AUC is higher that doesn't mean that the curve dominates! But the opposite should be true. i.e. if it dominates, then the AUC is higher. – Jack Twain Oct 29 '14 at 9:07
• @MarcClaesen pleeeease check this stats.stackexchange.com/questions/123697/… – Jack Twain Nov 12 '14 at 10:05