8
$\begingroup$

I have two methods/classifiers (completely different models) that I need to decide which one is better. The dataset is imbalanced. I trained both classifiers on the same dataset and then I computed the ROC-AUC and the Precision-Recall-AUC (PR-AUC). Then the surprise came!

  • Method 1 is better than method 2 when I compare them using ROC-AUC.

  • Method 2 is better than method 1 when I compare them using PR-AUC.

Now I'm so confused! How to say which method is better? As far as I know from this paper that if the ROC-AUC is high, then PR-AUC is also high. So if the ROC curve of method-1 dominates, so should method-1's PR curve. Is my understanding incorrect? Or am I missing something? Because I'm really going crazy!

Improve this question
$\endgroup$
3

2 Answers 2

Reset to default
8
$\begingroup$

Usually you would obtain the same conclusion based on both measures. It is possible to get conflicting conclusions if the performance curves (both PR and ROC) of the models cross, e.g. one model is better at low recall while the other is better at high recall. Relying on summaries like AUC is good, but don't neglect the actual curves.

Your result implies that neither model is better than the other over the full operating range. If you still want to make a statement about which is better, you will need to be more specific about your priorities: do you want high recall, high precision, high specificity? (instead of asking which is best in any setting, e.g. the full operating range)

ROC-AUC is high, then PR-AUC is also high.

Yes, but note that high is relative. Depending on the class balance, a PR-AUC of $20\%$ can already be excellent.

So if the ROC curve of method-1 dominates, so should method-1's PR curve.

To quote the paper of Davis and Goadrich "a curve dominates in ROC space if and only if it dominates in PR space". This means that if you have one model A whose PR/ROC curve is entirely above another model B's PR/ROC curve, the ROC/PR curve for A will also be above that of B in the entire range.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ Note that ROC area is too insensitive a measure to be used to compare two models. It is useful for summarizing predictive discrimination for a single model, in my opinion. $\endgroup$
    – Frank Harrell
    Oct 28, 2014 at 21:53
  • $\begingroup$ @FrankHarrell this is a problem indeed. Proper score functions are better options if all models yield probabilities. What measure would you suggest to compare models which only rank instances? (e.g. predictions are in $\mathbb{R}$ rather than probabilities in $[0,1]$ -- an example of such models would be SVM classifiers). In such cases, I am unaware of better measures than ROC area and the like. $\endgroup$
    – Marc Claesen
    Oct 28, 2014 at 22:01
  • 1
    $\begingroup$ It's best if the two model outputs can be transformed to the same scale; then there are more options. Otherwise consider the "is one model concordant for a pair of observations when the other model isn't" approach of the R Hmisc package rcorrp.cens function. $\endgroup$
    – Frank Harrell
    Oct 28, 2014 at 22:04
  • 2
    $\begingroup$ Aha, so what you mean here is that if the AUC is higher that doesn't mean that the curve dominates! But the opposite should be true. i.e. if it dominates, then the AUC is higher. $\endgroup$
    – Jack Twain
    Oct 29, 2014 at 9:07
  • $\begingroup$ @MarcClaesen pleeeease check this stats.stackexchange.com/questions/123697/… $\endgroup$
    – Jack Twain
    Nov 12, 2014 at 10:05
4
$\begingroup$

ROC-AUC and PR-AUC are all AUC, confined by two axes with all thresholds.

  1. ROC-AUC:
  • one axis is True Positive Rate (TPR), i.e., true positives / all positives
  • the other axis is False Positive Rate (FPR), i.e., false positives / all negatives
  1. PR-AUC:
  • one axis is Recall (which is another name for TPR)
  • the other axis is Precision, i.e., true positives / (true positives + false positives)

We can see they both have TRP as one axis, their difference comes from the other axis. Basically, PR-AUC misses the part true negatives, and focuses on true positives (both axes have true positives in the numerator, divided by different denominators).

In contrast, ROC-AUC has taken true negatives into consideration, the other axis FPR = false positives / (false positives + true negatives).

Thus, when we observe ROC-AUC increases while PR-AUC decreases, the model improves in FPR while deteriorating in Precision.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.