Effective validity of AUROC as performance measure: what about "very high" AUROC values?

The Area Under ROC curve (AUROC) is a quantity used to quantify performance of classifiers. I am currently interested in the most basic drawbacks of using AUROC as unique performance measure.

I got a negative feedback with AUROC in some recent analysis as I met a classification problem performed through a logistic regression in presence of complete separation; I presume that in this case one simply obtains a perfect AUROC=1. Is this correct?

In general, extremely high values of AUROC (I mean values $\geq 0.998$ ) would make me wonder whether there is something "wrong" or "artifical" with the model and/or data, as first feeling.

It would be nice if you could confirm this feeling of mine and support it with examples. At this stage, the opposite thesis is also interesting to me.

• It usually makes sense to calculate AUC "out-of-sample," e.g. through cross-valiation.
– Zach
Jul 11, 2013 at 15:10
• Thanks Zach: without cross-validation surely something can go wrong . Good point Jul 11, 2013 at 19:30
• I would say if you have a high (or perfect) AUC in-sample, but a very low AUC out of sample, something is probably wrong. If they're both high, you just have an easy problem to solve.
– Zach
Jul 11, 2013 at 19:59

The use of ROC curves may be misleading when your data is strongly imbalanced. In such cases, precision-recall curves and their AUC is often a better choice.

Particularly, the area under the PR curve may differ quite a lot between classifiers with comparable AUROC, even when the AUROC is very high.

This paper by Davis et al. is an excellent reference on the subject. Interestingly, they have shown that a curve dominates in ROC space if and only if it also dominates in PR space. This is a very useful practical result.

• ROC curves are hardly misleading in the imbalanced context, at least if you know what you are doing. ROC curves evaluate the overall discriminative power of a model, i.e. how well the produced score can separate negative and positive instances. This is independent of class imbalance. Apr 15, 2019 at 12:43

I recently did this kind of analysis with CRM data and logistic regression, ANN and recursive partioning regression trees. Software was R.

I used R package ROCR to discriminate between results obtained via these different classifiers.

But besides AUC I also used various other measures as error rate, accuracy and phi correlation between observed and predicted outcomes with different probability cut-offs.