# Is the AUC an incoherent measure of classifier performance?

I'm learning about performance measures for binary classifiers. Reading about the AUC-ROC score I came across the article Measuring classifier performance: a coherent alternative to the area under the ROC curve, Hand (2009). The author claims that:

..the AUC is equivalent to averaging the misclassification loss over a cost ratio distribution which depends on the score distributions. Since the score distributions depend on the classifier, this means that, when evaluating classifier performance, the AUC evaluates a classifier using a metric which depends on the classifier itself. That is, the AUC evaluates different classifiers using different metrics. It is in that sense that the AUC is an incoherent measure of classifier performance.

and furthermore:

..this is effectively what the AUC does—it evaluates different classifiers using different metrics. It is as if one measured person A's height using a ruler calibrated in inches and person B’s using one calibrated in centimetres, and decided who was the taller by merely comparing the numbers, ignoring the fact that different units of measurement had been used

Given that the usage of the AUC-ROC score is pretty widespread, this seems like a bold claim. If this is true, then using AUC-ROC to compare the performance of different classifiers is completely wrong. The author proposes a new (better?) performance metric called the $$H$$ measure. Unfortunately I can't entirely follow the maths involved in the article.

Is this author correct? Should we ditch the AUC-ROC completely in favour of this $$H$$ measure?

Just realized there's even an R implementation of this measure: https://cran.r-project.org/web/packages/hmeasure/index.html

REFERENCE

Hand, David J. "Measuring classifier performance: a coherent alternative to the area under the ROC curve." Machine learning 77.1 (2009): 103-123.

• "Given that the usage of the AUC-ROC score is pretty widespread, this seems like a bold claim." Well, by far the majority of people have no idea what p-values are or how to interpret them, but they form the basis of probably millions of papers. Jun 23, 2020 at 15:10
• Check out what Frank Harrell says about proper scoring rules and the use of AUC. The gist of his thoughts on AUC is that it is useful to see if a classifier is performing decently (e.g. "Hey, AUC=0.8...we must be pretty good" or "Yuck, AUC=0.55...back to work!") but is not to be used for comparing classifiers. Perhaps the H measure is a proper scoring rule, though there are easier ones than what I see in Hand's papers, such as Brier score and log loss (cross-entropy).
– Dave
Jun 23, 2020 at 15:14
• Here are a couple of links. I can't remember where Harrell said that AUC was fine for assessing one model but not comparing models, though. fharrell.com/post/class-damage stats.stackexchange.com/questions/339919/… stats.stackexchange.com/questions/464636/… But it does appear that Harrell is right that the machine learning world often makes a mistake in using threshold-based (or other improper) scoring rules.
– Dave
Jun 23, 2020 at 15:25
• Regarding the "widespread" part: accuracy is probably the most widespread classifier evaluation measure, which doesn't stop it from being crappy. Jun 23, 2020 at 15:25
• Here is a sampling of Frank Harrell on (AU)ROC. Jun 23, 2020 at 15:31

Seeing those quotes out of context, they seem extreme.

If there is a clear boundary between the predictions, then the model has done a good job of distinguishing between the two categories. If there is a clear area where predictions correspond to one category and another where predictions correspond to another category, but there is an ambiguous area between them, then the model has done less of a fantastic job of separating the categories. If the predictions corresponding to the categories are basically on top of each other, then the model does a poor job of distinguishing between the categories.

This is exactly what ROC AUC measures, the extent to which the predictions are separated by group.

Look at how good of a job it does at describing the separation in the plot below. ROC AUC is not without its flaws, such as those discussed by Frank Harrell, but it seems clear to me what the AUC measures and that it increases as something desirable happens.

# To reproduce the KDE plots with the ROC AUC

library(ggplot2)
library(pROC)
set.seed(2023)
N <- 1000
p1 <- c(runif(N, 0, 0.35), runif(N, 0.65, 1))
y1 <- c(rep(0, N), rep(1, N))
p2 <- rbeta(2*N, 1/2, 1/2)
y2 <- rbinom(2*N, 1, p2)
p3 <- rbeta(2*N, 1, 1)
y3 <- rbinom(2*N, 1, 0.5)
d1 <- data.frame(
Category = as.factor(y1),
Predictions = p1,
group = paste("AUC =", round(pROC::roc(y1, p1)$$auc, 3)) ) d2 <- data.frame( Category = as.factor(y2), Predictions = p2, group = paste("AUC =", round(pROC::roc(y2, p2)$$auc, 3))
)
d3 <- data.frame(
Category = as.factor(y3),
Predictions = p3,
group = paste("AUC =", round(pROC::roc(y3, p3)\$auc, 3))
)
d <- rbind(d1, d2, d3)
ggplot(d, aes(x = Predictions, fill = Category)) +
geom_density(alpha = 0.25) +
facet_grid(rows = vars(group)) +
theme(legend.position="bottom")

• See fharrell.com/post/addvalue for measures that are more sensitive and easier to interpret. The gist of the serious problem in comparing AUROC is that it is equivalent to comparing two Wilcoxon rank statistics which is a no-no. E.g. you don't compare a Wilcoxon statistic comparing groups A and B with one comparing A and C to get a comparison of B vs C. Sep 3 at 12:03
• @FrankHarrell It seems like it would be a comparison of A vs B and A' vs B'. The models will (probably) give different predictions for each group, not just for one of the groups.
– Dave
Sep 5 at 8:17
• The analogy I was using was a comparison of 3 treatments where one tried to learn about B-C by subtracting two comparisons instead of doing the obvious head-to-head comparison. When comparing prediction models AUROC doesn't do the right head-to-head comparison. Contrast that with these methods. Sep 5 at 11:43