Is the AUC an incoherent measure of classifier performance?

Question

I'm learning about performance measures for binary classifiers. Reading about the AUC-ROC score I came across the article Measuring classifier performance: a coherent alternative to the area under the ROC curve, Hand (2009). The author claims that:

..the AUC is equivalent to averaging the misclassification loss over a cost ratio distribution which depends on the score distributions. Since the score distributions depend on the classifier, this means that, when evaluating classifier performance, the AUC evaluates a classifier using a metric which depends on the classifier itself. That is, the AUC evaluates different classifiers using different metrics. It is in that sense that the AUC is an incoherent measure of classifier performance.

and furthermore:

..this is effectively what the AUC does—it evaluates different classifiers using different metrics. It is as if one measured person A's height using a ruler calibrated in inches and person B’s using one calibrated in centimetres, and decided who was the taller by merely comparing the numbers, ignoring the fact that different units of measurement had been used

(emphasis added)

Given that the usage of the AUC-ROC score is pretty widespread, this seems like a bold claim. If this is true, then using AUC-ROC to compare the performance of different classifiers is completely wrong. The author proposes a new (better?) performance metric called the $H$ measure. Unfortunately I can't entirely follow the maths involved in the article.

Is this author correct? Should we ditch the AUC-ROC completely in favour of this $H$ measure?

---

Add

Just realized there's even an R implementation of this measure: https://cran.r-project.org/web/packages/hmeasure/index.html

REFERENCE

Hand, David J. "Measuring classifier performance: a coherent alternative to the area under the ROC curve." Machine learning 77.1 (2009): 103-123.

Answer 1

Seeing those quotes out of context, they seem extreme.

If there is a clear boundary between the predictions, then the model has done a good job of distinguishing between the two categories. If there is a clear area where predictions correspond to one category and another where predictions correspond to another category, but there is an ambiguous area between them, then the model has done less of a fantastic job of separating the categories. If the predictions corresponding to the categories are basically on top of each other, then the model does a poor job of distinguishing between the categories.

This is exactly what ROC AUC measures, the extent to which the predictions are separated by group.

Look at how good of a job it does at describing the separation in the plot below.

ROC AUC is not without its flaws, such as those discussed by Frank Harrell, but it seems clear to me what the AUC measures and that it increases as something desirable happens.

# To reproduce the KDE plots with the ROC AUC

library(ggplot2)
library(pROC)
set.seed(2023)
N <- 1000
p1 <- c(runif(N, 0, 0.35), runif(N, 0.65, 1))
y1 <- c(rep(0, N), rep(1, N))
p2 <- rbeta(2*N, 1/2, 1/2)
y2 <- rbinom(2*N, 1, p2)
p3 <- rbeta(2*N, 1, 1)
y3 <- rbinom(2*N, 1, 0.5)
d1 <- data.frame(
  Category = as.factor(y1),
  Predictions = p1,
  group = paste("AUC =", round(pROC::roc(y1, p1)$auc, 3))
)
d2 <- data.frame(
  Category = as.factor(y2),
  Predictions = p2,
  group = paste("AUC =", round(pROC::roc(y2, p2)$auc, 3))
)
d3 <- data.frame(
  Category = as.factor(y3),
  Predictions = p3,
  group = paste("AUC =", round(pROC::roc(y3, p3)$auc, 3))
)
d <- rbind(d1, d2, d3)
ggplot(d, aes(x = Predictions, fill = Category)) +
  geom_density(alpha = 0.25) +
  facet_grid(rows = vars(group)) +
  theme(legend.position="bottom")