I'm learning about performance measures for binary classifiers. Reading about the AUC-ROC score I came across the article Measuring classifier performance: a coherent alternative to the area under the ROC curve, Hand (2009). The author claims that:
..the AUC is equivalent to averaging the misclassification loss over a cost ratio distribution which depends on the score distributions. Since the score distributions depend on the classifier, this means that, when evaluating classifier performance, the AUC evaluates a classifier using a metric which depends on the classifier itself. That is, the AUC evaluates different classifiers using different metrics. It is in that sense that the AUC is an incoherent measure of classifier performance.
..this is effectively what the AUC does—it evaluates different classifiers using different metrics. It is as if one measured person A's height using a ruler calibrated in inches and person B’s using one calibrated in centimetres, and decided who was the taller by merely comparing the numbers, ignoring the fact that different units of measurement had been used
Given that the usage of the AUC-ROC score is pretty widespread, this seems like a bold claim. If this is true, then using AUC-ROC to compare the performance of different classifiers is completely wrong. The author proposes a new (better?) performance metric called the $H$ measure. Unfortunately I can't entirely follow the maths involved in the article.
Is this author correct? Should we ditch the AUC-ROC completely in favour of this $H$ measure?
Just realized there's even an R implementation of this measure: https://cran.r-project.org/web/packages/hmeasure/index.html
Hand, David J. "Measuring classifier performance: a coherent alternative to the area under the ROC curve." Machine learning 77.1 (2009): 103-123.