I'm aware of some of the issues associated with using AUC for model comparison (see for example the articles referenced on Wikipedia: here, here, or here). But so far I have found nothing on an issue that at least to me is even more obvious:
The two models above have the same ROC surface so I would be unable to discriminate them based on their AUC. However, in real life I would likely go with model 2 as it offers more classification thresholds (points on the surface) and therefore allowing me to make more granular tradeoff decisions.
Of course comparing models based on on a single criterion such as AUC is probably a bad idea in general. But it seems like this particular issue could be eliminated by excluding the trivial points (0,0) and (1,1) (which bear no significance in real-life applications anyways):
For me it is much more intuitive to include only parts of the ROC surface that are attainable by thresholds that make sense.
The only immediate drawback I see is that we lose the comparability with the AUC against the random classifier (dotted line) but for me the issue of “Is my model any good?” is separate from “Which of my good models is the best?”.
So why is this not routinely done? What am I missing here?
As a more general aside: when comparing models it is quite natural to me to look at location, spread, and number of classification thresholds on the ROC curve as it will largely influence the usefulness of my model in the real world. However, I’m having trouble finding any literature or consistent approach to this. Happy about any help.
Gist to reproduce plots in R: https://gist.github.com/jakobludewig/b79678f04a6c0a0b023b8685608d1d47