2
$\begingroup$

I'm aware of some of the issues associated with using AUC for model comparison (see for example the articles referenced on Wikipedia: here, here, or here). But so far I have found nothing on an issue that at least to me is even more obvious:

AUC of two models

The two models above have the same ROC surface so I would be unable to discriminate them based on their AUC. However, in real life I would likely go with model 2 as it offers more classification thresholds (points on the surface) and therefore allowing me to make more granular tradeoff decisions.

Of course comparing models based on on a single criterion such as AUC is probably a bad idea in general. But it seems like this particular issue could be eliminated by excluding the trivial points (0,0) and (1,1) (which bear no significance in real-life applications anyways): cutting of trivial points forAUC

For me it is much more intuitive to include only parts of the ROC surface that are attainable by thresholds that make sense.

The only immediate drawback I see is that we lose the comparability with the AUC against the random classifier (dotted line) but for me the issue of “Is my model any good?” is separate from “Which of my good models is the best?”.

So why is this not routinely done? What am I missing here?

As a more general aside: when comparing models it is quite natural to me to look at location, spread, and number of classification thresholds on the ROC curve as it will largely influence the usefulness of my model in the real world. However, I’m having trouble finding any literature or consistent approach to this. Happy about any help.

Gist to reproduce plots in R: https://gist.github.com/jakobludewig/b79678f04a6c0a0b023b8685608d1d47

$\endgroup$
  • $\begingroup$ You answer your own question: you would immediately lose the ability to compare two models. I'm not quite sure what you're really asking here? $\endgroup$ – Calimo Oct 23 '18 at 17:22
  • $\begingroup$ As far as I can see I only lose comparability of the models against a random baseline model. But when choosing between two actual models I should have convinced myself that both of them they sufficiently outperform random guessing (ie modelling even makes sense). At this point the only reason that the two models above have the same AUC is the inclusion of the points (0,0) and (1,1) in the AUC calculation, none of which I would choose as a classification threshold. So why would I use these points when calculating the AUC? $\endgroup$ – jludewig Oct 24 '18 at 12:37
  • $\begingroup$ I think your mistake is to think that model 2 is better than 1 because it offers more classification thresholds. This is not relevant in any application I know. Ultimately you want to make a decision, so you will choose one threshold, probably the point at (0.5 FPR, 0.75 TPR) which offers the highest accuracy. Then you realize that both models actually perform exactly the same. $\endgroup$ – Calimo Nov 3 '18 at 8:44
2
$\begingroup$

Model_2 does not offer more classification thresholds. That's the number of instances in your test set.

You come up with the thresholds by ranking the predicted probabilities of your model (and ordering by confidence).

In practice you should have enough data points in your test set that "not enough thresholds to pick from" is not an issue.

The (0|0) and (1|1) points in the graph are two options you always have: e.g: always predicting one of two classes in a binary classification setting. If you end up considering (0|0) or (1|1), then maybe learning itself is not feasible (e.g: no relevant features, unpredictable events ...).

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks, I'm not sure I understand your first point though. Let's say I fit a logistic regression model with only one binary predictor. Then clearly my model will only predict two different probabilities for all cases it encounters, regardless of how many observations there are in my dataset. This may be less obvious in the case of continuous predictors but in essence the number of thresholds will still be driven by the amount of levels you encounter in your predictors (which may not go up by simply having more observations in my dataset). $\endgroup$ – jludewig Oct 18 '18 at 7:13
  • $\begingroup$ Regarding your last comment: I agree that it may be worthwhile to include these points when making the decision whether or not to apply ML to a given problem at all. But when you are at a stage where you decide which out of two (or more) models to implement this decision should already have been made. In my opinion at this point the inclusion of these points becomes irrelevant and potentially confusing. $\endgroup$ – jludewig Oct 18 '18 at 7:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.