4
$\begingroup$

https://queirozf.com/entries/introduction-to-auc-and-calibrated-models-with-examples-using-scikit-learn

I was reading this post on AUC and calibration and found one of the conclusions pretty interesting.

You want to create stratified groups depending on output scores If your model outputs credit default risk scores, one thing you may be asked to do is to group those clients into ratings. For example, you would want to assign credit rating "A" to clients on bottom 10% of default risk, "B" to clients having 10%-20% risk, and so on, until "H". In other words, if you need to get the order of your scores right, AUC isn't a good metric to help you with that (because it measures discrimination, not calibration).

I don't quite understand the reason behind that statement. Having a high AUC score means that the algorithm is good at ranking objects then why can't we use the outputs for, well, ranking.

For example, we trained a model using binary logloss as the loss function and AUC as the evaluation metric. Once we are satisfied with the results, we can sort the objects by the predicted scores and then create the so-called groups (for simplicity, three groups with 33% of items each).

What is wrong with that approach? If the article is right then what is recommended for cases when you need to create the stratified groups as described in the article?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

that part is not written very clearly. Yes, if you just want to rank people and then make your buckets based on this ranking, then AUC will work fine. If you don't want to rank people, but take the model's estimated probability of a default as a given and make your buckets based on that, then AUC is not good. Since a model could predict the probability of default to be less than 50% to 90% of people, but AUC doesn't care about that.

In the default example, you probably care about the probability more than about the ranking. Having a probability of a default of 10% is not the same as being in the top 10% of the people that are least likely to default. If I know I make money if at least 90% of people pay me back, I don't want to lend money to the top 10% or the top 90% of people most likely to pay me back, I want to lend money to anybody who has at least 90% chance to pay me back, which can be 1%, 50%, or 90% of people, doesn't matter.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ Exactly. If I understood your point correctly, it's okay to use AUC when just the order matters and not so when the absolute values have some meaning to the user. In the latter case, it seems that using the Brier score loss might be a better choice. In my use case, however, what is required is to create a few buckets based on the predicted scores. It doesn't really matter whether most scores are below 0.5 or not, it's just a matter of ordering them well enough. $\endgroup$
    – Don Draper
    May 15, 2021 at 19:12
  • 2
    $\begingroup$ yes, exactly. One thing to consider is that if you really care about some buckets, then it is good to have a measure to check how well the buckets themself perform, not only the scores that were used to create the buckets. If I can show you only 10 search results, I might not want to look at AUC of everything, but to use some measure that looks at how good am I at predicting the top 10 only etc. $\endgroup$
    – rep_ho
    May 15, 2021 at 19:22
  • $\begingroup$ Another question (probably, this is a candidate for another thread) is whether there is a way to somehow account for the highly imbalanced dataset. When most objects belong to the negative class (which is quite common in most business applications), it doesn't come as a surprise that the bottom bucket has nearly zero FPs, which is great when the number of FPs has a higher priority, but that might as well be achieved by always predicting the negative class. My approach for such tasks, in general, is analyzing cumulative gains curve and then selecting top N% of scores based on that. $\endgroup$
    – Don Draper
    May 15, 2021 at 19:27
  • $\begingroup$ balanced accuracy and AUC don't have a problem with that, but you can have high sensitivity and specificity but horrible positive predictive value or negative predictive value, which balanced accuracy or AUC won't show. $\endgroup$
    – rep_ho
    May 15, 2021 at 19:31
  • $\begingroup$ @DonDraper You're dealing with ranking the predicted probability of default. When you work with probabilities, there are no "positives", just probabilities. $\endgroup$
    – Dave
    Nov 8, 2023 at 0:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.