1
$\begingroup$

I got following chart: enter image description here

The algorithms have been applied to a dataset where an outcome is pretty rare, it happens 10% of the times (binary, 0- 90%, 1-10%). It is the response whether a client is going to default or not.

I struggle to believe its values. We see an increase in sensitivity, which makes totally sense to me and the accuracy being constant high (since the outcome is rare) throughout the algorithms.

But shouldn't the AUROC be correlated to the sensitivity, since it takes the hit and false alarm rate? What would we expect for the accuracy ratio (Gini)?

In credit risk, AUROC is one of the most important benchmarks. I try to figure out why. Looking at this table sensitivity would make much more sense to me.

Improve this question
$\endgroup$
7
  • $\begingroup$ Sensitivity is an improper scoring rule that does not consider the probability values from the model, just the side of some threshold on which that probability falls. AUROC considers those probability values. $\endgroup$
    – Dave
    Commented Apr 22, 2021 at 11:14
  • $\begingroup$ @Dave true, it just considers a cut-off value. But shouldn't the goal be to catch most defaulters? Also hit and false alarm rate should be connected to the sensitivity? Does that mean, that the specific cut-off value for the sensitivity has been poorly chosen? $\endgroup$
    – Romero Azzalini
    Commented Apr 22, 2021 at 11:22
  • $\begingroup$ Maybe the threshold is a poor one. Maybe there shouldn’t be a threshold at all and users should base interest rates on default probabilities. After all, credit scores are on a continuum (more or less), not just good credit/bad credit. Thresholding too early wrecks the ability to do this. Stephan Kolassa and Frank Harrell both have written about this on here; Harrell also has blog posts: fharrell.com/post/class-damage and fharrell.com/post/classification. // A standard software default is to threshold at $0.5$. I think that a $40\%$ chance of default is pretty high! $\endgroup$
    – Dave
    Commented Apr 22, 2021 at 11:27
  • $\begingroup$ Very relevant: Classification probability threshold. Do not use accuracy to evaluate a classifier: Why is accuracy not the best measure for assessing classification models? Is accuracy an improper scoring rule in a binary classification setting? The same problems apply to sensitivity and specificity, and indeed to all evaluation metrics that rely on hard classifications. $\endgroup$
    – Stephan Kolassa
    Commented Apr 22, 2021 at 11:36
  • $\begingroup$ ... The same problems apply to sensitivity and specificity, and indeed to all evaluation metrics that rely on hard classifications. Instead, use probabilistic classifications, and evaluate these using proper scoring rules. $\endgroup$
    – Stephan Kolassa
    Commented Apr 22, 2021 at 11:36

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.