0
$\begingroup$

I know AUC is supposed to be independent on the threshold, which means AUC does not change while the threshold changes. However, I'm getting different AUC values while changing the thresholds. I'm using roc_auc_score to calculate the AUC value in Python.

I've got the probability and assigned the group as 1 if the probability is greater than 0.5 and 0 if the probability is less than 0.5. I assume 0.5 is the threshold here? And then I wanted to try with 0.4, 0.3, and so on. The AUC for each case is not supposed to change, but mine was different everytime I used a different threshold.

Is there any idea of why I'm getting different values?

$\endgroup$
5
  • $\begingroup$ towardsdatascience.com/…. $\endgroup$ Nov 22 '21 at 9:44
  • 1
    $\begingroup$ "AUC is not dependent on classification threshold value. Changing the threshold value does not change AUC because it is an aggregate measure of ROC." said in the above article. So I'm confused. $\endgroup$ Nov 22 '21 at 9:45
  • 1
    $\begingroup$ AUROC is indeed independent of the threshold values as it is an aggregate of all the threshold, so in what way are you changing the thresholds? $\endgroup$ Nov 22 '21 at 9:55
  • $\begingroup$ I just edited my question. You can see the added part. $\endgroup$ Nov 22 '21 at 10:00
  • 1
    $\begingroup$ You are not supposed to threshold the probabilities (perform classification) and then calculate the AUC, you should pass the non-thresholded probabilities as is, the function will try the various different thresholds. $\endgroup$ Nov 22 '21 at 10:03
3
$\begingroup$

I think you might be doing the following: defining a threshold (k), calculating your output accordingly (y_k = 1 if p>k, 0 otherwise ), then calculating AUC from y_k and y_true. This is not the AUC for your initial model, but for a different model which doesn't see probabilities and only sees 1 or 0 depending on the threshold (and obviously depend on threshold). The right way would be to calculate AUC from probabilities and y_true, in which the function will go through all the thresholds between 0 and 1 and aggregate the results (and won't depend on a threshold)

$\endgroup$
3
  • $\begingroup$ OK. That's exactly what I'm doing since I have to compare the sensitivity/specificity and stuff for different thresholds. So do you mean I could just use the AUC from 'calculate AUC from probabilities and y_true' and that's all? $\endgroup$ Nov 22 '21 at 10:15
  • 1
    $\begingroup$ Yes, to compute AUC for a model, you should use y_estimated = probabilities from your model and y_true. Often, after calculating AUC, researches derive a threshold which maximizes sensitivity + specificity and report sensitivity and specificity at that point, it is called Youden index sciencedirect.com/topics/medicine-and-dentistry/youden-index. $\endgroup$
    – DianaS
    Nov 22 '21 at 10:21
  • 1
    $\begingroup$ +1. To be more precise, anything calculated per the first sentence will simply not be the AUC, but some other KPI, probably one of these. There is nothing wrong with using a custom KPI, but one should definitely not label it with a name that refers to something else in the established literature. $\endgroup$ Nov 22 '21 at 10:33
2
$\begingroup$

AUC is calculated by varying the threshold across the permissible space, e.g., $[0,1]$. This varying threshold gives you precision-recall curve. You then calculate the Area Under this Curve.

Thus, it makes little sense to say that "AUC is independent of the threshold". The threshold is integrated out. Equally, you would not say that "the expectation of a random variable is independent of the realization", because in calculating the expectation, you integrate over the realizations.

$\endgroup$
5
  • $\begingroup$ I don't really get the word "integrated out". Can you elaborate? I was reading this article where it's said that "AUC is independent of the threshold". towardsdatascience.com/…. $\endgroup$ Nov 22 '21 at 10:04
  • $\begingroup$ You integrate over the thresholds to obtain the area under the curve. This is standard calculus. $\endgroup$ Nov 22 '21 at 10:09
  • $\begingroup$ Ok. So I don't have to calculate AUC for each case. I can just 'calculate AUC from probabilities and y_true using roc_auc_score? $\endgroup$ Nov 22 '21 at 10:22
  • $\begingroup$ Yes indeed. (Or so I assume, not knowing roc_auc_score.) $\endgroup$ Nov 22 '21 at 10:34
  • 1
    $\begingroup$ Think of AUROC as the $c$-index (concordance probability; WIlcoxon statistic, Somers' $D_xy$) that is not needing to be derived from an ROC curve at all. Then you'll see that thresholds have nothing to do with it. $\endgroup$ Nov 22 '21 at 13:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.