0
$\begingroup$

I know AUC is supposed to be independent on the threshold, which means AUC does not change while the threshold changes. However, I'm getting different AUC values while changing the thresholds. I'm using roc_auc_score to calculate the AUC value in Python.

I've got the probability and assigned the group as 1 if the probability is greater than 0.5 and 0 if the probability is less than 0.5. I assume 0.5 is the threshold here? And then I wanted to try with 0.4, 0.3, and so on. The AUC for each case is not supposed to change, but mine was different everytime I used a different threshold.

Is there any idea of why I'm getting different values?

Improve this question
$\endgroup$
5
  • $\begingroup$ towardsdatascience.com/…. $\endgroup$
    – user10386405
    Commented Nov 22, 2021 at 9:44
  • 1
    $\begingroup$ "AUC is not dependent on classification threshold value. Changing the threshold value does not change AUC because it is an aggregate measure of ROC." said in the above article. So I'm confused. $\endgroup$
    – user10386405
    Commented Nov 22, 2021 at 9:45
  • 1
    $\begingroup$ AUROC is indeed independent of the threshold values as it is an aggregate of all the threshold, so in what way are you changing the thresholds? $\endgroup$
    – user2974951
    Commented Nov 22, 2021 at 9:55
  • $\begingroup$ I just edited my question. You can see the added part. $\endgroup$
    – user10386405
    Commented Nov 22, 2021 at 10:00
  • 1
    $\begingroup$ You are not supposed to threshold the probabilities (perform classification) and then calculate the AUC, you should pass the non-thresholded probabilities as is, the function will try the various different thresholds. $\endgroup$
    – user2974951
    Commented Nov 22, 2021 at 10:03

2 Answers 2

Reset to default
6
$\begingroup$

I think you might be doing the following: defining a threshold (k), calculating your output accordingly (y_k = 1 if p>k, 0 otherwise ), then calculating AUC from y_k and y_true. This is not the AUC for your initial model, but for a different model which doesn't see probabilities and only sees 1 or 0 depending on the threshold (and obviously depend on threshold). The right way would be to calculate AUC from probabilities and y_true, in which the function will go through all the thresholds between 0 and 1 and aggregate the results (and won't depend on a threshold)

Improve this answer
$\endgroup$
3
  • $\begingroup$ OK. That's exactly what I'm doing since I have to compare the sensitivity/specificity and stuff for different thresholds. So do you mean I could just use the AUC from 'calculate AUC from probabilities and y_true' and that's all? $\endgroup$
    – user10386405
    Commented Nov 22, 2021 at 10:15
  • 2
    $\begingroup$ Yes, to compute AUC for a model, you should use y_estimated = probabilities from your model and y_true. Often, after calculating AUC, researches derive a threshold which maximizes sensitivity + specificity and report sensitivity and specificity at that point, it is called Youden index sciencedirect.com/topics/medicine-and-dentistry/youden-index. $\endgroup$
    – DianaS
    Commented Nov 22, 2021 at 10:21
  • 1
    $\begingroup$ +1. To be more precise, anything calculated per the first sentence will simply not be the AUC, but some other KPI, probably one of these. There is nothing wrong with using a custom KPI, but one should definitely not label it with a name that refers to something else in the established literature. $\endgroup$
    – Stephan Kolassa
    Commented Nov 22, 2021 at 10:33
3
$\begingroup$

AUC is calculated by varying the threshold across the permissible space, e.g., $[0,1]$. This varying threshold gives you precision-recall curve. You then calculate the Area Under this Curve.

Thus, it makes little sense to say that "AUC is independent of the threshold". The threshold is integrated out. Equally, you would not say that "the expectation of a random variable is independent of the realization", because in calculating the expectation, you integrate over the realizations.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I don't really get the word "integrated out". Can you elaborate? I was reading this article where it's said that "AUC is independent of the threshold". towardsdatascience.com/…. $\endgroup$
    – user10386405
    Commented Nov 22, 2021 at 10:04
  • $\begingroup$ You integrate over the thresholds to obtain the area under the curve. This is standard calculus. $\endgroup$
    – Stephan Kolassa
    Commented Nov 22, 2021 at 10:09
  • $\begingroup$ Ok. So I don't have to calculate AUC for each case. I can just 'calculate AUC from probabilities and y_true using roc_auc_score? $\endgroup$
    – user10386405
    Commented Nov 22, 2021 at 10:22
  • $\begingroup$ Yes indeed. (Or so I assume, not knowing roc_auc_score.) $\endgroup$
    – Stephan Kolassa
    Commented Nov 22, 2021 at 10:34
  • 1
    $\begingroup$ Think of AUROC as the $c$-index (concordance probability; WIlcoxon statistic, Somers' $D_xy$) that is not needing to be derived from an ROC curve at all. Then you'll see that thresholds have nothing to do with it. $\endgroup$
    – Frank Harrell
    Commented Nov 22, 2021 at 13:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.