2
$\begingroup$

can you tell me if this is ok?

While the AUC (i.e. AUC of 0.6) we got is acceptable since it's bigger than 0.5, we may need to re-evaluate at our cutoff selections again. Because we can select cutoffs by our choice, sensitivity and specificity would change if we switch a cutoff from 0.7 to 0.68, for example. Then, we may have to re-draw a ROC again if we set new cutoffs. For new cutoffs, sensitivity and specificity are going to be different and we may find the new AUC more acceptable. We may end up finding the existing cutoffs that we used were not that great.

$\endgroup$
1
  • $\begingroup$ This is a XY problem. You already have a procedure you are not confident will answer your problem, and, instead of asking about the problem directly, you are asking if your procedure is correct. It is often easier to answer the root question than explaining why something is wrong (as can be seen by the long comment chain). $\endgroup$
    – Firebug
    Sep 15, 2023 at 9:15

3 Answers 3

3
$\begingroup$

Your model is giving you values on a continuum, perhaps the interval $[0,1]$. The ROC curve is created by setting a threshold, calculating the sensitivity and specificity according to classifications based on if the prediction is above or below that threshold, and then doing it again for another threshold. Therefore, the ROC curve depends on those original predictions, not on the classifications made according to any particular threshold, so you have the same ROC curve no matter what threshold you select.

$\endgroup$
17
  • $\begingroup$ thanks, Dave. So, what you are saying is y^ is the one that makes the change to sensitivity and specificity. But, setting a different cutoff will make a change to sensitivity and specifciity. That is what I am talking about. Isn't mine good? $\endgroup$
    – Shawn Kim
    Sep 15, 2023 at 0:52
  • 1
    $\begingroup$ @ShawnKim I’m struggling to understand what you’re saying. Could you please rephrase or clarify? $\endgroup$
    – Dave
    Sep 15, 2023 at 0:55
  • $\begingroup$ You can pick a cutoff how-ever you want. If you set 10 cutoffs that are really high, then your specificity will be high and sensitivity will be low. Then, your AUC will be low. What I am saying is we can re-evaluate 10 cutoffs so that we can adjust the cutoffs to be lower. In that way, we can have higher sensitivities and lower specificities. Then, our adjusted AUC will be higher. $\endgroup$
    – Shawn Kim
    Sep 15, 2023 at 0:57
  • 1
    $\begingroup$ You have one ROC curve that gives many possible pairings of sensitivity and specificity, depending on the threshold. $\endgroup$
    – Dave
    Sep 15, 2023 at 1:02
  • $\begingroup$ yup. So, we may have only 5 cutoffs and we cannot get a clear picture of the true AUC. We cannot really cross validate to get the optimal cutoffs. So, we may want to choose other 10 cutoffs then we can hopefully have a better and more reliable AUC. Does my thought process make sense? thx $\endgroup$
    – Shawn Kim
    Sep 15, 2023 at 1:04
3
$\begingroup$

Brief answer: the AUC is threshold agnostic, it's one of its main selling points. Therefore, it makes no sense to talk about the impact of changing thresholds and AUC, since AUC does not use that information, and is instead derived from the continuous decision scores or probabilities.

$\endgroup$
3
$\begingroup$

ROC, sensitivity, and specificity have nothing to do with decision making as they are all in reverse time and information-flow order. Optimum decisions involve forward predictions (e.g., predicted probability of an event) and knowledge about consequences of those decisions (utility function). See this for more. Never dichotomize information at the front end. You might possibly threshold a predicted probability (at the back end) but even then the use of dichotomization can be misleading.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.