0
$\begingroup$

Edit to explain how this is different from the suggested duplicate: Reduce Classification Probability Threshold

My question relates to the same topic, but is thoroughly different, so I'm surprised this was flagged as a duplicate. I want to know, specifically, how I go from looking at an ROC curve and identifying the place on the curve that I'm happy with (after considering my specific use case and all the unique ramifications of this choice), to then:

a) Figure out what threshold value $Z$ generated that point on the curve

b) If possible, change my model parameters to now label all $\hat{p}$ >= $Z$ as positive responders. (I.e, is there a "threshold" parameter in, say, scikit-learn with a default value of 0.5, but which I can change to $Z$, or is that final classification step always something that I have to do manually downstream using output probabilities?)

ORIGINAL QUESTION:

I'm well acquainted with the ROC curve and what it represents. I'm wondering what the next practical steps are in selecting a decision threshold for your model once you've plotted an ROC curve. The axes of the curve don't indicate decision thresholds. So you can point to some coordinates on the curve and say "I like this ratio of TPR/FPR," but then how do you find the decision threshold for that point on the plot?

$\endgroup$
2
  • 1
    $\begingroup$ Your optimal threshold will need to take the costs of decisions into account. These, in turn, will depend on your classifier's quality (i.e., the ROC curve) - but also on the wider context. See the proposed duplicate. $\endgroup$ Nov 18, 2022 at 16:59
  • $\begingroup$ Thanks. I'll check out the linked question that I duplicated. $\endgroup$
    – NaiveBae
    Nov 18, 2022 at 18:09

1 Answer 1

2
$\begingroup$

Given only the ROC curve itself, you can't determine thresholds. A set of response and predictor values maps uniquely to an ROC curve, but not the other way around - an ROC curve is defined by the rank order of the predictor values, but their actual numeric values are irrelevant beyond that. Adding 1 to all predictor values, for example, would result in a visually identical ROC curve, but with all the underlying threshold values increased by 1 unit. It's impossible to infer anything about numeric threshold values from an ROC curve alone.

Typically you'll have the underlying data (the response and predictor values) used to generate the ROC curve. Most software packages that generate ROC curves will take those two vectors and generate an array of thresholds and their corresponding TPRs and FPRs, which are used to plot the ROC curve. You'd need to examine this output to find the TPR/FPR you want, and which threshold it corresponds to. Some ROC packages also print threshold values for each point or optimal points directly on the ROC plot.

$\endgroup$
1
  • $\begingroup$ Thanks @Nuclear Hoagie. This was helpful. $\endgroup$
    – NaiveBae
    Nov 18, 2022 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.