Where in the ROC curve does it tell you what the threshold is?

Question

In my understanding, the ROC curve plots the True positive rate and the False positive rate.

However, I've also read in other places that the ROC curve helps determine where the threshold for classifying something as "1" should be. Eg. Lets say if the probability of an an object is a "dog" is greater than 50% or 0.5, the classifier would classify it as 1=Dog, and <0.5 = 0 (Not Dog).

So where on this ROC curve does it show that threshold, if it only plots the True positive rate and False positive rate? Is the threshold simply where the "elbow" is? And do we pull the x axis' number or the y axis' number?

Eg. If we use the x axis' number (false positive rate), the green classifier's threshold should be 0.3 (aka anything above a 30% probability will be classified as 1=Dog)

Answer 1

Each (FPR, TPR) point on a ROC curve is associated with a threshold. However, the thresholds are not typically drawn on the curve itself.

It is possible to reveal them, either adding extra annotation to the curve, or by coloring the curves. Here are some examples generated in R with pROC and ROCR, respectively:

Here is R code to generate these plots:

set.seed(42)
truth <- rbinom(30, 1, 0.5)
predictor <- rnorm(30) + truth + 1

library(pROC)
plot(roc(truth, predictor), print.thres="local")


library(ROCR)
pred <- prediction(predictor, truth)
perf <- performance(pred, measure = "tpr", x.measure = "fpr")
plot(perf, colorize=TRUE)

Answer 2

It doesn't. If you care about having the highest possible TPR and FPR, the threshold is where the elbow is. If you care more about TPR than FPR, or the other way around, it's something else. If you care about optimizing some other metric, it may be something else.

Answer 3

There is no universal truth to that question. There is always a tradeoff made with any threshold and the ROC visualizes all possible thresholds for you to pick the best.

Is it better to err on one side or better to err on the other? That will heavily depend on the topic at hand. A screening test for a disease with the implication to not send your kid to school for a week is less critical with a false positive then a diagnostic test with the implication of removing a limb or an organ.

If that is not good enough you may want to look into Youden's J or the Youden Index and the threshold that maximizes it.

Answer 4

Since you mentioned python, here is an example. It's pretty trivial to plot the threshold along with the ROC using sklearn.metrics.roc_curve.

from sklearn import metrics
import numpy as np
import matplotlib.pyplot as plt

fpr, tpr, th = metrics.roc_curve(true, score)
fig, ax = plt.subplots()
fig.set_size_inches(10,10)
ax.plot(fpr,tpr,label='ROC')

##Plotting the threshold. The value of the first index can be above one so I exclude that point when plotting.
ax.plot(fpr[1:],th[1:],label='Threshold')

ax.plot(fpr,fpr,c='k')
ax.xaxis.set_ticks(np.arange(0, 1.05, .1))
ax.yaxis.set_ticks(np.arange(0, 1.05, .1))
plt.grid(True)

auc = np.round(metrics.roc_auc_score(true, score),3)
plt.title(f' AUC: {auc}',fontsize=25)
plt.xlabel('FPR',fontsize=20)
plt.ylabel('TPR',fontsize=20)
plt.legend(fontsize=20)

With the threshold plotted along with the ROC you can see that as the threshold approaches 1 (on the left side of the figure) TPR and FPR both approach 0, and as the threshold approaches 0 the TPR and FPR both approach 1.

Answer 5

The ROC curve does not "tell" you what threshold to use by itself. Setting the threshold is a business decision made by the users of the model based on the specifics of their application.

The business people's job is to know how many false positives and false negatives they can take. The ROC curve tells them what are the business implications of picking a given threshold. If there is a threshold value that suits their needs they will use the model.

Determining the threshold involves quantifying several items that depend on the application. Usually false positives carry a cost while false negatives carry a risk. Their impact changes according to what the whole process is like, its goal and where in the process the model is used (is it after a screening? is this the first screening? Is the purpose medical? Security? Financial?). All this info needs to be considered in conjunction with the ROC curve to reach a decision.

Answer 6

As others have pointed out, ROC curve does not display threshold values, unless additionally annotated. For the purpose of determining the best decision threshold, classification plot may be more convenient.

To determine the threshold, draw a horizontal line from the desired sensitivity (90% in the figure) to the sensitivity curve (blue), then draw a vertical line to the x-axis. The ordinate of the point at which the vertical line intersects the specificity curve (red) is the matching specificity, and the abscissa is the corresponding decision threshold. This is similar to choosing the best combination of (sensitivity, specificity) on the ROC graph, but in addition it directly provides the decision threshold.