How to determine the optimal threshold for a classifier and generate ROC curve?

Let say we have a SVM classifier, how do we generate ROC curve? (Like theoretically) (because we are generate TPR and FPR with each of the threshold). And how do we determine the optimal threshold for this SVM classifier?

Use the SVM classifier to classify a set of annotated examples, and "one point" on the ROC space based on one prediction of the examples can be identified. Suppose the number of examples is 200, first count the number of examples of the four cases.

$$\begin{array} {|r|r|r|} \hline & \text{labeled true} & \text{labeled false} \\ \hline \text{predicted true} &71& 28\\ \hline \text{predicted false} &57&44 \\ \hline \end{array}$$

Then compute TPR (True Positive Rate) and FPR (False Positive Rate). $$TPR = 71/ (71+57)=0.5547$$, and $$FPR=28/(28+44) = 0.3889$$ On the ROC space, the x-axis is FPR, and the y-axis is TPR. So point $$(0.3889, 0.5547)$$ is obtained.

To draw an ROC curve, just

1. Adjust some threshold value that control the number of examples labelled true or false
For example, if concentration of certain protein above α% signifies a disease, different values of α yield different final TPR and FPR values. The threshold values can be simply determined in a way similar to grid search; label training examples with different threshold values, train classifiers with different sets of labelled examples, run the classifier on the test data, compute FPR values, and select the threshold values that cover low (close to 0) and high (close to 1) FPR values, i.e., close to 0, 0.05, 0.1, ..., 0.95, 1
2. Generate many sets of annotated examples
3. Run the classifier on the sets of examples
4. Compute a (FPR, TPR) point for each of them
5. Draw the final ROC curve

Some details can be checked in http://en.wikipedia.org/wiki/Receiver_operating_characteristic.

Besides, these two links are useful about how to determine an optimal threshold. A simple method is to take the one with maximal sum of true positive and false negative rates. Other finer criteria may include other variables involving different thresholds like financial costs, etc.
http://www.medicalbiostatistics.com/roccurve.pdf

• Thanks for your explaination, what about the optimal threshold? Nov 9 '14 at 3:22
• Sorry, I learned that optimal threshold is a special term just before. After searching, I found that chapter "3.5 Selecting an Optimal Threshold" of book "Analyzing Receiver Operating Characteristic Curves with SAS" on Google Book has some detailed explanation on selecting optimal threshold. The two widely used ways as described on it are to choose the threshold that will make the resulting binary prediction (1) as close to a perfect predictor as possible. (2) as far away from a non-informative predictor as possible
– Tom
Nov 9 '14 at 7:41
• Cool, where can I find the reference? Thanks! Nov 9 '14 at 20:32
• Yes, what does "far away from a non-informative predictor" mean? Please add the reference. Nov 9 '14 at 22:47
• Besides, I also just read from that there are many criteria to determine an optimal threshold. For example, a simple criterion is that among all thresholds, pick the one with maximal sum of true-positive and false-negative values. There are also other more sophisticated criteria.
– Tom
Nov 10 '14 at 2:44

The choice of a threshold depends on the importance of TPR and FPR classification problem. For example, if your classifier will decide which criminal suspects will receive a death sentence, false positives are very bad (innocents will be killed!). Thus you would choose a threshold that yields a low FPR while keeping a reasonable TPR (so you actually catch some true criminals). If there is no external concern about low TPR or high FPR, one option is to weight them equally by choosing the threshold that maximizes $$TPR-FPR$$.

Choose the point closest to the top left corner of your ROC space. Now the threshold used to generate this point should be the optimal one.

• How to do this automatically? Jan 29 '19 at 3:27
• This article (www0.cs.ucl.ac.uk/staff/W.Langdon/roc) has some good points under the heading "Choosing the Operating Point". picking the point closest to the top left corner of a ROC curve equates to choosing the operating point such that TPR = TNR, i.e. false positives are equally bad as false negatives.
– Will
Nov 13 '20 at 15:57
• Pick the threshold $t^*$ that minimizes the distance to the top left can be done by measuring its distance $t* = \arg\max_t \sqrt{(1 - FPR(t))^2 + TPR(t)^2}$.
– Will
Nov 13 '20 at 15:59

####################################

tpr - (1-fpr) is zero or near to zero is the optimal cut off point

####################################

def plot_roc_curve(fpr, tpr):
plt.plot(fpr, tpr, color='orange', label='ROC')
plt.plot([0, 1], [0, 1], color='darkblue', linestyle='--')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.legend()
plt.show()

y_true = np.array([0,0, 1, 1,1])
y_scores = np.array([0.0,0.09, .05, .75,1])

fpr, tpr, thresholds = roc_curve(y_true, y_scores)
print(tpr)
print(fpr)
print(thresholds)
print(roc_auc_score(y_true, y_scores))
optimal_idx = np.argmax(tpr - fpr)
optimal_threshold = thresholds[optimal_idx]
print("Threshold value is:", optimal_threshold)
plot_roc_curve(fpr, tpr)

Threshold value is: 0.75

• Optimal how? What cost function does this minimize?
– Sycorax
Dec 9 '20 at 20:17
• @Sycorax in this case, it maximizes Youden's J statistic, which is equivalent to maximizing the geometric mean between tpr and fpr. Ref: machinelearningmastery.com/… Mar 9 '21 at 17:03
• @VictorValente Thanks. I was hoping that OP would edit their answer to be specific about why their solution is optimal.
– Sycorax
Mar 9 '21 at 17:04

A really easy way to pick a threshold is to take the median predicted values of the positive cases for a test set. This becomes your threshold.

The threshold comes relatively close to the same threshold you would get by using the roc curve where true positive rate(tpr) and 1 - false positive rate(fpr) overlap. This tpr (cross) 1-fpr cross maximizes true positive while minimizing false negatives.

• I see. Median predicted value. Thanks for the suggestion. Feb 13 '18 at 3:38
• Is there a source for this method? Jun 7 '18 at 4:48
• This is equivalent to picking the point with TPR=0.5 in the ROC curve, which sounds really arbitrary. Jan 9 '19 at 21:19
• Median predicted value? And what happens if you have a class imbalance of 1000:1? Jan 29 '19 at 3:26

Following Will's comment.

This article (www0.cs.ucl.ac.uk/staff/W.Langdon/roc) has some good points under the heading "Choosing the Operating Point". picking the point closest to the top left corner of a ROC curve equates to choosing the operating point such that TPR = TNR, i.e. false positives are equally bad as false negatives. – Will Nov 13 at 15:57.

Using iscost line from the link www0.cs.ucl.ac.uk/staff/W.Langdon/roc. Using these concept:

alpha = cost_false_positive = cost of a false positive (false alarm)

beta = cost_false_negative = cost of missing a positive (false negative)

p = proportion of positive cases

Then the average expected cost of classification at point x,y in the ROC space is C = (1-p) alpha x + p beta (1-y).

To find the best threshold you have to minimize C so :

best_threshold = argmin ( (1-p) alpha x + p beta (1-y) ).

This seams to works.I am open to suggestion or remarks.

Here is the code. In needs to have binary_thresholds, fp_rate, recall. Here fp_rate and recall is of the shape (num_thresholds, 1) or (num_thresholds, num_classes).

def find_best_binary_auc_threshold(binary_thresholds,
fp_rate,
recall,
proportion_positive_case: float = 0.5,
cost_false_positive: float = 0.5,
cost_false_negative: float = 0.5,
argmin_axis: int = 0):
isocost_lines = cost_false_positive * (1 - proportion_positive_case) * fp_rate + cost_false_negative * proportion_positive_case * (1 - recall)
best_indexes = np.argmin(isocost_lines, axis=argmin_axis)
best_thresholds = binary_thresholds[best_indexes.tolist()]
return best_thresholds, best_indexes