# How to determine the optimal threshold for a classifier and generate ROC curve?

Let say we have a SVM classifier, how do we generate ROC curve? (Like theoretically) (because we are generate TPR and FPR with each of the threshold). And how do we determine the optimal threshold for this SVM classifier?

## 6 Answers

Use the SVM classifier to classify a set of annotated examples, and "one point" on the ROC space based on one prediction of the examples can be identified. Suppose the number of examples is 200, first count the number of examples of the four cases.

$$\begin{array} {|r|r|r|} \hline & \text{labeled true} & \text{labeled false} \\ \hline \text{predicted true} &71& 28\\ \hline \text{predicted false} &57&44 \\ \hline \end{array}$$

Then compute TPR (True Positive Rate) and FPR (False Positive Rate). $$TPR = 71/ (71+57)=0.5547$$, and $$FPR=28/(28+44) = 0.3889$$ On the ROC space, the x-axis is FPR, and the y-axis is TPR. So point $$(0.3889, 0.5547)$$ is obtained.

To draw an ROC curve, just

1. Adjust some threshold value that control the number of examples labelled true or false
For example, if concentration of certain protein above α% signifies a disease, different values of α yield different final TPR and FPR values. The threshold values can be simply determined in a way similar to grid search; label training examples with different threshold values, train classifiers with different sets of labelled examples, run the classifier on the test data, compute FPR values, and select the threshold values that cover low (close to 0) and high (close to 1) FPR values, i.e., close to 0, 0.05, 0.1, ..., 0.95, 1
2. Generate many sets of annotated examples
3. Run the classifier on the sets of examples
4. Compute a (FPR, TPR) point for each of them
5. Draw the final ROC curve

Some details can be checked in http://en.wikipedia.org/wiki/Receiver_operating_characteristic.

Besides, these two links are useful about how to determine an optimal threshold. A simple method is to take the one with maximal sum of true positive and false negative rates. Other finer criteria may include other variables involving different thresholds like financial costs, etc.
http://www.medicalbiostatistics.com/roccurve.pdf
http://www.kovcomp.co.uk/support/XL-Tut/life-ROC-curves-receiver-operating-characteristic.html

• Thanks for your explaination, what about the optimal threshold? Commented Nov 9, 2014 at 3:22
• Sorry, I learned that optimal threshold is a special term just before. After searching, I found that chapter "3.5 Selecting an Optimal Threshold" of book "Analyzing Receiver Operating Characteristic Curves with SAS" on Google Book has some detailed explanation on selecting optimal threshold. The two widely used ways as described on it are to choose the threshold that will make the resulting binary prediction (1) as close to a perfect predictor as possible. (2) as far away from a non-informative predictor as possible
– Tom
Commented Nov 9, 2014 at 7:41
• Cool, where can I find the reference? Thanks! Commented Nov 9, 2014 at 20:32
• Yes, what does "far away from a non-informative predictor" mean? Please add the reference. Commented Nov 9, 2014 at 22:47
• Besides, I also just read from that there are many criteria to determine an optimal threshold. For example, a simple criterion is that among all thresholds, pick the one with maximal sum of true-positive and false-negative values. There are also other more sophisticated criteria.
– Tom
Commented Nov 10, 2014 at 2:44

The choice of a threshold depends on the importance of TPR and FPR classification problem. For example, if your classifier will decide which criminal suspects will receive a death sentence, false positives are very bad (innocents will be killed!). Thus you would choose a threshold that yields a low FPR while keeping a reasonable TPR (so you actually catch some true criminals). If there is no external concern about low TPR or high FPR, one option is to weight them equally by choosing the threshold that maximizes $$TPR-FPR$$.

Choose the point closest to the top left corner of your ROC space. Now the threshold used to generate this point should be the optimal one.

• How to do this automatically? Commented Jan 29, 2019 at 3:27
• This article (www0.cs.ucl.ac.uk/staff/W.Langdon/roc) has some good points under the heading "Choosing the Operating Point". picking the point closest to the top left corner of a ROC curve equates to choosing the operating point such that TPR = TNR, i.e. false positives are equally bad as false negatives.
– Will
Commented Nov 13, 2020 at 15:57
• Pick the threshold $t^*$ that minimizes the distance to the top left can be done by measuring its distance $t* = \arg\max_t \sqrt{(1 - FPR(t))^2 + TPR(t)^2}$.
– Will
Commented Nov 13, 2020 at 15:59
• The Equal Error Rate could be also used in this context, i.e., eer = brentq(lambda x: 1. - x - interp1d(fpr, tpr)(x), 0., 1.). Commented Feb 11, 2022 at 6:36
• @Will Then, is it better to maximize (tpr-fpr) or (sqrt((1-fpr)^2)+tpr^2)? Anyway I think it sould be argmin.
– skan
Commented Aug 2, 2023 at 18:06

####################################

# tpr - (1-fpr) is zero or near to zero is the optimal cut off point

####################################

def plot_roc_curve(fpr, tpr):
plt.plot(fpr, tpr, color='orange', label='ROC')
plt.plot([0, 1], [0, 1], color='darkblue', linestyle='--')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver Operating Characteristic (ROC) Curve')
plt.legend()
plt.show()

y_true = np.array([0,0, 1, 1,1])
y_scores = np.array([0.0,0.09, .05, .75,1])

fpr, tpr, thresholds = roc_curve(y_true, y_scores)
print(tpr)
print(fpr)
print(thresholds)
print(roc_auc_score(y_true, y_scores))
optimal_idx = np.argmax(tpr - fpr)
optimal_threshold = thresholds[optimal_idx]
print("Threshold value is:", optimal_threshold)
plot_roc_curve(fpr, tpr)


Threshold value is: 0.75

• Optimal how? What cost function does this minimize?
– Sycorax
Commented Dec 9, 2020 at 20:17
• @Sycorax in this case, it maximizes Youden's J statistic, which is equivalent to maximizing the geometric mean between tpr and fpr. Ref: machinelearningmastery.com/… Commented Mar 9, 2021 at 17:03
• @VictorValente Thanks. I was hoping that OP would edit their answer to be specific about why their solution is optimal.
– Sycorax
Commented Mar 9, 2021 at 17:04

A really easy way to pick a threshold is to take the median predicted values of the positive cases for a test set. This becomes your threshold.

The threshold comes relatively close to the same threshold you would get by using the roc curve where true positive rate(tpr) and 1 - false positive rate(fpr) overlap. This tpr (cross) 1-fpr cross maximizes true positive while minimizing false negatives.

• I see. Median predicted value. Thanks for the suggestion. Commented Feb 13, 2018 at 3:38
• Is there a source for this method? Commented Jun 7, 2018 at 4:48
• This is equivalent to picking the point with TPR=0.5 in the ROC curve, which sounds really arbitrary. Commented Jan 9, 2019 at 21:19
• Median predicted value? And what happens if you have a class imbalance of 1000:1? Commented Jan 29, 2019 at 3:26

Following Will's comment.

This article (www0.cs.ucl.ac.uk/staff/W.Langdon/roc) has some good points under the heading "Choosing the Operating Point". picking the point closest to the top left corner of a ROC curve equates to choosing the operating point such that TPR = TNR, i.e. false positives are equally bad as false negatives. – Will Nov 13 at 15:57.

Using iscost line from the link www0.cs.ucl.ac.uk/staff/W.Langdon/roc. Using these concept:

alpha = cost_false_positive = cost of a false positive (false alarm)

beta = cost_false_negative = cost of missing a positive (false negative)

p = proportion of positive cases

Then the average expected cost of classification at point x,y in the ROC space is C = (1-p) alpha x + p beta (1-y).

To find the best threshold you have to minimize C so :

best_threshold = argmin ( (1-p) alpha x + p beta (1-y) ).

This seams to works.I am open to suggestion or remarks.

Here is the code. In needs to have binary_thresholds, fp_rate, recall. Here fp_rate and recall is of the shape (num_thresholds, 1) or (num_thresholds, num_classes).

def find_best_binary_auc_threshold(binary_thresholds,
fp_rate,
recall,
proportion_positive_case: float = 0.5,
cost_false_positive: float = 0.5,
cost_false_negative: float = 0.5,
argmin_axis: int = 0):
isocost_lines = cost_false_positive * (1 - proportion_positive_case) * fp_rate + cost_false_negative * proportion_positive_case * (1 - recall)
best_indexes = np.argmin(isocost_lines, axis=argmin_axis)
best_thresholds = binary_thresholds[best_indexes.tolist()]
return best_thresholds, best_indexes