How to determine the optimal threshold for a classifier and generate ROC curve?

Let say we have a SVM classifier, how do we generate ROC curve? (Like theoretically) (because we are generate TPR and FPR with each of the threshold). And how do we determine the optimal threshold for this SVM classifier?

Use the SVM classifier to classify a set of annotated examples, and "one point" on the ROC space based on one prediction of the examples can be identified. Suppose the number of examples is 200, first count the number of examples of the four cases.

\begin{array} {|r|r|r|} \hline & labeled true & labeled false \\ \hline predictedtrue &71& 28\\ \hline predicted false &57&44 \\ \hline \end{array}

Then compute TPR (True Positive Rate) and FPR (False Positive Rate). $TPR = 71/ (71+57)=0.5547$, and $FPR=28/(28+44) = 0.3889$ On the ROC space, the x-axis is FPR, and the y-axis is TPR. So point $(0.3889, 0.5547)$ is obtained.

To draw an ROC curve, just
(1) Adjust some threshold value that control the number of examples labelled true or false
For example, if concentration of certain protein above α% signifies a disease, different values of α yield different final TPR and FPR values. The threshold values can be simply determined in a way similar to grid search; label training examples with different threshold values, train classifiers with different sets of labelled examples, run the classifier on the test data, compute FPR values, and select the threshold values that cover low (close to 0) and high (close to 1) FPR values, i.e., close to 0, 0.05, 0.1, ..., 0.95, 1

(2) Generate many sets of annotated examples
(3) Run the classifier on the sets of examples
(4) Compute a (FPR, TPR) point for each of them
(5) Draw the final ROC curve

Some details can be checked in http://en.wikipedia.org/wiki/Receiver_operating_characteristic.

Besides, these two links are useful about how to determine an optimal threshold. A simple method is to take the one with maximal sum of true positive and false negative rates. Other finer criteria may include other variables involving different thresholds like financial costs, etc.
http://www.medicalbiostatistics.com/roccurve.pdf

• Thanks for your explaination, what about the optimal threshold? – RockTheStar Nov 9 '14 at 3:22
• Sorry, I learned that optimal threshold is a special term just before. After searching, I found that chapter "3.5 Selecting an Optimal Threshold" of book "Analyzing Receiver Operating Characteristic Curves with SAS" on Google Book has some detailed explanation on selecting optimal threshold. The two widely used ways as described on it are to choose the threshold that will make the resulting binary prediction (1) as close to a perfect predictor as possible. (2) as far away from a non-informative predictor as possible – Tom Nov 9 '14 at 7:41
• Cool, where can I find the reference? Thanks! – RockTheStar Nov 9 '14 at 20:32
• Yes, what does "far away from a non-informative predictor" mean? Please add the reference. – Simone Nov 9 '14 at 22:47
• Besides, I also just read from that there are many criteria to determine an optimal threshold. For example, a simple criterion is that among all thresholds, pick the one with maximal sum of true-positive and false-negative values. There are also other more sophisticated criteria. – Tom Nov 10 '14 at 2:44

A really easy way to pick a threshold is to take the median predicted values of the positive cases for a test set. This becomes your threshold.

The threshold comes relatively close to the same threshold you would get by using the roc curve where true positive rate(tpr) and 1 - false positive rate(fpr) overlap. This tpr (cross) 1-fpr cross maximizes true positive while minimizing false negatives.

• I see. Median predicted value. Thanks for the suggestion. – RockTheStar Feb 13 '18 at 3:38
• Is there a source for this method? – JEquihua Jun 7 '18 at 4:48
• This is equivalent to picking the point with TPR=0.5 in the ROC curve, which sounds really arbitrary. – Bananin Jan 9 at 21:19
• Median predicted value? And what happens if you have a class imbalance of 1000:1? – ldmtwo Jan 29 at 3:26

Choose the point closest to the top left corner of your ROC space. Now the threshold used to generate this point should be the optimal one.

• How to do this automatically? – ldmtwo Jan 29 at 3:27

The choice of a threshold depends on the importance of TPR and FPR classification problem. For example, if your classifier will decide which criminal suspects will receive a death sentence, false positives are very bad (innocents will be killed!). Thus you would choose a threshold that yields a low FPR while keeping a reasonable TPR (so you actually catch some true criminals). If there is no external concern about low TPR or high FPR, one option is to weight them equally by choosing the threshold that maximizes $$TPR-FPR$$.