How to determine the optimal threshold for a classifier and generate ROC curve?

Question

Let say we have a SVM classifier, how do we generate ROC curve? (Like theoretically) (because we are generate TPR and FPR with each of the threshold). And how do we determine the optimal threshold for this SVM classifier?

Answer 1

Use the SVM classifier to classify a set of annotated examples, and "one point" on the ROC space based on one prediction of the examples can be identified. Suppose the number of examples is 200, first count the number of examples of the four cases.

\begin{array} {|r|r|r|} \hline & \text{labeled true} & \text{labeled false} \\ \hline \text{predicted true} &71& 28\\ \hline \text{predicted false} &57&44 \\ \hline \end{array}

Then compute TPR (True Positive Rate) and FPR (False Positive Rate). $TPR = 71/ (71+57)=0.5547$, and $FPR=28/(28+44) = 0.3889$ On the ROC space, the x-axis is FPR, and the y-axis is TPR. So point $(0.3889, 0.5547)$ is obtained.

To draw an ROC curve, just

1. Adjust some threshold value that control the number of examples labelled true or false
   For example, if concentration of certain protein above α% signifies a disease, different values of α yield different final TPR and FPR values. The threshold values can be simply determined in a way similar to grid search; label training examples with different threshold values, train classifiers with different sets of labelled examples, run the classifier on the test data, compute FPR values, and select the threshold values that cover low (close to 0) and high (close to 1) FPR values, i.e., close to 0, 0.05, 0.1, ..., 0.95, 1
2. Generate many sets of annotated examples
   
3. Run the classifier on the sets of examples
   
4. Compute a (FPR, TPR) point for each of them
   
5. Draw the final ROC curve

Some details can be checked in http://en.wikipedia.org/wiki/Receiver_operating_characteristic.

Besides, these two links are useful about how to determine an optimal threshold. A simple method is to take the one with maximal sum of true positive and false negative rates. Other finer criteria may include other variables involving different thresholds like financial costs, etc.
http://www.medicalbiostatistics.com/roccurve.pdf
http://www.kovcomp.co.uk/support/XL-Tut/life-ROC-curves-receiver-operating-characteristic.html

Answer 2

The choice of a threshold depends on the importance of TPR and FPR classification problem. For example, if your classifier will decide which criminal suspects will receive a death sentence, false positives are very bad (innocents will be killed!). Thus you would choose a threshold that yields a low FPR while keeping a reasonable TPR (so you actually catch some true criminals). If there is no external concern about low TPR or high FPR, one option is to weight them equally by choosing the threshold that maximizes $TPR-FPR$.

Answer 3

Choose the point closest to the top left corner of your ROC space. Now the threshold used to generate this point should be the optimal one.

Answer 4

####################################

The optimal cut off would be where tpr is high and fpr is low

tpr - (1-fpr) is zero or near to zero is the optimal cut off point

####################################

def plot_roc_curve(fpr, tpr):
    plt.plot(fpr, tpr, color='orange', label='ROC')
    plt.plot([0, 1], [0, 1], color='darkblue', linestyle='--')
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('Receiver Operating Characteristic (ROC) Curve')
    plt.legend()
    plt.show()

y_true = np.array([0,0, 1, 1,1])
y_scores = np.array([0.0,0.09, .05, .75,1])

fpr, tpr, thresholds = roc_curve(y_true, y_scores)
print(tpr)
print(fpr)
print(thresholds)
print(roc_auc_score(y_true, y_scores))
optimal_idx = np.argmax(tpr - fpr)
optimal_threshold = thresholds[optimal_idx]
print("Threshold value is:", optimal_threshold)
plot_roc_curve(fpr, tpr)

Threshold value is: 0.75

Answer 5

A really easy way to pick a threshold is to take the median predicted values of the positive cases for a test set. This becomes your threshold.

The threshold comes relatively close to the same threshold you would get by using the roc curve where true positive rate(tpr) and 1 - false positive rate(fpr) overlap. This tpr (cross) 1-fpr cross maximizes true positive while minimizing false negatives.

Answer 6

Following Will's comment.

This article (www0.cs.ucl.ac.uk/staff/W.Langdon/roc) has some good points under the heading "Choosing the Operating Point". picking the point closest to the top left corner of a ROC curve equates to choosing the operating point such that TPR = TNR, i.e. false positives are equally bad as false negatives. – Will Nov 13 at 15:57.

Using iscost line from the link www0.cs.ucl.ac.uk/staff/W.Langdon/roc. Using these concept:

alpha = cost_false_positive = cost of a false positive (false alarm)

beta = cost_false_negative = cost of missing a positive (false negative)

p = proportion of positive cases

Then the average expected cost of classification at point x,y in the ROC space is C = (1-p) alpha x + p beta (1-y).

To find the best threshold you have to minimize C so :

best_threshold = argmin ( (1-p) alpha x + p beta (1-y) ).

This seams to works.I am open to suggestion or remarks.

Here is the code. In needs to have binary_thresholds, fp_rate, recall. Here fp_rate and recall is of the shape (num_thresholds, 1) or (num_thresholds, num_classes).

def find_best_binary_auc_threshold(binary_thresholds,
                               fp_rate,
                               recall,
                               proportion_positive_case: float = 0.5,
                               cost_false_positive: float = 0.5,
                               cost_false_negative: float = 0.5,
                               argmin_axis: int = 0):
isocost_lines = cost_false_positive * (1 - proportion_positive_case) * fp_rate + cost_false_negative * proportion_positive_case * (1 - recall)
best_indexes = np.argmin(isocost_lines, axis=argmin_axis)
best_thresholds = binary_thresholds[best_indexes.tolist()]
return best_thresholds, best_indexes