Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Given an ROC curve, are there well known methods (or approaches) to choosing a threshold from the curve, as the discriminating threshold for the sake of binary classification?
Of course we can assign weights between either FPR and TPR or precision and recall (either 50/50 or something else), and calculate an optimal point, yet I wonder whether there are some other complementary methods for choosing an optimal threshold, either from the ROC curve or through other measures/algorithms?
There are two I know of.
Picking the point where $TPR+FPR=1$, i.e. the point that lies on the "main" diagonal of the ROC curve.
Picking the point where $TPR+1-FPR$ is maximal.
Both are discussed in (Lobo, Jiménez‐Valverde & Real, 2008), which is also a good reference on misuses and misunderstandings on the AUC.
Paradoxically, once the AUC score was developed as a threshold-independent measure, researchers proposed methods for selecting a threshold from this curve. It has been assumed that in ROC plots the optimal classifier point is the one that maximizes the sum of sensitivity and specificity (Zweig & Campbell, 1993). However, Jiménez-Valverde & Lobo (2007) have found that a threshold that minimizes the difference between sensitivity and specificity performs slightly better than one that maximizes the sum if commission and omission errors are equally costly. When the threshold changes from 0 to 1, the rate of well-predicted presences diminishes while the rate of well-predicted absences increases. The point where both curves cross can be considered the appropriate threshold if both types of errors are equally weighted (Fig. 1a). In a ROC plot, this point lies at the intersection of the ROC curve and the line perpendicular to the diagonal of no discrimination (Fig. 1b), i.e., the ‘northwesternmost’ point of the ROC curve. The two thresholds can be easily computed without using the ROC curve. Both thresholds are highly correlated and, more importantly, they also correlate with prevalence (Liu et al., 2005; Jiménez-Valverde & Lobo, 2007). As a general rule, a good classifier needs to minimize the false positive and negative rates or, similarly, to maximize the true negative and positive rates. Thus, if we place equal weight on presences and absences there is only one correct threshold. This optimal threshold, the one that minimizes the difference between sensitivity and specificity, achieves this objective and provides a balanced trade-off between commission and omission errors. Nevertheless, as pointed out before, if different costs are assigned to false negatives and false positives, and the prevalence bias is always taken into account, the threshold should be selected according to the required criteria. It is also necessary to underline that the transformation of continuous probabilities into binary maps is frequently necessary for many practical applications that rely on making decisions (e.g., reserve selection).
