sklearn.metrics.roc_curve returns thresholds array which shape=[n_thresholds]. How does the n_thresholds parameter get selected?
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
2
By definition, a ROC curve represent all possible thresholds in the interval $(-\infty, +\infty)$.
This number is infinite and of course cannot be represented with a computer. Fortunately when you have some data you can simplify this and only visit a limited number of thresholds.
This number corresponds to the number of unique values in the data + 1, or something like:
n_thresholds = len(np.unique(x)) + 1
where x is the array holding your target scores (y_score).
-
$\begingroup$ can you elaborate a bit more? With X you mean features defining the population or the target variable (y)? I am asking this as if X are feature matrix, there potentially could be thousands of unique values or thresholds as you explain in the answer. Thanks for your time! $\endgroup$– BSPCommented Jan 2, 2020 at 13:32
-
1$\begingroup$ @BSP The use of an uppercase X was a mistake on my part. I edited my answer to use lowercase x as it is a 1D array with your predictions. I tried to clarify and fixed the snippet which was wrong of course as you noticed, you get only a single number of thresholds (which can be thousands or more). Thanks for pointing it out! $\endgroup$– CalimoCommented Jan 2, 2020 at 15:35
$\begingroup$
$\endgroup$
n_thresholds = len(np.unique(x)) + 1
The threshold is a continuous numerical variable but only some are not suboptimal/useful, meaning that only those thresholds would affect the confusion matrix(and hence true positive rate or/and false positive rate, and hence the ROC plot).
Which threshold would affect the confusion matrix? The unique values of data, if you move the threshold from a number greater than a unique value to a number smaller than that, the confusion matrix must change. But if you move the threshold between any two unique neighbouring data values, any value in the resultant confusion matrix would remain the same, and hence those thresholds are suboptimal.
But why add one?
thresholds[0] represents no instances being predicted and is arbitrarily set to max(y_score) + 1
Source: sklearn.metrics.roc_curve
Because it starts with the largest value in data + 1.
Reference:
ROC and AUC, Clearly Explained!
answered Mar 25, 2022 at 7:35
$\begingroup$
$\endgroup$
You can inspect the code for sklearn.metrics.roc_curve to see how it determines the number of thresholds returned. I looked at it briefly, and it says that it attempts to drop thresholds that are suboptimal (whatever that means) and these do not appear on the ROC curve. So the number of thresholds is not always equal to the number of scores.
