3
$\begingroup$

$AUC=0.5$ and an ROC curve of a 45-degree line often are considered the baseline performance of a model, one that gets absolutely nothing from the features.

If we predict the same (prior) probability of class membership every time, how do we wind up with a 45-degree line for the ROC curve? Either we get perfect sensitivity and zero specificity, or we get zero sensitivity and perfect specificity. The 45-degree line implies that a sensitivity of $0.4$ is possible when the specificity is $0.6$.

How does that happen?

And why, with the ROC curve being so funky, do we say that $AUC=0.5$ is the baseline performance? Is this related to how AUC is related to the Wilcoxon test (Harrell’s concordance index) and not just the geometry of the ROC curve?

$\endgroup$
7
  • $\begingroup$ I assume this is a duplicate, though I couldn’t find one. If this is not a duplicate, consider me surprised that Cross Validated took 12 years to get such a question. $\endgroup$
    – Dave
    Commented Sep 26, 2022 at 3:38
  • $\begingroup$ The AUC is the probability that a randomly selected positive is ranked higher than a randomly selected negative. So an AUC of 0.5 is the performance of a classifier that does not rank positives higher than negatives, nor the opposite. stats.stackexchange.com/questions/190216/… and stats.stackexchange.com/questions/180638/… fill in the particulars. $\endgroup$
    – Sycorax
    Commented Sep 26, 2022 at 4:15
  • $\begingroup$ @Sycorax I didn’t read those in detail (yet), but I figured it was something like that for the AUC/c-index. However, how do we justify plotting that diagonal line for the ROC curve, such as plot(pROC::roc(c(0,1,0,1,0,1), c(1,1,1,1,1,1)/2)) in R? $\endgroup$
    – Dave
    Commented Sep 26, 2022 at 4:23
  • $\begingroup$ Given that the probability of a positive is ranked higher than a negative is 0.5, then at each choice of threshold, the true positive rate must equal the false positive rate. $\endgroup$
    – Sycorax
    Commented Sep 26, 2022 at 5:08
  • 3
    $\begingroup$ My understanding of this is that the baseline used for ROC/AUC is a random ranking of the outcomes, rather than equal predictions. $\endgroup$
    – Lynn
    Commented Sep 26, 2022 at 9:16

3 Answers 3

2
$\begingroup$

The AUC is the probability that a randomly selected positive is ranked higher than a randomly selected negative. So an AUC of 0.5 is the performance of a classifier that does not rank positives higher than negatives, nor the opposite. Why is ROC AUC equivalent to the probability that two randomly-selected samples are correctly ranked? and How to derive the probabilistic interpretation of the AUC? fill in the particulars.

Therefore, given that the probability of a positive is ranked higher than a negative is 0.5, then at each choice of threshold, the true positive rate must equal the false positive rate. In other words, given some classification threshold, the model designates as positives some portion of the samples. Among those positives, half of them are true positives, and half of them are false positives.

$\endgroup$
2
  • $\begingroup$ If all of the predictions are equal, then the probability of ranking a positive case above a negative case is zero. $\endgroup$
    – Dave
    Commented Sep 26, 2022 at 13:03
  • $\begingroup$ All predictions being equal does not have any bearing on AUC = 0.5, because AUC can equal 0.5 even if the predictions are not all equal. This is the second time you've made reference to this, but it's still not relevant. As an example: pROC::roc(c(rep(0, N/2), rep(1, N/2)), runif(N)) will have AUC 0.5 in expectation, and the ROC curve will appear to be a diagonal line (but any particular sample will exhibit some small deviation, even for large N). Each prediction is unique (almost surely), yet the AUC is 0.5 (in expectation). $\endgroup$
    – Sycorax
    Commented Sep 26, 2022 at 13:05
0
$\begingroup$

When it comes to AUC, baseline performance means the performance of a model that has the same distribution of predictions for both classes.

From this, the diagonal line and $AUC=0.5$ follow naturally from the usual definitions.

If you want to make an argument related to a model that outputs the same value every time (something related to my thoughts on the baseline performance in $R^2$-style statistics that I discuss here, here, here, and here), then you can view it this way, which is how software, such as the pROC::roc function in R, seems to work when given distributions with just one value.

Picking a threshold above that single value being predicted every time results in $TPR=TNR=1$, and picking a threshold below that single value results in $TPR=TNR=0$. Now connect those two points with a straight line (one of many options, but the simplest option and how I would expect software to interpolate) to get your ROC curve. From that diagonal line, $AUC=0.5$ follows immediately.

$\endgroup$
0
$\begingroup$

The following minimal example might help. First, we build the most frequent class classifier from small toy data with the prevalent positive class (`True') having 66.6% probability. We then apply it to an even smaller test dataset, predicting the same probability of 66.6% for each instance.

y_train=[True,True,False,True,True,False]
# get the probability of the positive class
# we will use this as a prediction for all test instances
pos_probability=sum(y_train)/len(y_train) 
y_test=[False,True,True]
y_pred_prob = [pos_probability for i in range(0,len(y_test))] 

Next, we get the values of false positive rate and true positive rate used as an input for the ROC curve.

from sklearn.metrics import roc_curve
fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob, pos_label=True)

Finally, we plot the curve:

from sklearn.metrics import RocCurveDisplay
roc_display = RocCurveDisplay(fpr=fpr, tpr=tpr).plot()

enter image description here

How do we wind up with a 45-degree line for the ROC curve? This follows from the input data for the ROC plot. The output of fpr is [0., 1.] The output of tpr is [0., 1.] The ROC plot is created by connecting only these two points ([0,0] and [1,1]) with a straight line.

Why don't we get more points? The reason for this is that the value of the false positive rate and of the true positive rate (almost) do not react to changes in the threshold, which is used to create the input for the ROC curve. In our case, the output of thresholds is [1.66666667, 0.66666667]

The first threshold is arbitrarily set in scikit-learn above 1.0 to ensure that no instances get predicted as positive. The second threshold corresponds in our case to the value of `pos_probability'.

Recall that $tpr=\frac{tp}{positives}$ and $fpr=\frac{fp}{negatives}$ for a given threshold value. Since these relative values are used for axes, it does not matter how many positives or negatives (absolute) are in the dataset.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.