Why is $AUC=0.5$ and a 45-degree line for a ROC curve considered baseline performance?

Question

$AUC=0.5$ and an ROC curve of a 45-degree line often are considered the baseline performance of a model, one that gets absolutely nothing from the features.

If we predict the same (prior) probability of class membership every time, how do we wind up with a 45-degree line for the ROC curve? Either we get perfect sensitivity and zero specificity, or we get zero sensitivity and perfect specificity. The 45-degree line implies that a sensitivity of $0.4$ is possible when the specificity is $0.6$.

How does that happen?

And why, with the ROC curve being so funky, do we say that $AUC=0.5$ is the baseline performance? Is this related to how AUC is related to the Wilcoxon test (Harrell’s concordance index) and not just the geometry of the ROC curve?

Answer 1

The AUC is the probability that a randomly selected positive is ranked higher than a randomly selected negative. So an AUC of 0.5 is the performance of a classifier that does not rank positives higher than negatives, nor the opposite. Why is ROC AUC equivalent to the probability that two randomly-selected samples are correctly ranked? and How to derive the probabilistic interpretation of the AUC? fill in the particulars.

Therefore, given that the probability of a positive is ranked higher than a negative is 0.5, then at each choice of threshold, the true positive rate must equal the false positive rate. In other words, given some classification threshold, the model designates as positives some portion of the samples. Among those positives, half of them are true positives, and half of them are false positives.

Answer 2

When it comes to AUC, baseline performance means the performance of a model that has the same distribution of predictions for both classes.

From this, the diagonal line and $AUC=0.5$ follow naturally from the usual definitions.

If you want to make an argument related to a model that outputs the same value every time (something related to my thoughts on the baseline performance in $R^2$-style statistics that I discuss here, here, here, and here), then you can view it this way, which is how software, such as the pROC::roc function in R, seems to work when given distributions with just one value.

Picking a threshold above that single value being predicted every time results in $TPR=TNR=1$, and picking a threshold below that single value results in $TPR=TNR=0$. Now connect those two points with a straight line (one of many options, but the simplest option and how I would expect software to interpolate) to get your ROC curve. From that diagonal line, $AUC=0.5$ follows immediately.

Answer 3

The following minimal example might help. First, we build the most frequent class classifier from small toy data with the prevalent positive class (`True') having 66.6% probability. We then apply it to an even smaller test dataset, predicting the same probability of 66.6% for each instance.

y_train=[True,True,False,True,True,False]
# get the probability of the positive class
# we will use this as a prediction for all test instances
pos_probability=sum(y_train)/len(y_train) 
y_test=[False,True,True]
y_pred_prob = [pos_probability for i in range(0,len(y_test))] 

Next, we get the values of false positive rate and true positive rate used as an input for the ROC curve.

from sklearn.metrics import roc_curve
fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob, pos_label=True)

Finally, we plot the curve:

from sklearn.metrics import RocCurveDisplay
roc_display = RocCurveDisplay(fpr=fpr, tpr=tpr).plot()

How do we wind up with a 45-degree line for the ROC curve? This follows from the input data for the ROC plot. The output of fpr is [0., 1.] The output of tpr is [0., 1.] The ROC plot is created by connecting only these two points ([0,0] and [1,1]) with a straight line.

Why don't we get more points? The reason for this is that the value of the false positive rate and of the true positive rate (almost) do not react to changes in the threshold, which is used to create the input for the ROC curve. In our case, the output of thresholds is [1.66666667, 0.66666667]

The first threshold is arbitrarily set in scikit-learn above 1.0 to ensure that no instances get predicted as positive. The second threshold corresponds in our case to the value of `pos_probability'.

Recall that $tpr=\frac{tp}{positives}$ and $fpr=\frac{fp}{negatives}$ for a given threshold value. Since these relative values are used for axes, it does not matter how many positives or negatives (absolute) are in the dataset.