# Why would pROC::roc calculate $\max\{AUC, 1 - AUC\}$ by default?

There is some interesting behavior in the pROC::roc function in R.

library(pROC)
set.seed(2024)
N <- 100
p <- rbeta(N, 1/2, 1/2)
y <- rbinom(N, 1, p)
r1 <- pROC::roc(y, p)
r2 <- pROC::roc(y, 1 - p)
r1$auc # I get 0.8894 r2$auc # I get 0.8894


Whether we use the probabilities (p) or flip all of the probabilities (1-p), the function calculates the same area under the ROC curve.

But that is not consistent with the usual way of thinking about how to calculate ROC curves by varying the cutoff threshold and calculating sensitivity-specificity pairs at each threshold. One of these should show a rather high AUC near one, while the other should show a rather low AUC near zero. After all, when we flip p to 1-p, instead of the high probability values typically corresponding to values of one and low probability values typically corresponding to values of zero, the reverse is true, leading to terrible sensitivity and specificity values.

library(ModelMetrics)
thresholds <- r1\$thresholds
sens1 <- spec1 <- sens2 <- spec2 <- rep(NA, length(thresholds))
yhat1 <- yhat2 <- rep(0, length(thresholds))
for (i in 1:length(thresholds)){

# I'm forgetting how to compare to a threshold in a cleaner way
#
idx1 <- which(p > thresholds[i])
idx2 <- which(1 - p > thresholds[i])
#
yhat1[idx1] <- 1
yhat1[-idx1] <- 0
yhat2[idx2] <- 1
yhat2[-idx2] <- 0

sens1[i] <- ModelMetrics::sensitivity(y, yhat1)
sens2[i] <- ModelMetrics::sensitivity(y, yhat2)
#
spec1[i] <- ModelMetrics::specificity(y, yhat1)
spec2[i] <- ModelMetrics::specificity(y, yhat2)

}
plot(r1)
points(spec1, sens1, col = 'blue')
points(spec2, sens2, col = 'red')


Plotting this, the red curve corresponding to the 1-p predictions has a terrible area under the curve, as expected.

Consequently, I am left wondering why pROC::roc would calculate the AUC to be the same for such different inputs, one with good probability values and one with terrible probability values. Fortunately, pROC::roc can be run with direction = "<" to keep the function from outputting $$\max\{AUC, 1 - AUC\}$$, instead returning the true AUC value. However, that is not the default behavior? Is there some statistical reason why the function returns $$\max\{AUC, 1 - AUC\}$$ instead of just the AUC? I guess I can see an argument that $$AUC<0.5$$ means that a simple transformation of the predicted values gives $$AUC>0.5$$, but you have to know to do such a calibration step, lest you work with the terrible predicted values.

As the package developer is a member of Cross Validated, I would be especially interested in an answer from such a source.

• The ROC curve shows the parametric plot of (TPR(threshold),(1-TNR(threshold)) I guess it has to do with: the positive class and the negative class label can be exchanged without changing the difficulty of the problem ... Maybe someone can connect the dots Commented Jul 1 at 18:19
• @Ggjj11 That seems to allude to what I wrote about a transformation leading to good predictions. However, this behavior masks poor predictions by presenting the performance of post-processed values without giving any kind of alert that such post-processing is required to get good predictions.
– Dave
Commented Jul 1 at 18:23
• This related question might be relevant: stats.stackexchange.com/questions/371293/proc-versus-rocr Commented Jul 1 at 19:12
• @Calimo I have figured out how to use both packages and can use yours without precluding AUC < 0.5. However, I am most interested in hearing from you, the developer, why the default behavior is good default behavior.
– Dave
Commented Jul 2 at 14:46

## 1 Answer

First a correction: the roc function of pROC (for simplicity, let me simply refer to it as "pROC" from now) doesn't compute $$\max\{AUC, 1 - AUC\}$$. Instead, by default, that is if the direction argument is set to the "auto" value, it determines the direction of positivity with the following rule:

$$direction= \begin{cases} <, & median(controls) \leq median(cases) \\ >, & \text{otherwise} \end{cases}$$

This is going to be almost equivalent to taking the maximum AUC, but in some cases you can obtain an AUC slightly below 0.5.

Now, to answer your question, here are a few thoughts:

• The initial use-case for pROC was protein biomarkers. These data are not nice, calibrated probabilities between 0 and 1, but wildly distributed values, sometimes not even always numerical (when they are outside of the detection range), and can be either elevated or decreased in the positive groups of interest. From what I have seen (from bug reports, questions, etc.), most users of pROC work on scores that aren't probabilities.

• The pROC package was designed to be user-friendly for users with little to no knowledge of programming (at the time of release there was even a GUI for S+). User-friendly here means, make as many decisions as possible about the analysis without pushing the responsibility to make them back to the user.

• The first thing you can auto-detect to make a user-friendly ROC function is which group is the positive, and which one is the negative. Rather than requesting users to convert their group to {0, 1}, or to supply the positive group as a mandatory argument, pROC auto-detects the levels of the grouping factor, and takes the higher one as positive. This works well for 0 and 1 where 1 is higher, but is a bit more error-prone with text labels.

• The second thing is the direction of the comparison. If the positive group was mis-detected, this will still result in a correct AUC value (although the curve itself will be flipped, which comes with its own set of issues).

• Auto-detecting only one of these two things will result in AUCs erroneously < 0.5. There are a ton of "My AUC is low" and "My ROC curve is reversed" questions on the various stack exchanges, which highlight how hard it is to get this right.

• Transparency: to try and mitigate the risk of errors, pROC messages the user about these decisions. So when you executed the code you wrote above, the following messages appeared in the output:

  Setting levels: control = 0, case = 1
Setting direction: controls < cases
Setting levels: control = 0, case = 1
Setting direction: controls > cases


Of course I know no-one looks at output, and no-one reads the documentation and the FAQ.

• One additional reason to implement it this way was to mimic the behavior of other statistical software. I know at least SPSS does something similar, but I remember others did that too.

• This is the kind of response I was hoping to see! I will award the bounty in a few days (probably during the grace period).
– Dave
Commented Jul 6 at 14:38
• In place of <= , you could use \leq or \leqslant if you wish. Commented Jul 6 at 14:51