Different ROC value for different packages in R, which one is correct?

Question

I noticed that computing ROC with caret package and PROC packege sometimes gives different results. Usually they are the same, but if the predictions are worse than chance, caret will flip them and output 1 - ROC from PRROC. However there is sometimes exception when PRROC = 1-caret's.

example data and code:

library(caret)
library(PRROC)

# True labels
obs  <- c("c2", "c1", "c1", "c2", "c2", "c1", "c2", "c1", "c1")
# Probability of class1
c1 <- c(0.968, 0.282, 0.940, 0.940, 0.532, 0.312, 0.308, 0.730, 0.676)
# probability of class2
c2 <- 1 - c1
# actual prediction
pred <- c("c1", "c2", "c1", "c1", "c1", "c2", "c2", "c1", "c1")
dat1  <- data.frame(obs = obs, c1 = c1, c2 = c2, pred = pred)

obs  <- c("c2", "c1", "c2", "c2", "c1", "c1", "c1", "c2", "c1")
c1 <- c(0.622, 0.816, 0.662, 0.400, 0.434, 0.634, 0.550, 0.500, 0.482)
c2   <- 1 - c1
pred <- c("c1", "c1", "c1", "c2", "c2", "c1", "c1", "c2", "c2")
dat2 <- data.frame(obs = obs, c1 = c1, c2 = c2, pred = pred)

dat <- dat1
#caret ROC
twoClassSummary(dat, lev = c("c1", "c2")) 
# -> ROC 0.625
#PRROC ROC
roc.curve(dat[dat$obs == "c1", "c1"], dat[dat$obs == "c2", "c1"]) 
# -> ROC 0.375 

dat <- dat2
twoClassSummary(dat, lev = c("c1", "c2"))
# -> ROC 0.45 
roc.curve(dat[dat$obs == "c1", "c1"], dat[dat$obs == "c2", "c1"])
# -> 0.55 

Can ROC be < 0.5? Which package should I use? I suppose caret will artificially inflate ROC in cross-validation if there is no signal in the data.

Answer 1

ROC can be worse than 0.5. When ROC < 0.5, if you reverse the model recommendation, (i.e. if the model is so bad it recommends all 0 as 1 and all 1 as 0) you can turn a bad classifier into a better one. So that's what caret is doing.

To be clear, I'm not advising that this is good practice or recommended for any particular purpose -- I actually think that reversing recommendations based on AUC can be incredibly deceptive, and can conceal tremendous flaws with a model.