Seeing those quotes out of context, they seem extreme.
If there is a clear boundary between the predictions, then the model has done a good job of distinguishing between the two categories. If there is a clear area where predictions correspond to one category and another where predictions correspond to another category, but there is an ambiguous area between them, then the model has done less of a fantastic job of separating the categories. If the predictions corresponding to the categories are basically on top of each other, then the model does a poor job of distinguishing between the categories.
This is exactly what ROC AUC measures, the extent to which the predictions are separated by group.
Look at how good of a job it does at describing the separation in the plot below.
ROC AUC is not without its flaws, such as those discussed by Frank Harrell, but it seems clear to me what the AUC measures and that it increases as something desirable happens.
# To reproduce the KDE plots with the ROC AUC
library(ggplot2)
library(pROC)
set.seed(2023)
N <- 1000
p1 <- c(runif(N, 0, 0.35), runif(N, 0.65, 1))
y1 <- c(rep(0, N), rep(1, N))
p2 <- rbeta(2*N, 1/2, 1/2)
y2 <- rbinom(2*N, 1, p2)
p3 <- rbeta(2*N, 1, 1)
y3 <- rbinom(2*N, 1, 0.5)
d1 <- data.frame(
Category = as.factor(y1),
Predictions = p1,
group = paste("AUC =", round(pROC::roc(y1, p1)$auc, 3))
)
d2 <- data.frame(
Category = as.factor(y2),
Predictions = p2,
group = paste("AUC =", round(pROC::roc(y2, p2)$auc, 3))
)
d3 <- data.frame(
Category = as.factor(y3),
Predictions = p3,
group = paste("AUC =", round(pROC::roc(y3, p3)$auc, 3))
)
d <- rbind(d1, d2, d3)
ggplot(d, aes(x = Predictions, fill = Category)) +
geom_density(alpha = 0.25) +
facet_grid(rows = vars(group)) +
theme(legend.position="bottom")