Relationship between AUC of ROC curve and AUC of PR (precision-recall) curve

Question

I know that both the ROC curve and the PR curve can be used to evaluate the performance of a binary classification prediction model, and PR curve is preferred in the case of imbalanced class distribution. In this regard, I've got a question but couldn't figure out the answer by myself nor could find any in the textbook or other sources.

Here is my question:

Let's assume that the test set is fixed and we have two binary classification models, A and B. If the model A has a larger AUC in terms of ROC curve than the model B, does it necessarily mean that the model A has a larger AUC in terms of PR curve as well? In other words, does the AUC of ROC curve has a monotonic relationship with the AUC of PR curve given a fixed test set? If it's not the case, I would greatly appreciate if some counterexamples can be shown.

Thank you in advance!

Answer 1

NO

I have created an example where one set of predictions has a higher ROCAUC while the other set of predictions has a higher PRAUC.

library(PRROC)
set.seed(2023)
N <- 10000
p0 <- rbeta(N, 1/2, 1/2)
p1 <- rbeta(N, 2, 2)
y0 <- rep(0, N)
y1 <- rep(1, N)
y <- c(y0, y1)
p <- c(p0, p1)
roc1 <- PRROC::roc.curve(p, weights.class0 = y, curve = T) # ROCAUC = 0.5051313 
prc1 <- PRROC::pr.curve(p, weights.class0 = y, curve = T)  # PRAUC = 0.4523495

p2 <- runif(N*2, 0, 1)

roc2 <- PRROC::roc.curve(p2, weights.class0 = y, curve = T) # ROCAUC = 0.4892505  
prc2 <- PRROC::pr.curve(p2, weights.class0 = y, curve = T)  # PRAUC = 0.4930516 
roc2
prc2

For given test set values y, let p and p2 be the predictions from two models. The p predictions have a higher ROCAUC than the p2 predictions. However, the p2 predictions have a higher PRAUC than the p predictions, giving the desired example.

It would be kind of boring if PR curves and ROC curves always agreed. The whole point of having PR curves is to assess something different about the predictions than what the ROC curves assess.