When is a dataset “too imbalanced” for AUC ROC and PR is preferred?

Question

I’ve read that precision-recall (PR) curves are preferred over AUC-ROC curves when a dataset is imbalanced as there’s more of a focus on the model’s performance in correctly identifying the minority/positive class.

At what point (rule of thumb?) does it make more sense to primarily use PR to evaluate a classifier instead of AUC-ROC score? I imagine if the dataset has 40% positive class, AUC is still appropriate? But what about at 30% or 20% positive class? What level is considered “imbalanced” where PR is preferred?

Answer 1

Context

The imbalance depends on the dataset size also.

A model with 5-10% positive class and 90-95% negative class with 50 or 500 samples is different from a model that has 10'000 samples.

Opinion

A model seeing 1 positive sample and trying to learn from it is different from seeing hundreds of positive samples (even if they represent only 5% of the whole data).

Anyway, as anything between 20-40% positives is considered imbalanced, too imbalanced is around 5-10%, and extremely imbalanced is below 5%.

Resampling

Multiple resampling methods exist, however, it is very tricky on whether or not they improve your model, since an increase in the recall, causes also a huge decrease in precision in most of the times (if you oversample the minority).

Answer 2

Agree with the comments, I have used AUC ROC for binary classification with a class imbalance of 5% positive and 95% negative. I was actually able to get a pretty good model still.

Answer 3

The argument for PR curves over ROC curves in imbalanced settings seems to go something like this:

1. ROCAUC is calculated to be high (for some loose definition is high).

2. Then the predictions are run through some code to calculate the classification accuracy, sensitivity, specificity, precision, or full confusion matrix, and these results are deemed to be poor.

3. Then the PRAUC is calculated and found to be somewhat low, more consistent with the poor accuracy score. In code, that would go something like...

library(ModelMetrics)
library(ROCR)
set.seed(2024)
N <- 10000
p <- rbeta(N, 1/2, 14)
y <- rbinom(N, 1, p)
phat <- p
yhat <- round(p) # Use the default threshold of 0.5
mean(y) # 3.34% are 1, the rest 0
hist(p)
preds <- ROCR::prediction(phat, y)
rounded <- ROCR::prediction(yhat, y)
rocauc <- ROCR::performance(preds, "auc")
prauc  <- ROCR::performance(preds, "aucpr")
sens <- ModelMetrics::sensitivity(y, yhat)
[email protected] # Yay, ROCAUC is a high value of 0.83!
table(yhat)     # Wait, but all of my predictions are 0...
sens            # Then my sensitivity is 0...
[email protected]  # Better use PRAUC, not ROCAUC, and I get a low value of 0.138

However...

...steps 1 and 3 use different data than step 2 and do not refer to the same predictions!

To calculate the classification accuracy, sensitivity, specificity, precision, or full confusion matrix, categorical predictions are needed. A typical way to do this, as I did in the code above, is to pick a threshold and consider all predictions above that threshold to be one category and all below that threshold to be the other category.

Consequently, the accuracy/sensitivity/specificity/precision/recall/F1 does not refer to the same predictions as do area under the ROC or PR curve. The area under the curves evaluates the raw model predictions, while the accuracy/sensitivity/specificity/precision/recall/F1 evaluate the predictions made by the two-stage pipeline of raw predictions > decision rule, which is not the same.

Now, the reason the yhat variable in the code (the mock predictions) is all zero is because the probability values in phat are all less than $0.5$. However, in an imbalanced problem, this should not be surprising. Unless you have highly compelling evidence that the outcome is likely to belong to an rather unusual category, shouldn't you believe the common category is more likely? (As far as I can tell, this is a fairly straightforward application of Bayes' theorem.)

Overall, the idea that imbalance means you should use PR curves instead of ROC curves seems to be based on not understanding the difference between evaluating raw model predictions and evaluating the predictions made by that two-stage pipeline.

Ultimately, both curves give somewhat distinct information. There is a legitimate argument to use both every time to get a sense of how well the model can distinguish between the two classes (ROC curve) and how often the model will "cry wolf" for a given level of detecting wolves are really present (PR curve). Also worth consideration, arguably more than the ROC and PR curves, is to assess model calibration and overall performance according to a strictly proper scoring rule like log loss or Brier score, which can be normalized to McFadden's and Efron's pseudo $R^2$, respectively, to ease interpretation.

I will close from a quote from the bounty-worthy answer I got when I asked a related question last year.

Saito and Rehmsmeier argue that the problem is precisely that AUROC is not affected by class imbalance.