Area under the ROC curve when there is imbalance: is there a problem, and if not, why does this rumor exist?

Question

THE BOUNTY

As promised, a bounty of $250$ points has been issued. A bounty-worthy answer should address the apparent controversy in the answers here that ROC curve interpretation does not depend on class ratio, yet ROC curves likely do not address the questions of interest in an imbalanced problem, especially in light of the relationship between ROC curves and Wilcoxon testing the predictions made for each of the two categories (a rather reasonable measure of how well the categories are distinguished from each other).

ORIGINAL QUESTION

Cross Validated has a rather thorough debunking of class imbalance being an inherent problem that must be fixed in order to do quality predictive modeling of categorical outcomes [1, 2]. However, there are measures of model performance that can be misleading when there is imbalance. The obvious one, whose (mis)use seems to drive many misconceptions about class imbalance, is that high classification accuracy need not correspond with a quality model. Yes, $99\%$ classification accuracy sounds like an $\text{A}$ in school, yet if the imbalance is $1000$$:$$1$, you could score higher classification accuracy just by predicting the majority category every time.

Another measure of performance that has been claimed to have issues in imbalanced problems is the area under the receiver-operator characteristic (ROC) curve. I struggle to see why this would be the case. The imbalance is just the prior probability of class membership, and altering the prior leads to a monotonic transformation of the predicted probability values, leaving the ROC curve unchanged. When I have simulated ROC curves under imbalance, I have gotten basically the same curves no matter the class ratio. Area under the ROC curve is related to Wilcoxon testing the two groups of predictions, and there is nothing inherently wrong with using a Wilcoxon test when the group sizes are uneven. Finally, Fawcet (2006) says that ROC curves are not sensitive to the class ratio (see the beginning of section 4.2 as well as figure 5).

Despite this, data science seems to believe that ROC curves are problematic or illegitimate when the categories are imbalanced. Even Cross Validated and the Data Science Stack seem to give mixed results on this topic.

The accepted answer here argues against ROC curves in imbalanced settings.

Harrell's answer here argues that there is no issue.

A post on data science argues that ROC curves are problematic in imbalanced settings, citing an ACM publication that states this.

The accepted answer here says that the ROC curve does not depend on the class ratio but that PR curves may answer the more interesting questions.

Have I missed something about why ROC curves are problematic when the classes are imbalanced? If my stance is correct that imbalance poses no problem for ROC curves, why does this misconception exist and persist?

My guesses for why this misconception exists and persists (if it is a misconception) are:

1. There is a general misunderstanding of class imbalance among practitioners, perhaps disliking the very real possibility of a high AUC yet all observations classified as the majority class according to the software-default argmax decision rule.

2. Class imbalance is associated with issues that do degrade ROC curves, even if the imbalance isn’t the direct cause. For instance, if imbalance leads to neural network optimization not converging like it would with balanced classes, there is a sub-optimal solution for the model parameters, leading to worse predictions (in some sense) and, perhaps, affecting the ROC curve. In this case, the ROC curve would be fine if we let the optimization run forever and reach the global minimum that we want it to reach, but we train our models in finite time and get predictions from those suboptimal models.

REFERENCE

Fawcett, Tom. "An introduction to ROC analysis." Pattern Recognition Letters 27.8 (2006): 861-874.

EDIT

I have found a few articles online about why ROC curves are problematic when there is imbalance. So far, they leave me with one of two thoughts.

1. If you find ROC curves problematic in the imbalanced setting but fine in the balanced setting, you're using ROC curves in the balanced setting to tell you something that they do not claim to tell you. For instance, this article claims that precison-recall curves are more useful than ROC curves if you view your task as information retrieval. However, this is not a matter of class imbalance: if you want to view your task as selecting the $A$s from a mix of $A$s and $B$s, then precision-recall curves might just be more informative.

2. There are issues when the raw count of the minority class is small, not when there are just relatively few of one category vs the other. For instance, this article gives an example with just ten observations of the minority category, and this article says that "a small number of correct or incorrect predictions can result in a large change in the ROC Curve or ROC AUC score," the effect of which will be lessened by increasing the sample size. I could buy this as being an example of what I wrote earlier about the imbalance itself not being a problem but imbalanace being associated with a problem, in this case, a low count of minority-class observations.

Answer 1

This is actually a very simple issue. The area under the ROC curve (AUROC) equals the Wilcoxon-Mann-Whitney-Somers concordance probability, a $U$-statistic, i.e., take all possible pairs of an observation with Y=0 and an observation with Y=1 and compute the fraction of such pairs such that the predicted value when Y=1 exceeded the predicted value when Y=0. You can then see that AUROC conditions on Y so AUROC cannot have an altered meaning depending on the relative frequencies of Y=0 and Y=1. The only "harm" that imbalance can cause is a higher standard error of the concordance probability ($c$-index) which is just a fact to live with. "Balancing" a dataset will only raise the estimate of its standard error.

Likewise every point on the ROC curve conditions on Y so the entire curve is conditional on Y. Each point is made up of probabilities like $\Pr(X > x | Y=y)$ ($y=0$ for x-axis, $y=1$ for y-axis).

Answer 2

This seems to me to be a misunderstanding of the criticism of AUROC in the (strongly) imbalanced case. Rephrasing the argument made by Saito and Rehmsmeier, it's not that AUROC is affected by class (im)balance - this has been thoroughly discussed/debunked by the other excellent answers - but that AUROC may, in the strongly imbalanced case, be less aligned with what one is actually interested in. In fact, Saito and Rehmsmeier argue that the problem is precisely that AUROC is not affected by class imbalance.

Consider the following simple example: at a specific decision threshold, a classifier has $\mathrm{TPR}=0.9$ and $\mathrm{FPR}=0.1$.

Scenario A, fully balanced, $n_{\text{pos,A}}=n_{\text{neg,A}}=1000$: this results in 100 false negative (FN) predictions and 100 false positive (FP) predictions, and a positive predictive value (PPV) / precision of 0.9.

Scenario B, imbalanced, $n_{\text{pos,B}}=1000$ and $n_{\text{neg,B}}=10\,000$: this still results in 100 FNs but now we have 1000 FPs and a PPV/precision of only 0.47.

Now, while this was only considering one particular point along the ROC curve, we can observe by an analogous argument that precision will be reduced in the imbalanced case compared to the balanced case at every single point along the ROC curve.¹

The two scenarios A and B are certainly different in some sense, and AUROC is simply not designed to reflect this difference. (This does not mean that it is "broken" or anything; it is just not designed for this purpose.)

The argument then goes that in strongly imbalanced cases, the information conveyed in, e.g., a PR curve may be more closely aligned with the notion of model performance that practitioners are interested in.

Two other, related factoids:

1. Cortes and Mohri (2003), Eq. (7) and Fig. 3 show how the relationship between expected AUROC and error rate / accuracy depends on the class balance. Kwegyir-Aggrey et al. (2023) also have some simple experiments illustrating this. This again indicates that in a certain sense, identical AUROC values (unsurprisingly) "mean" different things at different class imbalance ratios.
2. Hand and Anagnostopoulos (2023) (and earlier work by them) show how AUROC can be understood as an expected misclassification loss, where the cost of the different error types (FP/FN) is implicitly defined and depends both on the used classifier (!) and the classification problem at hand, including its class distribution. (They propose their H-measure as a supposedly superior alternative, which has been discussed a few times on stats.SE.)

To summarize:

• No, AUROC is not affected by class imbalance.
• Some people argue that precisely this is a problem because the metric becomes less aligned with an intuitive notion of classifier performance in the strongly imbalanced case.

References:

• Cortes and Mohri (2003), AUC optimization vs. error rate minimization
• Saito and Rehmsmeier (2015), The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets
• Hand and Anagnostopoulos (2023), Notes on the H-measure of classifier performance
• Kwegyir-Aggrey et al. (2023), The Misuse of AUC: What High Impact Risk Assessment Gets Wrong

---

^{1Pick any point $(\mathrm{TPR}, \mathrm{FPR})$ along the ROC curve. This will give us $\mathrm{TP}_A = \mathrm{TPR} \cdot n_{\text{pos,A}}$ and $\mathrm{FP}_A = \mathrm{FPR} \cdot n_{neg,A}$ for scenario A (balanced), whereas for scenario B (imbalanced), we obtain $\mathrm{TP}_B = \mathrm{TPR} \cdot n_{\text{pos,B}} = \mathrm{TP}_A$ and $\mathrm{FP}_B = \mathrm{FPR} \cdot n_{neg,B} = 10 \,\,\mathrm{FP}_A$. Since $\mathrm{PPV} = P(y{=}1 \mid \hat{y}{=}1) = \mathrm{TP} / (\mathrm{TP} + \mathrm{FP})$, we have $\mathrm{PPV}_B = \mathrm{TP}_B / (\mathrm{TP}_B + \mathrm{FP}_B) < \mathrm{TP}_A / (\mathrm{TP}_A + \mathrm{FP}_A) = \mathrm{PPV}_A \; \forall \, (\mathrm{TPR}, \mathrm{FPR}).$}

Answer 3

Whether an ROC curve depends on class imbalance or not will depend on which ROC curve you are referring to.

More specifically, in this answer, we will consider two types of ROC curves: "theoretical" and "practical". The theoretical ROC curve can be motivated by a simple example, while the practical ROC curve is an estimate of the theoretical ROC curve.

Theoretical ROC curve

Consider the Bernoulli random variable $Y$, which we do not directly observe. Suppose we instead observe $X \in \mathbb R$, which is related to $Y$ via the conditional pdf $f_{X \mid Y}(x \mid Y = y)$. Furthermore, we choose a function $g : \mathbb R \to \mathbb R$ and define the test statistic $\tilde X = g(X)$. Given $\tilde X$, we now want to decide whether $Y = 0$ or $Y = 1$. We denote our decision by the Bernoulli random variable $\hat Y$, such that $$ \hat Y = \begin{cases} 0, \tilde X < T \\ 1, \tilde X \geq T\end{cases} $$ for some threshold $T \in \mathbb R$. We see that each unique value of $T$ defines a unique decision rule. To summarize, we have the following chain of conditional independencies: $$ Y \to X \to \tilde X \to \hat Y $$ Keep this chain in mind as it will be easier to remember which random variables are which throughout this answer.

Ideally, because $\tilde X = g(X)$ depends on our chosen function $g$ (which could be thought of as a "model"), then we would want $g$ to perform well (in some sense) for all values of $T$. This is the ultimate goal of computing the area under the ROC curve (AUROC).

Suppose we wanted to evaluate how well some function $g$ performs as a "model". To do so, we would need to compare our decision $\hat Y$ to the ground truth $Y$, and so we can no longer assume that $Y$ is unobserved. This is an important step, as we are no longer in the sample space where the prior probabilities $\Pr(Y = 0)$ and $\Pr(Y = 1)$ have any meaning. We are either in the sample space where $Y = 0$, or in the sample space where $Y = 1$.

Going back to the ROC curve, to precisely define it, and to be able to evaluate $g$ for an individual decision rule defined by $T$, we consider the following probabilities, $$ \begin{align} P_F(T) &= \Pr(\hat Y = 1 \mid Y = 0) \\ &= \Pr(\tilde X \geq T \mid Y = 0) \\ &= \Pr(g(X) \geq T \mid Y = 0) \\ P_D(T) &= \Pr(\hat Y = 1 \mid Y = 1) \\ &= \Pr(\tilde X \geq T \mid Y = 1) \\ &= \Pr(g(X) \geq T \mid Y = 1) \\ \end{align} $$ where $P_F(T)$ is the probability of a false alarm, and $P_D(T)$ is the probability of a correct detection. I am purposefully not calling these "false positive rate" (FPR) and "true positive rate" (TPR), as we will see soon that the FPR and TPR are only estimates of $P_F(T)$ and $P_D(T)$ respectively.

Finally, we define the "theoretical" ROC curve as the parametric curve $\text{ROC}(T) = (P_F(T),P_D(T))$. If we let $f_{\tilde X \mid Y}(\tilde x \mid Y = y)$ be the conditional pdf of $\tilde X = g(X)$ given $Y = y$, then $$ \begin{align} P_F(T) &= \Pr(g(X) \geq T \mid Y = 0) \\ &= \int_T^\infty f_{\tilde X \mid Y}(\tilde x \mid Y = 0) \ d\tilde x \\ P_D(T) &= \Pr(g(X) \geq T \mid Y = 1) \\ &= \int_T^\infty f_{\tilde X \mid Y}(\tilde x \mid Y = 1) \ d\tilde x \\ \end{align} $$ We can see from these definitions that $\text{ROC}(T)$ is independent of $\Pr(Y = 0)$ and $\Pr(Y = 1)$. Furthermore, because the AUROC depends on $\text{ROC}(T)$, then the AUROC is also independent of these prior probabilities.

Practical ROC curve

In practice, however, to evaluate $g$, we are given a set of jointly i.i.d observations of the pair $(Y,X)$, which are generated using an unknown prior probability $\Pr(Y = 1)$ and unknown conditional pdfs $f_{X \mid Y}(x \mid Y = 0)$ and $f_{X \mid Y}(x \mid Y = 1)$. We then pass the given observations of $X$ through the function $g$ to generate the corresponding observations of $\tilde X$. Finally, for a chosen threshold $T$, we apply the corresponding decision rule for each observation of $\tilde X$ to generate the observations of $\hat Y$.

Going back to the concepts of FPR and TPR we mentioned above, in practice, we compute these as $$ \begin{align} \mathrm {TPR} &= {\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FN} }} \\ \mathrm {FPR} &= {\frac {\mathrm {FP} }{\mathrm {FP} +\mathrm {TN} }} \\ \end{align} $$ for different values of $T$. We then plot $(\text{FPR},\text{TPR})$ for different values of $T$, which we refer to as the "practical" ROC curve.

It can be shown that the formulae above for TPR and FPR, evaluated for a given value of $T$, are the maximum likelihood estimates of $P_D(T)$ and $P_F(T)$ respectively. Because maximum likelihood estimates, in general, depend on the sample size and empirical distribution of the samples, then we see that the TPR and FPR estimates will depend on $\Pr(Y = 1)$ and $\Pr(Y = 0)$. Note that these estimates can be obtained via other methods besides maximum likelihood estimation. However, if our sample size is large enough, then we will see that the dependence on $\Pr(Y = 1)$ and $\Pr(Y = 0)$ will become less noticeable.

We demonstrate all of this with a numerical experiment in Python. In the following code, we sample $Y$ from a Bernoulli distribution with mean $p = 0.1,0.5,0.9$. Next, we generate the observation $X$ as $$ X = Y + Z $$ where $Z \sim N(0,1)$. We then set $\tilde X = X$, and finally compute $\hat Y$ for different threshold values $T$. We run this experiment for sample sizes of $N = 20,100,1000,10000$. The corresponding ROC curves are shown in the figure below. We can see that, for $N=20$, the differences between the curves for different values of $p$ are significant. However, as $N$ gets larger, the ROC curves start to get closer to each other. Eventually, as $N \to \infty$, they will converge to the same curve, regardless of the value of $p$.

import numpy as np
import matplotlib.pyplot as plt

fig, ax = plt.subplots(2, 2, sharex=True, sharey=True, constrained_layout=True)
rng = np.random.default_rng(seed=42)
N = [20, 100, 1000, 10000]
num_thresholds = 100
T = np.linspace(start=-5, stop=5, num=num_thresholds)
p = [0.1, 0.5, 0.9]
color = ["red", "green", "blue"]

for i in range(len(N)):
    ax[np.unravel_index(i, ax.shape)].set_title(f"N = {N[i]}")
    for ii in range(len(p)):
        roc = np.empty(shape=(num_thresholds, 2))
        Y = rng.binomial(n=1, p=p[ii], size=N[i])
        Z = rng.normal(loc=0, scale=1, size=N[i])
        X = Y + Z
        for iii in range(num_thresholds):
            Y_hat = X >= T[iii]
            TP = np.sum(np.logical_and(Y == 1, Y_hat == 1))
            FP = np.sum(np.logical_and(Y == 0, Y_hat == 1))
            TN = np.sum(np.logical_and(Y == 0, Y_hat == 0))
            FN = np.sum(np.logical_and(Y == 1, Y_hat == 0))
            TPR = TP / (TP + FN)
            FPR = FP / (FP + TN)
            roc[iii] = (FPR, TPR)
        ax[np.unravel_index(i, ax.shape)].plot(
            roc[:, 0],
            roc[:, 1],
            label=f"p = {p[ii]}",
            color=color[ii],
            linewidth=3,
            marker="o",
            markersize=5,
        )
    ax[np.unravel_index(i, ax.shape)].legend()
plt.show(block=True)

Answer 4

"Despite this, data science seems to believe that ROC curves are problematic or illegitimate when the categories are imbalanced."

This is because many in the data science community seem to think that class imbalance a an inherent problem, and ROC curves, and specifically the AUROC statistic "hide" the problem.

The real problem is usually cost-sensitive learning. If your classifier classifies everything as belonging to the majority class, it may well be that is simply the optimal solution if the misclassification costs are equal. There is no class imbalance problem here, how can there be a problem if the classifier is behaving optimally for the question as posed?

If this isn't acceptable for the practical application, it means that the minority class is "more important" in some sense than the majority class, so the practitioner should go and work out plausible values for the misclassification cost and build those into the classifier (preferably by using a probabilistic classifier and adjusting the threshold).

ROC analysis can help with this (the slope of the tangent line to the curve gives the ratio of misclassification costs IIRC).

The AUROC is a useful statistic where you are only interested in the ranking of patterns, perhaps because the misclassification costs are unknown or the operational class fequencies are unknown, and therefore you can't know the ideal threshold and hence can't use any statistic based on that threshold (such as accuracy or F1 or ...).

We need to understand the problem we are trying to solve, and work out what we are really interested in, and then choose a suitable performance metric based on that (rather than focus on characteristics of the data, such as imbalance).

Answer 5

The problem

Area under the ROC curve when there is imbalance: is there a problem?

I believe that the answer is that ROC curves and AUC are already more generally a problem, with or without class imbalance.

ROC curves display the performance of classifiers for a wide range of true and false positive rates, but often only a small part of that range is of interest. So an ROC curve and especially a simplified statistic like AUC may not need to be of much use.

A related question to problems with AUC and whether they can be applied without any other considerations is the question: Is higher AUC always better? Sidenote occuring in that question: aside from the considerations with plainly comparing the statistic, you also have considerations about the costs of using a classifier and the accuracy of the estimates of the ROC curve.

The rumor

if not, why does this rumor exist?

The more general principle for comparison of classifiers is the cost function and the matter which classifier optimizes this. A simple type of cost is used in the question: Are non-crossing ROC curves sufficient to rank classifiers by expected loss?

In that question the expectation of the cost is a product of several terms

$$\begin{array}{} E[Loss] &=& p_{Y=1} (1-f_{TP}) a + p_{Y=0} (f_{FP}) b \\ \end{array}$$

• $p_{Y=1}$ and $p_{Y=0}$ are the class frequencies
• $f_{TP}$ and $f_{FP}$ are the true and false positive rates.
• $a$ and $b$ are the costs of making a false classification

The class frequencies (in the first bullet point) play a role in the expected loss of a particular classifier. This opens the door for people to consider situations where $p_{Y=1}$ and $p_{Y=0}$ are a lot different (class imbalance). And that may be a reason why class balance is being discussed in relation to AUC and ROC. But, the situation is more generally about the entire cost function. The class imbalance is just a part of the story.

Class imbalance plays a role, but it is not the only factor. The links in the question like Unbalanced Data? Stop Using ROC-AUC and Use AUPRC Instead speaks about imbalance, but are actually more related to the more general principle of the cost function and happen in the example to be about class balance.

Depending on the costs $a$ and $b$ of the types of miss classifications the imbalance may be either good or bad. There is not something special to balanced classes. It just happens to be a starting point that people often discuss.

More about class imbalance

Another way how class imbalance can become part of the rumour is because it is a popular topic. Sometimes it can be really a problem, but then it is not about the AUC which is a much more general problem. An example occurs in the question Was Amazon's AI tool, more than human recruiters, biased against women? where class imbalance is a mechanism for the bias in classifiers towards particular classes. In this case it is not about imbalance in positive versus negative cases, but about imbalance in additional classes/variables like gender. If a model is trained on mostly particular set of examples, then it may perform badly in predicting examples that are different. E.g. an algorithm that is used for headhunting new employees may give an advantage to men over women when it has been trained on data with mostly men.

Answer 6

The most informative metric I use is the Matthews Correlation Coefficient (MCC), which captures the true balance between positive and negative classifications, minimizing errors. Four out of the top six MCC values came from models utilizing combined z-scores. While the AUC is often used as a performance metric, it can be misleading, especially in datasets with high disease prevalence. The MCC is better than the AUC.

For instance, an 80% accuracy in a population with 74% prevalence is not a notable achievement. The MCC, which aggregates true positives, negatives, false positives, and negatives, offers a more truthful representation, especially when both true positives and negatives are equally significant.

Matthews correlation coefficient (MCC) the MCC is an especially reliable metric when evaluating binary categories in datasets where the number of disease cases does not match non-disease cases. High MCC scores are only achieved when predictions accurately classify a significant proportion of diseased and non-diseased patients, regardless of any class imbalance.

Use the MCC instead of the ROC.

References

Chicco D, Jurman G. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics 2020: 21(1): 6.

Chicco D, Totsch N, Jurman G. The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation. BioData Min 2021: 14(1): 13.

Boughorbel S, Jarray F, El-Anbari M. Optimal classifier for imbalanced data using Matthews Correlation Coefficient metric. PLoS One 2017: 12(6): e0177678.