26
$\begingroup$

THE BOUNTY

As promised, a bounty of $250$ points has been issued. A bounty-worthy answer should address the apparent controversy in the answers here that ROC curve interpretation does not depend on class ratio, yet ROC curves likely do not address the questions of interest in an imbalanced problem, especially in light of the relationship between ROC curves and Wilcoxon testing the predictions made for each of the two categories (a rather reasonable measure of how well the categories are distinguished from each other).

ORIGINAL QUESTION

Cross Validated has a rather thorough debunking of class imbalance being an inherent problem that must be fixed in order to do quality predictive modeling of categorical outcomes [1, 2]. However, there are measures of model performance that can be misleading when there is imbalance. The obvious one, whose (mis)use seems to drive many misconceptions about class imbalance, is that high classification accuracy need not correspond with a quality model. Yes, $99\%$ classification accuracy sounds like an $\text{A}$ in school, yet if the imbalance is $1000$$:$$1$, you could score higher classification accuracy just by predicting the majority category every time.

Another measure of performance that has been claimed to have issues in imbalanced problems is the area under the receiver-operator characteristic (ROC) curve. I struggle to see why this would be the case. The imbalance is just the prior probability of class membership, and altering the prior leads to a monotonic transformation of the predicted probability values, leaving the ROC curve unchanged. When I have simulated ROC curves under imbalance, I have gotten basically the same curves no matter the class ratio. Area under the ROC curve is related to Wilcoxon testing the two groups of predictions, and there is nothing inherently wrong with using a Wilcoxon test when the group sizes are uneven. Finally, Fawcet (2006) says that ROC curves are not sensitive to the class ratio (see the beginning of section 4.2 as well as figure 5).

Despite this, data science seems to believe that ROC curves are problematic or illegitimate when the categories are imbalanced. Even Cross Validated and the Data Science Stack seem to give mixed results on this topic.

The accepted answer here argues against ROC curves in imbalanced settings.

Harrell's answer here argues that there is no issue.

A post on data science argues that ROC curves are problematic in imbalanced settings, citing an ACM publication that states this.

The accepted answer here says that the ROC curve does not depend on the class ratio but that PR curves may answer the more interesting questions.

Have I missed something about why ROC curves are problematic when the classes are imbalanced? If my stance is correct that imbalance poses no problem for ROC curves, why does this misconception exist and persist?

My guesses for why this misconception exists and persists (if it is a misconception) are:

  1. There is a general misunderstanding of class imbalance among practitioners, perhaps disliking the very real possibility of a high AUC yet all observations classified as the majority class according to the software-default argmax decision rule.

  2. Class imbalance is associated with issues that do degrade ROC curves, even if the imbalance isn’t the direct cause. For instance, if imbalance leads to neural network optimization not converging like it would with balanced classes, there is a sub-optimal solution for the model parameters, leading to worse predictions (in some sense) and, perhaps, affecting the ROC curve. In this case, the ROC curve would be fine if we let the optimization run forever and reach the global minimum that we want it to reach, but we train our models in finite time and get predictions from those suboptimal models.

REFERENCE

Fawcett, Tom. "An introduction to ROC analysis." Pattern Recognition Letters 27.8 (2006): 861-874.

EDIT

I have found a few articles online about why ROC curves are problematic when there is imbalance. So far, they leave me with one of two thoughts.

  1. If you find ROC curves problematic in the imbalanced setting but fine in the balanced setting, you're using ROC curves in the balanced setting to tell you something that they do not claim to tell you. For instance, this article claims that precison-recall curves are more useful than ROC curves if you view your task as information retrieval. However, this is not a matter of class imbalance: if you want to view your task as selecting the $A$s from a mix of $A$s and $B$s, then precision-recall curves might just be more informative.

  2. There are issues when the raw count of the minority class is small, not when there are just relatively few of one category vs the other. For instance, this article gives an example with just ten observations of the minority category, and this article says that "a small number of correct or incorrect predictions can result in a large change in the ROC Curve or ROC AUC score," the effect of which will be lessened by increasing the sample size. I could buy this as being an example of what I wrote earlier about the imbalance itself not being a problem but imbalanace being associated with a problem, in this case, a low count of minority-class observations.

$\endgroup$
7
  • 3
    $\begingroup$ I will be issuing a bounty of 250 points that seeks a "canonical answer" to this controversial issue. $\endgroup$
    – Dave
    Commented Aug 31, 2023 at 16:38
  • 4
    $\begingroup$ I wonder whether the King & Zeng (2001) paper that Dikran Marsupial mentioned here has anything pertinent to say. I do recall it being informative, but don't know exactly what I liked so much about it... and am about to leave for a trip, so won't be able to dig into this. Looking forward to answers here! $\endgroup$ Commented Aug 31, 2023 at 17:02
  • 2
    $\begingroup$ Misconception possibility $\#1$ seems to what is considered here. My response is that the ROC curve evaluates the predictions made by a model that outputs on a continuum, while the accuracy evaluates predictions made by that model followed by a decision rule about how to use those model predictions, likely an argmax decision rule that is used simply because it is the software default in some kind of predict method. If you're tuning the threshold and find it problematic to misclassify minority points, you should pick a different threshold. $\endgroup$
    – Dave
    Commented Aug 31, 2023 at 19:07
  • $\begingroup$ I would question whether there are too many questions here? a) is Area under ROC a good metric? b) is it affected by class imbalance? c) should it be affected by class imbalance? ... $\endgroup$
    – seanv507
    Commented Sep 1, 2023 at 8:59
  • $\begingroup$ You write that class imbalance is considered by some a problem, but could you be more specific what the problem actually is? Yes, the problem is 'class imbalance' but what does it do that makes it a problem? Otherwise there can be many interpretations of the problem related to this question. $\endgroup$ Commented Nov 5, 2023 at 14:48

6 Answers 6

18
$\begingroup$

This is actually a very simple issue. The area under the ROC curve (AUROC) equals the Wilcoxon-Mann-Whitney-Somers concordance probability, a $U$-statistic, i.e., take all possible pairs of an observation with Y=0 and an observation with Y=1 and compute the fraction of such pairs such that the predicted value when Y=1 exceeded the predicted value when Y=0. You can then see that AUROC conditions on Y so AUROC cannot have an altered meaning depending on the relative frequencies of Y=0 and Y=1. The only "harm" that imbalance can cause is a higher standard error of the concordance probability ($c$-index) which is just a fact to live with. "Balancing" a dataset will only raise the estimate of its standard error.

Likewise every point on the ROC curve conditions on Y so the entire curve is conditional on Y. Each point is made up of probabilities like $\Pr(X > x | Y=y)$ ($y=0$ for x-axis, $y=1$ for y-axis).

$\endgroup$
13
  • 1
    $\begingroup$ I was hoping I’d hear from you. I’ve got a few points I’d like to clarify. First, why would balancing raise the standard error? I can see why downsampling the majority class would do that (fewer total points), but it is not clear why SMOTE or ROSE would raise the standard error when they raise the total number of points. $\endgroup$
    – Dave
    Commented Sep 2, 2023 at 14:47
  • 3
    $\begingroup$ The SE will always reduce when one of the categories is larger. It reduces slower and slower as the ratio gets beyond 3:1. But it still goes down. Likewise if you started with 10:1 frequency ratio and you threw away observations to make it 2:1 the real and estimated SE will increase. It never pays to through away samples that have already been paid for. $\endgroup$ Commented Sep 2, 2023 at 15:38
  • 6
    $\begingroup$ That's what we've been talking about. A really bad idea. Never disrespect the sample you are analyzing. $\endgroup$ Commented Sep 2, 2023 at 16:52
  • 3
    $\begingroup$ Changing the distribution of $Y$ does not change what is being estimated. Downsampling clearly increases SEs because the sample size is reduced. Upsampling means sampling the real data with replacement which adds no new information because of duplicate observations, so the real SE stays the same even though the apparent SE will be falsely low. More importantly upsampling like downsampling destroys the meaning of the data and will make the result not apply to future samples that are not over- or under-sampled. $\endgroup$ Commented Sep 5, 2023 at 11:41
  • 4
    $\begingroup$ Upsampling is like taking a dataset we run OLS on, copying it once, or twice, or 100 times, and then re-running the model on the "augmented" dataset. The parameter estimates will stay the same, but the standard errors will "decrease" - but these "reduced" SEs do not reflect more precision in the estimation, but a spurious certainty we do not have. (I will never understand why upsampling is seriously proposed in the ML/classification community, but for some strange reason, nobody in the medical statistics community has ever thought of creating "knowledge" out of thin air in this way.) $\endgroup$ Commented Sep 6, 2023 at 14:07
11
+250
$\begingroup$

This seems to me to be a misunderstanding of the criticism of AUROC in the (strongly) imbalanced case. Rephrasing the argument made by Saito and Rehmsmeier, it's not that AUROC is affected by class (im)balance - this has been thoroughly discussed/debunked by the other excellent answers - but that AUROC may, in the strongly imbalanced case, be less aligned with what one is actually interested in. In fact, Saito and Rehmsmeier argue that the problem is precisely that AUROC is not affected by class imbalance.

Consider the following simple example: at a specific decision threshold, a classifier has $\mathrm{TPR}=0.9$ and $\mathrm{FPR}=0.1$.

Scenario A, fully balanced, $n_{\text{pos,A}}=n_{\text{neg,A}}=1000$: this results in 100 false negative (FN) predictions and 100 false positive (FP) predictions, and a positive predictive value (PPV) / precision of 0.9.

Scenario B, imbalanced, $n_{\text{pos,B}}=1000$ and $n_{\text{neg,B}}=10\,000$: this still results in 100 FNs but now we have 1000 FPs and a PPV/precision of only 0.47.

Now, while this was only considering one particular point along the ROC curve, we can observe by an analogous argument that precision will be reduced in the imbalanced case compared to the balanced case at every single point along the ROC curve.1

The two scenarios A and B are certainly different in some sense, and AUROC is simply not designed to reflect this difference. (This does not mean that it is "broken" or anything; it is just not designed for this purpose.)

The argument then goes that in strongly imbalanced cases, the information conveyed in, e.g., a PR curve may be more closely aligned with the notion of model performance that practitioners are interested in.

Two other, related factoids:

  1. Cortes and Mohri (2003), Eq. (7) and Fig. 3 show how the relationship between expected AUROC and error rate / accuracy depends on the class balance. Kwegyir-Aggrey et al. (2023) also have some simple experiments illustrating this. This again indicates that in a certain sense, identical AUROC values (unsurprisingly) "mean" different things at different class imbalance ratios.
  2. Hand and Anagnostopoulos (2023) (and earlier work by them) show how AUROC can be understood as an expected misclassification loss, where the cost of the different error types (FP/FN) is implicitly defined and depends both on the used classifier (!) and the classification problem at hand, including its class distribution. (They propose their H-measure as a supposedly superior alternative, which has been discussed a few times on stats.SE.)

To summarize:

  • No, AUROC is not affected by class imbalance.
  • Some people argue that precisely this is a problem because the metric becomes less aligned with an intuitive notion of classifier performance in the strongly imbalanced case.

References:


1Pick any point $(\mathrm{TPR}, \mathrm{FPR})$ along the ROC curve. This will give us $\mathrm{TP}_A = \mathrm{TPR} \cdot n_{\text{pos,A}}$ and $\mathrm{FP}_A = \mathrm{FPR} \cdot n_{neg,A}$ for scenario A (balanced), whereas for scenario B (imbalanced), we obtain $\mathrm{TP}_B = \mathrm{TPR} \cdot n_{\text{pos,B}} = \mathrm{TP}_A$ and $\mathrm{FP}_B = \mathrm{FPR} \cdot n_{neg,B} = 10 \,\,\mathrm{FP}_A$. Since $\mathrm{PPV} = P(y{=}1 \mid \hat{y}{=}1) = \mathrm{TP} / (\mathrm{TP} + \mathrm{FP})$, we have $\mathrm{PPV}_B = \mathrm{TP}_B / (\mathrm{TP}_B + \mathrm{FP}_B) < \mathrm{TP}_A / (\mathrm{TP}_A + \mathrm{FP}_A) = \mathrm{PPV}_A \; \forall \, (\mathrm{TPR}, \mathrm{FPR}).$

$\endgroup$
38
  • 2
    $\begingroup$ "less aligned with what one is actually interested in." indeed! The problem is that the practitioner often has not really thought through what exactly they are interested in, and have not set up the analysis to answer the correct question (for instance using methods that assume by default that the misclassification costs are equal). IMHO that is the imbalanced learning problem problem! ;o) $\endgroup$ Commented Nov 5, 2023 at 21:13
  • 3
    $\begingroup$ This whole discussion has drifted so far from decision making that it’s’ astounding. Optimum decisions are based on utility/cost/loss functions and the probability of outcomes, the latter coming from a probability model that respects the original sample sizes. $\endgroup$ Commented Nov 6, 2023 at 12:37
  • 2
    $\begingroup$ When SVM doesn’t result in a continuous output or at least a “we don’t know” output then yes I’d write it off. Any method that is a forced-choice classification without a gray zone is very problematic: fharrell.com/post/classification . The fact that ML has taken a bad path should never be excused by “the wisdom of the crowd”. $\endgroup$ Commented Nov 6, 2023 at 16:41
  • 2
    $\begingroup$ I do not feel a total resolution to my concerns, but Saito and Rehmsmeier argue that the problem is precisely that AUROC is not affected by class imbalance is a provocative and useful comment, and that gets you the bounty. +250 for you! (I meant to post this yesterday when I awarded the bounty.) $\endgroup$
    – Dave
    Commented Nov 9, 2023 at 15:24
  • 2
    $\begingroup$ @Dave Thank you for the bounty and the very interesting question that has certainly incited heated debates! ;-) $\endgroup$
    – Eike P.
    Commented Nov 9, 2023 at 16:50
9
$\begingroup$

Whether an ROC curve depends on class imbalance or not will depend on which ROC curve you are referring to.

More specifically, in this answer, we will consider two types of ROC curves: "theoretical" and "practical". The theoretical ROC curve can be motivated by a simple example, while the practical ROC curve is an estimate of the theoretical ROC curve.

Theoretical ROC curve

Consider the Bernoulli random variable $Y$, which we do not directly observe. Suppose we instead observe $X \in \mathbb R$, which is related to $Y$ via the conditional pdf $f_{X \mid Y}(x \mid Y = y)$. Furthermore, we choose a function $g : \mathbb R \to \mathbb R$ and define the test statistic $\tilde X = g(X)$. Given $\tilde X$, we now want to decide whether $Y = 0$ or $Y = 1$. We denote our decision by the Bernoulli random variable $\hat Y$, such that $$ \hat Y = \begin{cases} 0, \tilde X < T \\ 1, \tilde X \geq T\end{cases} $$ for some threshold $T \in \mathbb R$. We see that each unique value of $T$ defines a unique decision rule. To summarize, we have the following chain of conditional independencies: $$ Y \to X \to \tilde X \to \hat Y $$ Keep this chain in mind as it will be easier to remember which random variables are which throughout this answer.

Ideally, because $\tilde X = g(X)$ depends on our chosen function $g$ (which could be thought of as a "model"), then we would want $g$ to perform well (in some sense) for all values of $T$. This is the ultimate goal of computing the area under the ROC curve (AUROC).

Suppose we wanted to evaluate how well some function $g$ performs as a "model". To do so, we would need to compare our decision $\hat Y$ to the ground truth $Y$, and so we can no longer assume that $Y$ is unobserved. This is an important step, as we are no longer in the sample space where the prior probabilities $\Pr(Y = 0)$ and $\Pr(Y = 1)$ have any meaning. We are either in the sample space where $Y = 0$, or in the sample space where $Y = 1$.

Going back to the ROC curve, to precisely define it, and to be able to evaluate $g$ for an individual decision rule defined by $T$, we consider the following probabilities, $$ \begin{align} P_F(T) &= \Pr(\hat Y = 1 \mid Y = 0) \\ &= \Pr(\tilde X \geq T \mid Y = 0) \\ &= \Pr(g(X) \geq T \mid Y = 0) \\ P_D(T) &= \Pr(\hat Y = 1 \mid Y = 1) \\ &= \Pr(\tilde X \geq T \mid Y = 1) \\ &= \Pr(g(X) \geq T \mid Y = 1) \\ \end{align} $$ where $P_F(T)$ is the probability of a false alarm, and $P_D(T)$ is the probability of a correct detection. I am purposefully not calling these "false positive rate" (FPR) and "true positive rate" (TPR), as we will see soon that the FPR and TPR are only estimates of $P_F(T)$ and $P_D(T)$ respectively.

Finally, we define the "theoretical" ROC curve as the parametric curve $\text{ROC}(T) = (P_F(T),P_D(T))$. If we let $f_{\tilde X \mid Y}(\tilde x \mid Y = y)$ be the conditional pdf of $\tilde X = g(X)$ given $Y = y$, then $$ \begin{align} P_F(T) &= \Pr(g(X) \geq T \mid Y = 0) \\ &= \int_T^\infty f_{\tilde X \mid Y}(\tilde x \mid Y = 0) \ d\tilde x \\ P_D(T) &= \Pr(g(X) \geq T \mid Y = 1) \\ &= \int_T^\infty f_{\tilde X \mid Y}(\tilde x \mid Y = 1) \ d\tilde x \\ \end{align} $$ We can see from these definitions that $\text{ROC}(T)$ is independent of $\Pr(Y = 0)$ and $\Pr(Y = 1)$. Furthermore, because the AUROC depends on $\text{ROC}(T)$, then the AUROC is also independent of these prior probabilities.

Practical ROC curve

In practice, however, to evaluate $g$, we are given a set of jointly i.i.d observations of the pair $(Y,X)$, which are generated using an unknown prior probability $\Pr(Y = 1)$ and unknown conditional pdfs $f_{X \mid Y}(x \mid Y = 0)$ and $f_{X \mid Y}(x \mid Y = 1)$. We then pass the given observations of $X$ through the function $g$ to generate the corresponding observations of $\tilde X$. Finally, for a chosen threshold $T$, we apply the corresponding decision rule for each observation of $\tilde X$ to generate the observations of $\hat Y$.

Going back to the concepts of FPR and TPR we mentioned above, in practice, we compute these as $$ \begin{align} \mathrm {TPR} &= {\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FN} }} \\ \mathrm {FPR} &= {\frac {\mathrm {FP} }{\mathrm {FP} +\mathrm {TN} }} \\ \end{align} $$ for different values of $T$. We then plot $(\text{FPR},\text{TPR})$ for different values of $T$, which we refer to as the "practical" ROC curve.

It can be shown that the formulae above for TPR and FPR, evaluated for a given value of $T$, are the maximum likelihood estimates of $P_D(T)$ and $P_F(T)$ respectively. Because maximum likelihood estimates, in general, depend on the sample size and empirical distribution of the samples, then we see that the TPR and FPR estimates will depend on $\Pr(Y = 1)$ and $\Pr(Y = 0)$. Note that these estimates can be obtained via other methods besides maximum likelihood estimation. However, if our sample size is large enough, then we will see that the dependence on $\Pr(Y = 1)$ and $\Pr(Y = 0)$ will become less noticeable.

We demonstrate all of this with a numerical experiment in Python. In the following code, we sample $Y$ from a Bernoulli distribution with mean $p = 0.1,0.5,0.9$. Next, we generate the observation $X$ as $$ X = Y + Z $$ where $Z \sim N(0,1)$. We then set $\tilde X = X$, and finally compute $\hat Y$ for different threshold values $T$. We run this experiment for sample sizes of $N = 20,100,1000,10000$. The corresponding ROC curves are shown in the figure below. We can see that, for $N=20$, the differences between the curves for different values of $p$ are significant. However, as $N$ gets larger, the ROC curves start to get closer to each other. Eventually, as $N \to \infty$, they will converge to the same curve, regardless of the value of $p$.

enter image description here

import numpy as np
import matplotlib.pyplot as plt

fig, ax = plt.subplots(2, 2, sharex=True, sharey=True, constrained_layout=True)
rng = np.random.default_rng(seed=42)
N = [20, 100, 1000, 10000]
num_thresholds = 100
T = np.linspace(start=-5, stop=5, num=num_thresholds)
p = [0.1, 0.5, 0.9]
color = ["red", "green", "blue"]

for i in range(len(N)):
    ax[np.unravel_index(i, ax.shape)].set_title(f"N = {N[i]}")
    for ii in range(len(p)):
        roc = np.empty(shape=(num_thresholds, 2))
        Y = rng.binomial(n=1, p=p[ii], size=N[i])
        Z = rng.normal(loc=0, scale=1, size=N[i])
        X = Y + Z
        for iii in range(num_thresholds):
            Y_hat = X >= T[iii]
            TP = np.sum(np.logical_and(Y == 1, Y_hat == 1))
            FP = np.sum(np.logical_and(Y == 0, Y_hat == 1))
            TN = np.sum(np.logical_and(Y == 0, Y_hat == 0))
            FN = np.sum(np.logical_and(Y == 1, Y_hat == 0))
            TPR = TP / (TP + FN)
            FPR = FP / (FP + TN)
            roc[iii] = (FPR, TPR)
        ax[np.unravel_index(i, ax.shape)].plot(
            roc[:, 0],
            roc[:, 1],
            label=f"p = {p[ii]}",
            color=color[ii],
            linewidth=3,
            marker="o",
            markersize=5,
        )
    ax[np.unravel_index(i, ax.shape)].legend()
plt.show(block=True)
$\endgroup$
5
  • $\begingroup$ This is a good start -- as I understand, your argument is that the appropriateness of AUC under class imbalance depends on sample size. But then isn't the solution reporting error bars around the ROC curves (e.g., via bootstraped TPR across thresholds achieving FPR) -- making it clear that sample size is the "culprit," not AUC as a metric? What if you remake your plots using AUPR (the "recommended" alternative metric for class-imbalanced data)? $\endgroup$ Commented Sep 2, 2023 at 1:44
  • 1
    $\begingroup$ @chang_trenton I haven’t dealt too much with PR curves, but my understanding is that we are still computing maximum likelihood estimates of precision and recall for different threshold values, which means that the dependence of the AUPR on class imbalance should still depend on the sample size. What I’m essentially arguing in my answer is that, if an oracle gave us the theoretical values instead of maximum likelihood estimates, then both the ROC and the PR curve will be independent of class probabilities. $\endgroup$
    – mhdadk
    Commented Sep 2, 2023 at 1:49
  • 2
    $\begingroup$ The essence of this is what I found in my simulations, that it isn’t about the imbalance ratio but the raw number of members of the minority class. This is basically what Dikran Marsupial writes here, that class imbalance on its own isn’t a problem but can lead to a smaller effective sample size for $N$ total observations than would be available for a balanced problem. // Ia the reference to maximum likelihood estimation necessary? There are many other ways to fit predictive models that can be analyzed using ROC curves. $\endgroup$
    – Dave
    Commented Sep 2, 2023 at 1:50
  • $\begingroup$ @Dave yes, the reference to maximum likelihood estimation is necessary, as this is where the dependence of practical ROC curves on the class probability $\Pr(Y = 1)$ comes from. Any time we compute a maximum likelihood estimate of a probability metric, such as precision, recall, sensitivity, or specificity, any consequent computations will depend on this estimate, including ROC and PR curves. $\endgroup$
    – mhdadk
    Commented Sep 2, 2023 at 9:45
  • 1
    $\begingroup$ @mhdadk "if an oracle gave us the theoretical values instead of maximum likelihood estimates, then both the ROC and the PR curve will be independent of class probabilities" - this is not true for the PR curve, as far as I know. The class balance determines the baseline of the PR curve, and so e.g. the random classifier has different PR curves and different AUPR depending on the class ratios, even leaving finite-sample issues aside. See e.g. Saito and Rehmsmeier or Flach and Kull. $\endgroup$
    – Eike P.
    Commented Sep 5, 2023 at 19:52
3
$\begingroup$

"Despite this, data science seems to believe that ROC curves are problematic or illegitimate when the categories are imbalanced."

This is because many in the data science community seem to think that class imbalance a an inherent problem, and ROC curves, and specifically the AUROC statistic "hide" the problem.

The real problem is usually cost-sensitive learning. If your classifier classifies everything as belonging to the majority class, it may well be that is simply the optimal solution if the misclassification costs are equal. There is no class imbalance problem here, how can there be a problem if the classifier is behaving optimally for the question as posed?

If this isn't acceptable for the practical application, it means that the minority class is "more important" in some sense than the majority class, so the practitioner should go and work out plausible values for the misclassification cost and build those into the classifier (preferably by using a probabilistic classifier and adjusting the threshold).

ROC analysis can help with this (the slope of the tangent line to the curve gives the ratio of misclassification costs IIRC).

The AUROC is a useful statistic where you are only interested in the ranking of patterns, perhaps because the misclassification costs are unknown or the operational class fequencies are unknown, and therefore you can't know the ideal threshold and hence can't use any statistic based on that threshold (such as accuracy or F1 or ...).

We need to understand the problem we are trying to solve, and work out what we are really interested in, and then choose a suitable performance metric based on that (rather than focus on characteristics of the data, such as imbalance).

$\endgroup$
2
$\begingroup$

The problem

Area under the ROC curve when there is imbalance: is there a problem?

I believe that the answer is that ROC curves and AUC are already more generally a problem, with or without class imbalance.

ROC curves display the performance of classifiers for a wide range of true and false positive rates, but often only a small part of that range is of interest. So an ROC curve and especially a simplified statistic like AUC may not need to be of much use.

A related question to problems with AUC and whether they can be applied without any other considerations is the question: Is higher AUC always better? Sidenote occuring in that question: aside from the considerations with plainly comparing the statistic, you also have considerations about the costs of using a classifier and the accuracy of the estimates of the ROC curve.

The rumor

if not, why does this rumor exist?

The more general principle for comparison of classifiers is the cost function and the matter which classifier optimizes this. A simple type of cost is used in the question: Are non-crossing ROC curves sufficient to rank classifiers by expected loss?

In that question the expectation of the cost is a product of several terms

$$\begin{array}{} E[Loss] &=& p_{Y=1} (1-f_{TP}) a + p_{Y=0} (f_{FP}) b \\ \end{array}$$

  • $p_{Y=1}$ and $p_{Y=0}$ are the class frequencies
  • $f_{TP}$ and $f_{FP}$ are the true and false positive rates.
  • $a$ and $b$ are the costs of making a false classification

The class frequencies (in the first bullet point) play a role in the expected loss of a particular classifier. This opens the door for people to consider situations where $p_{Y=1}$ and $p_{Y=0}$ are a lot different (class imbalance). And that may be a reason why class balance is being discussed in relation to AUC and ROC. But, the situation is more generally about the entire cost function. The class imbalance is just a part of the story.

Class imbalance plays a role, but it is not the only factor. The links in the question like Unbalanced Data? Stop Using ROC-AUC and Use AUPRC Instead speaks about imbalance, but are actually more related to the more general principle of the cost function and happen in the example to be about class balance.

Depending on the costs $a$ and $b$ of the types of miss classifications the imbalance may be either good or bad. There is not something special to balanced classes. It just happens to be a starting point that people often discuss.

More about class imbalance

Another way how class imbalance can become part of the rumour is because it is a popular topic. Sometimes it can be really a problem, but then it is not about the AUC which is a much more general problem. An example occurs in the question Was Amazon's AI tool, more than human recruiters, biased against women? where class imbalance is a mechanism for the bias in classifiers towards particular classes. In this case it is not about imbalance in positive versus negative cases, but about imbalance in additional classes/variables like gender. If a model is trained on mostly particular set of examples, then it may perform badly in predicting examples that are different. E.g. an algorithm that is used for headhunting new employees may give an advantage to men over women when it has been trained on data with mostly men.

$\endgroup$
1
$\begingroup$

The most informative metric I use is the Matthews Correlation Coefficient (MCC), which captures the true balance between positive and negative classifications, minimizing errors. Four out of the top six MCC values came from models utilizing combined z-scores. While the AUC is often used as a performance metric, it can be misleading, especially in datasets with high disease prevalence. The MCC is better than the AUC.

For instance, an 80% accuracy in a population with 74% prevalence is not a notable achievement. The MCC, which aggregates true positives, negatives, false positives, and negatives, offers a more truthful representation, especially when both true positives and negatives are equally significant.

Matthews correlation coefficient (MCC) the MCC is an especially reliable metric when evaluating binary categories in datasets where the number of disease cases does not match non-disease cases. High MCC scores are only achieved when predictions accurately classify a significant proportion of diseased and non-diseased patients, regardless of any class imbalance.

Use the MCC instead of the ROC.

References

Chicco D, Jurman G. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics 2020: 21(1): 6.

Chicco D, Totsch N, Jurman G. The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation. BioData Min 2021: 14(1): 13.

Boughorbel S, Jarray F, El-Anbari M. Optimal classifier for imbalanced data using Matthews Correlation Coefficient metric. PLoS One 2017: 12(6): e0177678.

$\endgroup$
11
  • 3
    $\begingroup$ Welcome to Cross Validated! As I wrote in one of the comments to my question, the ROC curve and the ROCAUC apply to a model, and then metrics like accuracy and this MCC you like apply to a pipeline of such a model followed by a decision rule applied to the model outputs. However, the “ROC curve” (if you can even call it a curve) for a set of classifications such as those assessed by this MCC is a single point, not an entire curve of sensitivity-specificity pairs. Thus, I cannot buy an argument like that presented here. Of course MCC can disagree with AUC. Different models are considered! $\endgroup$
    – Dave
    Commented Nov 6, 2023 at 4:57
  • 1
    $\begingroup$ "For instance, an 80% accuracy in a population with 74% prevalence is not a notable achievement." that depends on how difficult the classification task actually is. It may be that 80% is the Bayes optimal accuracy. This answer doesn't actually say why the AUC is misleading. It isn't misleading due to class imbalance, which affects the threshold and the AUC doesn't measure the performance of the threshold value, so it is not misleading as it doesn't claim to do that. $\endgroup$ Commented Nov 6, 2023 at 7:47
  • 1
    $\begingroup$ @mhdadk And then you have a collection of MCC values, not just one. $\endgroup$
    – Dave
    Commented Nov 6, 2023 at 11:19
  • 1
    $\begingroup$ @mhdadk I like the idea of plotting threshold-based metrics according to the threshold. There’s still more model evaluation than this (such an approach gives no commentary on calibration, for instance), but it gives some sense of how well the mode is performing without being held to a particular threshold. Maybe the software-default threshold leads to some unacceptable performance, but if there is more acceptable performance at another threshold, that is at least a positive sign about your ability to discriminate between the categories. (However, that isn’t being advocated for in this answer.) $\endgroup$
    – Dave
    Commented Nov 6, 2023 at 13:06
  • 1
    $\begingroup$ Once you use thresholds that are not expected-utility-derived you become inconsistent with how decisions are made. $\endgroup$ Commented Nov 8, 2023 at 12:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.