AUC and class imbalance in training/test dataset

Question

I just start to learn the Area under the ROC curve (AUC). I am told that AUC is not reflected by data imbalance. I think it means that AUC is insensitive to imbalance in test data, rather than imbalance in training data.

In other words, only changing the distribution of positive and negative classes in the test data, the AUC value may not change much. But if we change the distribution in the training data, the AUC value may largely change. The reason is that the classifier cannot be learned well. In this case, we have to use undersampling and oversampling. Am I right? I just want to make sure my understanding on AUC is correct.

Answer 1

It depends how you mean the word sensitive. The ROC AUC is sensitive to class imbalance in the sense that when there is a minority class, you typically define this as the positive class and it will have a strong impact on the AUC value. This is very much desirable behaviour. Accuracy is for example not sensitive in that way. It can be very high even if the minority class is not well predicted at all.

In most experimental setups (bootstrap or cross validation for example) the class distribution of training and test sets should be similar. But this is a result of how you sample those sets, not of using or not using ROC. Basically you are right to say that the ROC makes abstraction of class imbalance in the test set by giving equal importance to sensitivity and specificity. When the training set doesn't contain enough examples to learn the class, this will still affect ROC though, as it should.

What you do in terms of oversampling and parameter tuning is a separate issue. The ROC can only ever tell you how well a specific configuration works. You can then try multiple config's and select the best.

Answer 2

(a 3-years late answer, but maybe still useful!)

ROC is sensitive to the class-imbalance issue, meaning that it favors the class with larger population solely because of its higher population. In other words, it is biased toward the larger population when it comes to classification/prediction.

This is indeed problematic. Imagine in different trials when data go under rounds of sampling (e.g., in cross validation), populations of subclasses may vary in each iteration. In such a case, the trained models are no longer comparable using a sensitive metric (like accuracy or ROC). To remedy this, either the number of each subclass should be kept fixed, or an insensitive metric must be used. True Skill Statistic (also known as Youden J Index) is a metric that is indeed insensitive to this issue. These metrics are very popular in the domains which deal with extreme-imbalanced data, such as weather forecasting, fraud detection, and of course in bioinformatics.

Also, people modified ROC and introduced Precision-Recall curve for this very reason. PR curve seems to be less sensitive to this issue.

---

For Youden J Index, see Youden 1950, for True Skill Statistic see Bloomfield et al. 2018.

For a thorough example, read this blog post on Machine Learning Master.

For an applied analysis on extreme-imbalance data, see Ahmadzadeh et al. 2019.

Answer 3

I think it is not safe to say that the AUC is insensitive to class imbalance, as it introduces some confusion to the reader. In case you mean that the score itself doesn't detect class imbalance, that's wrong, that's why the AUC is there. In case you mean insensitive such that changes in the class distribution don't have influence on calculating the AUC, that's true.

I happened to be prompted about this by my supervisor. In fact, that's literally the advantage of using the AUC as classification measure in comparison to others (e.g. accuracy). AUC tells you your model's performance pretty much, while addressing the issue of class imbalance. To be scientifically safe, I'd rather say it is insensitive to changes in class distribution.

For example, and to make this as simple as possible, let's take a look at a binary classification problem where the positive class is dominant.
Say, we have a sample distribution and a randomly-predicting model with default accuracy 0.8 (predicts positive constantly without even looking at the data). You can see that this model will return a high accuracy score, although its precision is rather low $$Precision = \frac{TP}{TP+FP}$$because the number of false positives will grow and therefore the denominator is larger ...

What the AUC on the other hand does, is that it notifies you that you have several wrongly classified positives $FP$ despite the fact that you have a high accuracy because of the dominant class, and therefore it would return a low score in this case.
I hope I made this clear!

If you are interested in AUC changes with different class distributions or AUC analysis for other classification tasks, I would definitely recommend you Fawcett's paper on ROC curve analysis. One of the best out there and easily put.

Answer 4

I choose to disgaree with the answer given by @Azim. Emphirical research has shown ROC is insentive to class imbalance. This has been extensively discussed by Tom Fawcett, see Section 4.2 of his paper An introduction to ROC analysis

4.2. Class skew ROC curves have an attractive property: they are insensitive to changes in class distribution. If the proportion of positive to negative instances changes in a test set, the ROC curves will not change. To see why this is so, consider the confusion matrix in Fig. 1. Note that the class distribution—the proportion of positive to negative instances—is the relationship of the left (+) column to the right (-) column. Any performance metric that uses values from both columns will be inherently sensitive to class skews. Metrics such as accuracy, precision, lift and F score use values from both columns of the confusion matrix. As a class distribution changes these measures will change as well, even if the fundamental classifier performance does not. ROC graphs are based upon tp rate and fp rate, in which each dimension is a strict columnar ratio, so do not depend on class distributions