# AUC and class imbalance in training/test dataset

I just start to learn the Area under the ROC curve (AUC). I am told that AUC is not reflected by data imbalance. I think it means that AUC is insensitive to imbalance in test data, rather than imbalance in training data.

In other words, only changing the distribution of positive and negative classes in the test data, the AUC value may not change much. But if we change the distribution in the training data, the AUC value may largely change. The reason is that the classifier cannot be learned well. In this case, we have to use undersampling and oversampling. Am I right? I just want to make sure my understanding on AUC is correct.

It depends how you mean the word sensitive. The ROC AUC is sensitive to class imbalance in the sense that when there is a minority class, you typically define this as the positive class and it will have a strong impact on the AUC value. This is very much desirable behaviour. Accuracy is for example not sensitive in that way. It can be very high even if the minority class is not well predicted at all.

In most experimental setups (bootstrap or cross validation for example) the class distribution of training and test sets should be similar. But this is a result of how you sample those sets, not of using or not using ROC. Basically you are right to say that the ROC makes abstraction of class imbalance in the test set by giving equal importance to sensitivity and specificity. When the training set doesn't contain enough examples to learn the class, this will still affect ROC though, as it should.

What you do in terms of oversampling and parameter tuning is a separate issue. The ROC can only ever tell you how well a specific configuration works. You can then try multiple config's and select the best.

• In binary classification, do we usually distinguish "AUC for positive class" and "AUC for negative class"? OR there is only one AUC? I think it depends on whether "default" AUC calculated by TPR and NPR can reflect the ability of identifying both positive and negative classes OR only positive class. – Munichong Feb 6 '17 at 14:01
• That auc would look at the positive class. Nothing prevents you from computing and with respect to both classes and averaging it. But most of the time, your application scenario makes it clear which one is the positive class. – David Ernst Feb 6 '17 at 14:12
• Thanks. I tried some experiments. But I get confused on calculating AUC for class 0: y_true=[1,0], y_pred=[0.9, 0.8], I use the sklearn.metrics.auc function to compute AUC. The result is 1.0. I assume it only reflects how the classifier identifies class1. I then switch 1 and 0 in y_true: y_true=[0,1], y_pred=[0.9, 0.8], The result is 0.0. This is how the classifier identifies class0. Am I right? – Munichong Feb 6 '17 at 15:57
• Trying different inputs, I also find that the sum of the two results is always 1. Am I right? – Munichong Feb 6 '17 at 16:02
• Roc is based on multiple confusion matrices based on different cutoffs. Read more theory till you understand what this means. I'm not familiar with scikit learn and its syntax. No the sum of two aucs with regard to both classes doesn't need to be 1. – David Ernst Feb 6 '17 at 16:03

I think it is not safe to say that the AUC is insensitive to class imbalance, as it introduces some confusion to the reader. In case you mean that the score itself doesn't detect class imbalance, that's wrong, that's why the AUC is there. In case you mean insensitive such that changes in the class distribution don't have influence on calculating the AUC, that's true.

I happened to be prompted about this by my supervisor. In fact, that's literally the advantage of using the AUC as classification measure in comparison to others (e.g. accuracy). AUC tells you your model's performance pretty much, while addressing the issue of class imbalance. To be scientifically safe, I'd rather say it is insensitive to changes in class distribution.

For example, and to make this as simple as possible, let's take a look at a binary classification problem where the positive class is dominant.
Say, we have a sample distribution and a randomly-predicting model with default accuracy 0.8 (predicts positive constantly without even looking at the data). You can see that this model will return a high accuracy score, although its precision is rather low $$Precision = \frac{TP}{TP+FP}$$because the number of false positives will grow and therefore the denominator is larger ...

What the AUC on the other hand does, is that it notifies you that you have several wrongly classified positives $$FP$$ despite the fact that you have a high accuracy because of the dominant class, and therefore it would return a low score in this case.
I hope I made this clear!

If you are interested in AUC changes with different class distributions or AUC analysis for other classification tasks, I would definitely recommend you Fawcett's paper on ROC curve analysis. One of the best out there and easily put.

• You state that it's not safe to say that AUC is insensitive to class imbalance but don't elaborate on this in your answer. Why do think this is the case? – Scholar Apr 2 '19 at 10:31
• I'm not sure that I understand your answer completely. Let's make an example. We have a dataset with 1000 samples, 10 positive and 990 negative cases, 1 TP case, 950 TN cases, 1 FN case and 48 FP cases. It returns 2% as precison but more than 90% AUC. Is it an accurate model in your opinion? Let's say this model is going to predict risk of heart attack! In my opinion, this is a useless model because it just identify those obvious negative cases. Please correct me if I'm wrong. – MTT Aug 20 '19 at 22:48
• @MTT in such a case you will define your minority class as your positive class – christopher May 9 '20 at 12:53

(a 3-years late answer, but maybe still useful!)

ROC is sensitive to the class-imbalance issue, meaning that it favors the class with larger population solely because of its higher population. In other words, it is biased toward the larger population when it comes to classification/prediction.

This is indeed problematic. Imagine in different trials when data go under rounds of sampling (e.g., in cross validation), populations of subclasses may vary in each iteration. In such a case, the trained models are no longer comparable using a sensitive metric (like accuracy or ROC). To remedy this, either the number of each subclass should be kept fixed, or an insensitive metric must be used. True Skill Statistic (also known as Youden J Index) is a metric that is indeed insensitive to this issue. These metrics are very popular in the domains which deal with extreme-imbalanced data, such as weather forecasting, fraud detection, and of course in bioinformatics.

Also, people modified ROC and introduced Precision-Recall curve for this very reason. PR curve seems to be less sensitive to this issue.

For Youden J Index, see Youden 1950, for True Skill Statistic see Bloomfield et al. 2018.

For a thorough example, read this blog post on Machine Learning Master.

For an applied analysis on extreme-imbalance data, see Ahmadzadeh et al. 2019.