Should we use AUC as an indicator of overfitting when dataset is highly imbalanced?

Question

In my problem, there are 2 class labels, but one label only counts for 1% of the total data. I first divided my data set by train_test_split such that only 10% are test set, then I performed 10-Fold cross validation and below is the AUC on the validation set for 10 folds:

0.8339720951796847
0.8339720951796847
0.8340767220106542
0.8331529270822466
0.8293208019913448
0.8474731942921908
0.8545871857725601
0.8235138776279672

which seems to have very low variances between each fold. However on the test set: AUC=0.543546.

The situation is even worse if I use StratifiedShuffleSplit: while the average AUC for cross validation is still around 0.85, the AUC on the test set is 0.2.

My question is: Can we use AUC as an indicator for overfitting when dataset is highly imbalanced? Since the test set now is very small and the auc should not be expected to be as accuracy as when cross validation.

Answer 1

AUC seems to be OK to use with imbalanced classes, although the precision-recall curve is better, as explained for example here AUC and class imbalance in training/test dataset

However, you MUST use StratifiedShuffleSplit (or any other stratified sampling which ensure the splits preserve class distributions) with imbalanced classes, otherwise you risk getting really bad test sets (missing the minority class, and overestimating your model's performance). This is reflected in your results: without Stratified sampling, the test AUC is better, 0.54 vs 0.2 with StratifiedShuffleSplit. I am quite sure the true performance metric of that model is 0.2, not 0.54.

Answer 2

One way to check if you're overfitting using AUC is also to apply it directly on your train data, and compare it to your validation. If you have a huge difference (with a train AUC way better, meaning you know the train data "too much"), that means you're overfitting.