# Should we use AUC as an indicator of overfitting when dataset is highly imbalanced?

In my problem, there are 2 class labels, but one label only counts for 1% of the total data. I first divided my data set by train_test_split such that only 10% are test set, then I performed 10-Fold cross validation and below is the AUC on the validation set for 10 folds:

0.8339720951796847
0.8339720951796847
0.8340767220106542
0.8331529270822466
0.8293208019913448
0.8474731942921908
0.8545871857725601
0.8235138776279672


which seems to have very low variances between each fold. However on the test set: AUC=0.543546.

The situation is even worse if I use StratifiedShuffleSplit: while the average AUC for cross validation is still around 0.85, the AUC on the test set is 0.2.

My question is: Can we use AUC as an indicator for overfitting when dataset is highly imbalanced? Since the test set now is very small and the auc should not be expected to be as accuracy as when cross validation.