Does test AUC of 0.98 mean overfitting if we have highly imbalanced dataset (0.5% minority class)?

Question

I built a Random Forest model to classify imbalanced data (0.5% for minority class.) I used grid search to hyper tune parameters. I got the test AUC of almost 0.99. The test AUC is completely out of bag. I am not sure how we can extrapolate that this signifies overfitting?

I got some answer from https://stats.stackexchange.com/questions/333199/random-forest-has-almost-perfect-training-auc-compared-to-other-models. But I was curious about the behavior when we have imbalanced class at our desposal.

Answer 1

If you have a drastically imbalanced class, it is very likely that you are overfitting. Let's say that your test set comprises of 99 cases of sample A and 1 case of sample B. Then, any model you train could get "high" score, simply by classifying any value it encounters as a sample A type.

As random forest are built from decision trees which use information theory to determine which feature threshold would best partition the data, they are susceptible to imbalanced sets. This is especially true since the proportion of any one class in your dataset will affect the importance the model associates to the accurate classification of samples of this class. Intuitively, you can think that the model wants to be as right as possible, so it will focus on accurately binning (classifying) the most items.

For imbalanced testing sets, I would suggest using the F1 score to evaluate your model which is calculated as 2 * (precision * recall)/(precision + recall) to corroborate any other metrics you use (like accuracy or AUC). Additionally, I would suggest training on a balanced dataset (whether this means over sampling the minority class or undersampling the majority classes).