I'm working on a 2-class classification problem with very unbalanced class size (95% vs. 5%). The overall data size is 500k+ and I did a 70%-30% train test split. So far I have tried the following models (all
- Logistic regression: train AUC ~0.5, test AUC ~0.5
- Gradient boosting: train AUC ~0.74, test AUC ~0.69
- Random Forest: train AUC 0.9999999, test AUC ~0.80
I'm seeing a perfect AUC for random forest but only ~0.8 on the testing set. Numbers in #1 and #2 looks much normal to me but I'm really scared of the "perfect" AUC on random forest training set.
Is this something that I should expect or within normal range? Why is this happening to random forest but not to some other classifiers? Are there any reasonable explanation or guess to this?
Update: I have done 10-fold cv and parameter grid search on the random forest model and here's some result:
- Random Forest (original): train AUC 0.9999999, test AUC ~0.80
- Random Forest (10-fold cv): average test AUC ~0.80
- Random Forest (grid search max depth 12): train AUC ~0.73 test AUC ~0.70
I can see that with the optimal parameter settings from grid search, the train and test AUCs are not that different anymore and look normal to me. However, this test AUC of 0.71 is much worse than the test AUC of original random forest (~0.80).
If it's an overfitting problem, after regularization, the test AUC should increase, but it's now the opposite to me, and I'm very confused.
Are there anything I'm missing here? Why is this happening? If I were to choose between the two models, I would choose the one with higher test AUC, which is the "probably" overfitted random forest, does it make sense?