Random Forest has almost perfect training AUC compared to other models

Question

I'm working on a 2-class classification problem with very unbalanced class size (95% vs. 5%). The overall data size is 500k+ and I did a 70%-30% train test split. So far I have tried the following models (all sklearn):

1. Logistic regression: train AUC ~0.5, test AUC ~0.5
2. Gradient boosting: train AUC ~0.74, test AUC ~0.69
3. Random Forest: train AUC 0.9999999, test AUC ~0.80

I'm seeing a perfect AUC for random forest but only ~0.8 on the testing set. Numbers in #1 and #2 looks much normal to me but I'm really scared of the "perfect" AUC on random forest training set.

Is this something that I should expect or within normal range? Why is this happening to random forest but not to some other classifiers? Are there any reasonable explanation or guess to this?

---

Update: I have done 10-fold cv and parameter grid search on the random forest model and here's some result:

1. Random Forest (original): train AUC 0.9999999, test AUC ~0.80
2. Random Forest (10-fold cv): average test AUC ~0.80
3. Random Forest (grid search max depth 12): train AUC ~0.73 test AUC ~0.70

I can see that with the optimal parameter settings from grid search, the train and test AUCs are not that different anymore and look normal to me. However, this test AUC of 0.71 is much worse than the test AUC of original random forest (~0.80).

If it's an overfitting problem, after regularization, the test AUC should increase, but it's now the opposite to me, and I'm very confused.

Are there anything I'm missing here? Why is this happening? If I were to choose between the two models, I would choose the one with higher test AUC, which is the "probably" overfitted random forest, does it make sense?

Answer 1

Hello guys! Sorry for my english. I undestand that so your problem is a question about model choise. You have to want the best test error, no the simility between train and test error. The minumum in the image shows your goal, to be sure about your decision, you can graph the errors versus complexity (depth pode, trees or min samples leaf). Indeed, the best parameters in the model securely will be for a lower train error than 0.999.

Answer 2

Random forests have many many degrees of freedom, so it is relatively easy for them to get to the point that they have near 100% accuracy in-sample. This is merely an overfitting problem. Likely you want to use some tuning parameters to reduce the model complexity some (reduce tree depth, raise minimal node size, etc). Some degree of cross-validation would help you here. Alternatively, it still has the best oob performance, so you can just use it anyhow.

Answer 3

Because the ML algorithms works minimizing the error on the training, the expected accuracy on this data would be "naturally" better than your test results. Effectively when the training error is too low (aka accuracy too high) maybe there is something that has gone wrong (aka overfitting)

As suggested by user5957401, you can try to cross-validate the training process. For example, if you have a good amount of instances, a 10 fold cross-validation would be fine. If you need also to tune hyper parameters, a nested-cross validation would be necessary.

In this way the estimated error from the test-set will be "near" the expected one (aka, the one that you'll get on real Data). In this way, you can check if your result (AUC 0.80 on the test set) is a good estimate, or if you got this by chance

You can try also other techniques, like shuffling several times your data before the cross-validation task, to increase the result reliability.