I'm working on a 2-class classification problem with very unbalanced class size (95% vs. 5%). The overall data size is 500k+ and I did a 70%-30% train test split. So far I have tried the following models (all sklearn):

  1. Logistic regression: train AUC ~0.5, test AUC ~0.5
  2. Gradient boosting: train AUC ~0.74, test AUC ~0.69
  3. Random Forest: train AUC 0.9999999, test AUC ~0.80

I'm seeing a perfect AUC for random forest but only ~0.8 on the testing set. Numbers in #1 and #2 looks much normal to me but I'm really scared of the "perfect" AUC on random forest training set.

Is this something that I should expect or within normal range? Why is this happening to random forest but not to some other classifiers? Are there any reasonable explanation or guess to this?

Update: I have done 10-fold cv and parameter grid search on the random forest model and here's some result:

  1. Random Forest (original): train AUC 0.9999999, test AUC ~0.80
  2. Random Forest (10-fold cv): average test AUC ~0.80
  3. Random Forest (grid search max depth 12): train AUC ~0.73 test AUC ~0.70

I can see that with the optimal parameter settings from grid search, the train and test AUCs are not that different anymore and look normal to me. However, this test AUC of 0.71 is much worse than the test AUC of original random forest (~0.80).

If it's an overfitting problem, after regularization, the test AUC should increase, but it's now the opposite to me, and I'm very confused.

Are there anything I'm missing here? Why is this happening? If I were to choose between the two models, I would choose the one with higher test AUC, which is the "probably" overfitted random forest, does it make sense?

  • 1
    $\begingroup$ This AUC seems indeed rather huge showing that the RF has overfitted training set. That being said, training set metrics are little more than bellwethers so nobody really judges their performance. From your description it is unclear if you are using repeated CV or not. Usually with such a large data-set repeated CV would be necessary but as you seem to indeed over-fit at time I would try repeated resampling. Finally AUC is just one metric. Try looking into additional metrics like the PR-curves or the calibration plots. $\endgroup$ – usεr11852 says Reinstate Monic Mar 19 '18 at 11:23
  • $\begingroup$ @usεr11852 I have updated the post with more details, hope you can provide more insights. $\endgroup$ – Yilun Zhang Mar 19 '18 at 18:53
  • $\begingroup$ The fact you have only "one" test-sample that lead to an AUC of 0.80 but now you have an AUC of 0.71 when using a 10-fold CV is not horribly surprising. It just goes to show a common pit that might happen, people overfit their test-set. :) (Yes, this can happen too) You are not missing anything, just having a hold-out set is not a silver bullet when it comes to error estimation. I am a bit surprised that the performance deterioration was more than 10% but a ~5% difference would be almost common-place when going from a "single set" to a "CV estimate." $\endgroup$ – usεr11852 says Reinstate Monic Mar 19 '18 at 20:00
  • $\begingroup$ (BTW in my first post I meant to say "with such a large data-set repeated CV would be unnecessary" instead of "would be necessary", apologies for that!) $\endgroup$ – usεr11852 says Reinstate Monic Mar 19 '18 at 20:01
  • $\begingroup$ @usεr11852 You misunderstood it. The 10-fold cv is returning average 0.80 AUC, which is consistent with the original random forest without any regularization. The grid search (with smaller max_depth) of random forest with 70%-30% train test split is giving AUC 0.71. $\endgroup$ – Yilun Zhang Mar 19 '18 at 20:02

Because the ML algorithms works minimizing the error on the training, the expected accuracy on this data would be "naturally" better than your test results. Effectively when the training error is too low (aka accuracy too high) maybe there is something that has gone wrong (aka overfitting)

As suggested by user5957401, you can try to cross-validate the training process. For example, if you have a good amount of instances, a 10 fold cross-validation would be fine. If you need also to tune hyper parameters, a nested-cross validation would be necessary.

In this way the estimated error from the test-set will be "near" the expected one (aka, the one that you'll get on real Data). In this way, you can check if your result (AUC 0.80 on the test set) is a good estimate, or if you got this by chance

You can try also other techniques, like shuffling several times your data before the cross-validation task, to increase the result reliability.

  • $\begingroup$ I did cross-validate and also grid searched for best parameters on the test set, but the simpler random forest model (with smaller max_depth) is performing much worse on the test set (test AUC only around 0.71). And I think it's not an overfitting problem anymore because for overfitting problem, it can usually be solved by controlling the complexity of the model and the resulting model should perform better than the overfitting model on the test set, but it's not the case for me. $\endgroup$ – Yilun Zhang Mar 19 '18 at 18:40

Random forests have many many degrees of freedom, so it is relatively easy for them to get to the point that they have near 100% accuracy in-sample. This is merely an overfitting problem. Likely you want to use some tuning parameters to reduce the model complexity some (reduce tree depth, raise minimal node size, etc). Some degree of cross-validation would help you here. Alternatively, it still has the best oob performance, so you can just use it anyhow.

  • $\begingroup$ So I did use grid search to find optimal parameters for random forest. Now I have: (1) the one without parameter optimization: train AUC 0.9999, test AUC 0.80, and (2) after grid search: train AUC 0.73, test AUC 0.69. How do I choose between them? I know (1) if overfitting, but it does also have a better test AUC.. $\endgroup$ – Yilun Zhang Mar 15 '18 at 14:56
  • $\begingroup$ Sorry for the delay in reply -- so long as your test sample is used after the grid search, use 1. if the grid search optimized against test sample performance, use 2. $\endgroup$ – user5957401 Sep 26 '18 at 22:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.