I have 2000 observations in a dataset with features and a binary-class outcome. I split the dataset into two sets for split sample validation. I use 80% to train the model and internal perform Cross validation (CV). I then test this model on the 20% validation dataset. I found the cross-validated AUC in my internal validation set is 0.86 and the AUC of external validation set is 0.73. Now I have few questions:

  1. Does it reflect that my model is overfitting?

  2. What should be the standard/acceptable difference between two error rate to conclude that the model is optimal?

  3. Any other advice will be helpful.

  • $\begingroup$ How is the 20% selected? If randomly, does the same difference appear if you make other random splits? Is the internal CV used to optimize any tuning parameters? $\endgroup$ – NRH Jul 16 '16 at 23:53

It is completely understandable that the cross-validated AUC is not as good as the one obtained in your split sample validation set. That is because some of the observations included in the cross validation are the same as those used to train the model, even if it does not equal the training dataset entirely.

So we should always expect external validation to show a slightly less favorable predictive model performance. This does not necessarily suggest that overfitting is an issue. Overfitting is a highly subjective matter, like "outliers", prone to misinterpretation.

The usual culprits of overfitting are: small sample size, too many features, nonconservative choices of tuning parameters in model selection procedures like LASSO, and evaluating too many modeling procedures in an exploratory fashion rather than prespecifying those of interest and controlling for multiple comparisons.

All of these, coincidentally, explain in some part also a reduced external validation metric compared to CV in sample. But they do not objectively suggest that the model is "bad", rather just suboptimal. As you alluded to, no there are no general results about the optimality of model selection. So long as the background of the scientific problem is properly applied and good modeling procedures are used, then conditions like these may be best simply described to those who use them as a "caveat emptor".


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.