3
$\begingroup$

I have a set of 100 examples evaluated using 10-fold cross validation, providing 94% classification accuracy on the test folds. However, when I test the model on a different test set, it provides 0% accuracy. Apparently, the model is overfitting, but why 10-fold cross validation does not indicate overfitting of the model?

$\endgroup$

1 Answer 1

2
$\begingroup$

Obtaining an accuracy of 0% would make me suspicious of anything else being wrong: the chances of getting 0% straight are likely quite low, and a test performance somewhere between your training set performance and random performance would be more reasonable. In such cases I would double check that the setup is correct.

My first suggestion would be to use repeated cross validation instead of cross validation (with a sufficient amount of repeats), and see if your performance estimates change. Look at the spread of performance across the repeats: this gives you a hint how likely you obtain a model from given training data that performs bad on new data. If you see such cases in your repeated CV performance, the same could be true with your test set. A second suggestion would be to check if your test data has known differences to the training data (e.g. different diversity captured/represented in the data: such could happen with different recording conditions between training and test set). It could be that the feature-target-relation is different in your training and test set.

BTW: the model fit and measured performance are influenced by a number of factors (beside those already mentioned), like: the amount of features, the amount of samples in the test set, the amount of classes do distinguish, and if those are balanced in the training and test set, and of course, the models types. Depending on those factors, obtaining a good CV performance but bad test performance could have many reasons. For example, if you evaluate multiple models on your training set and select the best performing one, overfitting can happen during model selection too. This could happen rather easily in your case due to the small test partition sizes of 10 samples each. The more possibilities you evaluate in such cases, the bigger the chances for one of them getting good results in CV by chance, even if they overfitted. But more detailed thoughts about this would probably require more information about your problem and current setup.

$\endgroup$
1
  • $\begingroup$ Another possible reason for good CV but bad test performance is structure in the data, e.g. multiple measurements of the same case. Random splits for CV then have some of the repetitions in training, some in the cross-validation test set, so that the CV performance does not guard against overfitting. But if the test set consists of independent cases, such overfitting will be indicated by bad test set performance. $\endgroup$ May 20, 2016 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.