10-fold cross validation on small number of examples

Question

I have a set of 100 examples evaluated using 10-fold cross validation, providing 94% classification accuracy on the test folds. However, when I test the model on a different test set, it provides 0% accuracy. Apparently, the model is overfitting, but why 10-fold cross validation does not indicate overfitting of the model?

Answer 1

Obtaining an accuracy of 0% would make me suspicious of anything else being wrong: the chances of getting 0% straight are likely quite low, and a test performance somewhere between your training set performance and random performance would be more reasonable. In such cases I would double check that the setup is correct.

My first suggestion would be to use repeated cross validation instead of cross validation (with a sufficient amount of repeats), and see if your performance estimates change. Look at the spread of performance across the repeats: this gives you a hint how likely you obtain a model from given training data that performs bad on new data. If you see such cases in your repeated CV performance, the same could be true with your test set. A second suggestion would be to check if your test data has known differences to the training data (e.g. different diversity captured/represented in the data: such could happen with different recording conditions between training and test set). It could be that the feature-target-relation is different in your training and test set.

BTW: the model fit and measured performance are influenced by a number of factors (beside those already mentioned), like: the amount of features, the amount of samples in the test set, the amount of classes do distinguish, and if those are balanced in the training and test set, and of course, the models types. Depending on those factors, obtaining a good CV performance but bad test performance could have many reasons. For example, if you evaluate multiple models on your training set and select the best performing one, overfitting can happen during model selection too. This could happen rather easily in your case due to the small test partition sizes of 10 samples each. The more possibilities you evaluate in such cases, the bigger the chances for one of them getting good results in CV by chance, even if they overfitted. But more detailed thoughts about this would probably require more information about your problem and current setup.