I have a dataset with 12k samples where each sample is a feature vector of length 4096.
The task it to classify the samples into 12 categories.
I fitted a network with 2 fully connected layers on the entire dataset and the model was able to separate it almost perfectly (it has 60 errors on it) which means the data is almost perfectly separable.
After that I split the data into 10 folds.
For each fold I evaluated the model on it after fitting it to the 9 remaining folds.
The model is always able to fit perfectly to the 9 folds, while having very poor performance on the evaluation fold.
This means that the model cannot explain the evaluation fold by the rest of the folds, even though I've shown that all of the data can be explained by a single model of the same architecture,
What could be causing this issue? And how can I solve it?