The task
I am working on a binary classification problem using a SVM classifier. The feature matrix X (650x20) holds brain activity features for 12 subjects. The target array y (650x1) carries the diagnosis of a particular disorder (0 for controls, 1 for patients).
My approach
To evaluate the performance of my classifier, I am using scikit-learn's GridSearchCV
following a LeaveOneGroupOut
cross-validation (CV) scheme. I think this is the most reasonable way to proceed, because for each CV iteration, the information of a particular subject is retained in the test set (this way no positive bias towards this subject occurs).
The issue
For each iteration in the CV process (12 iterations corresponding to the 12 subjects) I have plotted the training accuracy and the test accuracy. Surprisingly, for every fold, the accuracy on the training set (11 subjects) is close to 0.35, while for the test set (1 subject), the accuracy is always 1 but for one of the subjects (0 in this case).
Here is a scheme of the CV approach that I applied:
And here is the accuracy score for each CV iteration:
My thoughts
- My first thought was that, since for each fold the amount of training data is roughly 10 times bigger than the test data, classification on the training set would be more challenging compared to the test set (which only holds data from a particular subject). However, 0.35 seems a very poor score compared to the results yielded on the test set (0.92 on average).
- Then, I also considered that the classifier could be just too bad. However, if this was the case, I would expect a very poor accuracy also on the test set.
I would appreciate if someone could shed some light on this.