0
$\begingroup$

The task

I am working on a binary classification problem using a SVM classifier. The feature matrix X (650x20) holds brain activity features for 12 subjects. The target array y (650x1) carries the diagnosis of a particular disorder (0 for controls, 1 for patients).

My approach

To evaluate the performance of my classifier, I am using scikit-learn's GridSearchCV following a LeaveOneGroupOut cross-validation (CV) scheme. I think this is the most reasonable way to proceed, because for each CV iteration, the information of a particular subject is retained in the test set (this way no positive bias towards this subject occurs).

The issue

For each iteration in the CV process (12 iterations corresponding to the 12 subjects) I have plotted the training accuracy and the test accuracy. Surprisingly, for every fold, the accuracy on the training set (11 subjects) is close to 0.35, while for the test set (1 subject), the accuracy is always 1 but for one of the subjects (0 in this case).

Here is a scheme of the CV approach that I applied: enter image description here

And here is the accuracy score for each CV iteration: enter image description here

My thoughts

  • My first thought was that, since for each fold the amount of training data is roughly 10 times bigger than the test data, classification on the training set would be more challenging compared to the test set (which only holds data from a particular subject). However, 0.35 seems a very poor score compared to the results yielded on the test set (0.92 on average).
  • Then, I also considered that the classifier could be just too bad. However, if this was the case, I would expect a very poor accuracy also on the test set.

I would appreciate if someone could shed some light on this.

$\endgroup$
4
  • $\begingroup$ 650 is not divisible by 12, what causes a differing number of rows per subject? Is it possible rows belonging to some subjects have gotten mislabeled as belonging to others due to a bad join or something like that? $\endgroup$ Jul 19, 2021 at 10:46
  • $\begingroup$ Thanks for answering. For each subject there are a different number of EEG segments (always between 45 and 55). Since each row in X refers to a particular EEG segment, the total number of rows is not divisible by the number of subjects @JonnyLomond. With regard to labeling, I have checked the labels of each participant and everything is correct in that sense. $\endgroup$
    – Eduardo
    Jul 19, 2021 at 10:49
  • $\begingroup$ Are there any features which are constant across all rows related to a subject? Patient age or something like that. The fact that in the test set the accuracy is always 0 or 1 means that the model is always making the same prediction for each row belonging to a subject, even though the EEG segments are different. Maybe it is learning only on some subject level covariate, identifying this could show the problem. If there are such covariates, try using only them without EEG information and see if the same behavior occurs $\endgroup$ Jul 19, 2021 at 11:01
  • $\begingroup$ I am afraid there aren't. All the features are EEG-based and thus are computed for every EEG segment. I could try checking if any of those features is somehow constant. $\endgroup$
    – Eduardo
    Jul 19, 2021 at 11:17

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.