SVM for fMRI: unbalanced data - leave-one-triplet-out?

Question

I am using SVM (sklearn.svm) to classify fMRI data from two groups of people. One group has n = 25 and the other n = 26. Everyone except for one person has seen 96 trials, so in total I have 2,496 image files in one category and 2,398 in the other. Since the data is slightly unbalanced, I thought of two options:

1. Using a leave-one-triplet-out crossvalidation strategy, where the test data always includes data from two people from the larger group and one from the smaller one (going through all possible leave-outs) therefore using unbalanced test data.
2. Using a leave-one-out crossvalidation strategy with all possibilities meaning sometimes using unbalanced training data.

Does anyone have any experience with cases like this? Or general recommendations which option is better or why I should refrain from one (or both!) of these options?

I would be very thankful for any comments or recommendations!

Answer 1

Imbalance is often much less of a problem for classification than is feared. See this thread for some discussion. The important thing is to have enough cases in the minority class to be able to model its difference from the majority class adequately. Your limit is thus that you only have 25 cases in the minority class, so that you might be at risk of overfitting with such a large amount of data per case. As ISLR points out in Section 9.5, your SVM is closely related to logistic ridge regression, a method that penalizes individual predictors to avoid overfitting.

A key to avoiding overfitting is choosing the tuning parameter for the SVM, which is usually done by cross-validation. There's are problems with the types of leave-one-out CV approaches you have been considering. In particular, your number of training sets can't ever exceed the number of observations, and the models can't change very much from fold to fold. See this page for example. In practice, this can lead to high variance in estimates so that 5- or 10-fold CV is often preferred. And with those 5- or 10-fold CV approaches you aren't limited to 5 or 10 folds as you can, if desired, just repeat the CV on multiple different randomizations of the data to make sure that you have a stable and reliable choice of the tuning parameter.

With a small number of cases like this, you can't really estimate the quality of your model but you can estimate the quality of your modeling process. You build the model on the entire data set with a defined modeling process for deciding on things like the choice of kernel and the tuning parameter. Then repeat all steps of the modeling process on multiple bootstrapped samples of the data while testing the results on the full data set. By the bootstrap principle, that testing of the results from the bootstrapped samples on the full data set is analogous to testing the modeling process of your data set against the underlying population. The number of bootstraps will depend on the precision with which you want to estimate the quality of the modeling process and the variability of the process. See this page for more discussion.