I am using SVM (sklearn.svm) to classify fMRI data from two groups of people. One group has n = 25 and the other n = 26. Everyone except for one person has seen 96 trials, so in total I have 2,496 image files in one category and 2,398 in the other. Since the data is slightly unbalanced, I thought of two options:

  1. Using a leave-one-triplet-out crossvalidation strategy, where the test data always includes data from two people from the larger group and one from the smaller one (going through all possible leave-outs) therefore using unbalanced test data.
  2. Using a leave-one-out crossvalidation strategy with all possibilities meaning sometimes using unbalanced training data.

Does anyone have any experience with cases like this? Or general recommendations which option is better or why I should refrain from one (or both!) of these options?

I would be very thankful for any comments or recommendations!

  • $\begingroup$ Why do you expect this level of unbalance (almost none) will lead to problems? Have you considered using bootstrapping to validate your modeling approach instead of the (often unstable) leave-one-out CV methods? $\endgroup$
    – EdM
    Aug 25, 2020 at 11:50
  • $\begingroup$ To be honest, I haven't found any good information telling me when is an imbalance large enough to cause problems and just assumed that this is already too much of a difference. I haven't thought about using bootstrapping but that's a good pointer, thanks! Do you have any recommendations on how many iterations one would do in such a case? $\endgroup$
    – Max
    Aug 25, 2020 at 16:08

1 Answer 1


Imbalance is often much less of a problem for classification than is feared. See this thread for some discussion. The important thing is to have enough cases in the minority class to be able to model its difference from the majority class adequately. Your limit is thus that you only have 25 cases in the minority class, so that you might be at risk of overfitting with such a large amount of data per case. As ISLR points out in Section 9.5, your SVM is closely related to logistic ridge regression, a method that penalizes individual predictors to avoid overfitting.

A key to avoiding overfitting is choosing the tuning parameter for the SVM, which is usually done by cross-validation. There's are problems with the types of leave-one-out CV approaches you have been considering. In particular, your number of training sets can't ever exceed the number of observations, and the models can't change very much from fold to fold. See this page for example. In practice, this can lead to high variance in estimates so that 5- or 10-fold CV is often preferred. And with those 5- or 10-fold CV approaches you aren't limited to 5 or 10 folds as you can, if desired, just repeat the CV on multiple different randomizations of the data to make sure that you have a stable and reliable choice of the tuning parameter.

With a small number of cases like this, you can't really estimate the quality of your model but you can estimate the quality of your modeling process. You build the model on the entire data set with a defined modeling process for deciding on things like the choice of kernel and the tuning parameter. Then repeat all steps of the modeling process on multiple bootstrapped samples of the data while testing the results on the full data set. By the bootstrap principle, that testing of the results from the bootstrapped samples on the full data set is analogous to testing the modeling process of your data set against the underlying population. The number of bootstraps will depend on the precision with which you want to estimate the quality of the modeling process and the variability of the process. See this page for more discussion.

  • $\begingroup$ Thanks for the helpful tips and links! It will take me a while to properly understand it all, but it seems like 5-fold CV (maybe with repetitions) and bootstrapping are definitely two things to try out on my data! $\endgroup$
    – Max
    Aug 29, 2020 at 7:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.