I am trying to perform multiclass classification on a 10 class dataset with around 650 data points. But whenever trying to run the code, it gives the above-mentioned error. Although, I understand what does it mean but is there any way to get away with this?


Why is this error occurring?

I'm guessing you are using sklearn.model_selection.cross_val_score or sklearn.model_selection.GridSearchCV when the error occurs. Both functions internally use a StratifiedKFold cross-validator, which splits the data into $k$ stratified folds. This means that the folds are made in such a way that the percentage of samples in each class is preserved. This however requires you to have at least $k$ samples in each class (so that there at least be one sample assigned to each fold), a requirement that your dataset does not satisfy.

How to fix it?

In order to use it you need to pass an instance of a KFold cross-validator to the cv parameter of your functions.

For example:

from sklearn.model_selection import KFold, cross_val_score

estimator = ...  # an sklearn estimator
X = ...  # training data
y = ...  # training labels

kf = KFold(n_splits=3)
scores = cross_val_score(estimator, X, y, cv=kf)

This should solve your problem.

Final Notes

If you have a class with less than 3 samples, your models will have a really hard time learning this class! If possible, you should consider obtaining more data (especially from this class), merging some classes together (so that they have more samples), over-sampling the minority classes or using a a model that supports class weighting.

  • $\begingroup$ I am using CalibratedClassifierCV and I have also raised an issue in GitHub. Here is the link which also has the code github.com/scikit-learn/scikit-learn/issues/12030 Also, 'using a model that supports class weighting', can you explain it? Which models support class weighting? $\endgroup$ – Akash Dubey Sep 6 '18 at 20:48
  • $\begingroup$ It is actually not a bug. If you take a look at the code it is clearly intentional. In order to make proper stratified folds you need at least 1 sample per fold. $\endgroup$ – Djib2011 Sep 6 '18 at 20:53
  • 1
    $\begingroup$ Yes, CalibratedClassifierCV does have a cv parameter you can use to pass a KFold cross-validator. Just do it like I showed above. P.S there was a typo in the code I posted; it's fixed now. $\endgroup$ – Djib2011 Sep 6 '18 at 21:52
  • 1
    $\begingroup$ It will let you bypass the ValueError being raised. However, you should worry about your classes having few samples, as your models will have a hard time predicting them. $\endgroup$ – Djib2011 Sep 6 '18 at 22:41
  • 1
    $\begingroup$ Okay. I will rather try to merge some of the classes. That way might be helpful as there would be more data points per class. $\endgroup$ – Akash Dubey Sep 6 '18 at 22:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.