I have good an imbalanced very small dataset (58 instances) and whould like to create a multiclass classification model. I'd like to use cross-validation in order to make the most out of the data I have, and choose the model that would best fit my data.

By reading online I think I have understood how cross validation works, but I ma not sure about one thing.

Most pages, also on this site, claim that you should divide the data in train and test and than apply cross validation only on the train data.

Since I am using cross validation only for comparing models, why am I not allowed to train and test the performance with cross validation on the complete dataset, and then develop a model which is trained and tested on different data?

This is the steps I think it would make sense to follow:

  1. Divide in test and train dataset by using stratified sampling
  2. Evaluate many models by using cross validation with the whole dataset (each existing instance is used for both training and testing at separate times)
  3. Choose the algorithm that worked best (presumibly by using F1 score)
  4. Train the algorithm on only train data
  5. Test the algorithm on only test data
  6. Provide the results on the test data, but with more guaratees than without cross-validation (the test data is small, so good performance may be dictated by luck)

Where is my approach wrong?

  • $\begingroup$ Splitting a small dataset into even smaller training & test subsets is not a good way to validate a model, eg. here, here. And you should be skeptical about how much you can learn from the data that you have (58 instances): how many classes are there and how many instances do you have per class? $\endgroup$
    – dipetkov
    Commented Jun 22, 2022 at 23:30
  • $\begingroup$ Also it seems you are rephrasing and reposting the same question, selectively including some information about your data in one post but not others. Case in point: Dealing with very small and unbalanced data It's better to include all relevant information in one stand-alone question. $\endgroup$
    – dipetkov
    Commented Jun 22, 2022 at 23:42

1 Answer 1


Part of your approach is right: you should divide your dataset into train and test by using stratified sampling. However, the test set should remain unseen during the modelling process to make sure that your model is not overfitting and will work with a new, unseen dataset.

In other words: you can use CV to train your data, using the train set, dividing it into different folds to train+test at the same time. With CV you can chose the best parameter sets and try different models. Once that you have chosen the best parameter set and/or model, then you can finally test the model in that unseen test set, to make sure that your model will generalize well to new datasets.

With so few instances, another approach is to just use all of them and parametrize the model with CV (because CV is in any case splitting the data into different folds that are used as train and test sets).



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.