I have constructed a novel ML (NLP) dataset for classification and labeled it with three classes. The dataset is rather small with about 700 examples, out of which the classes have about 400, 200, and 100 examples respectively. I would like to publish it and describe it in an official publication for a workshop or a conference.

When looking at related datasets and publication, I see that it is common for authors to publish the dataset already split into three chunks - train, dev, test dataset (see the images). It is also common in these papers to provide the performance of baseline models on the dataset. Considering the dataset's small size, I feel like doing a 5-fold cross-validation would be a good alternative for such a small dataset, rather than doing something like a split into 450-100-150 train-dev-test datasets and then evaluating only on the very small dataset with 150 examples. Still, I believe that for better replicability, doing an "official" split is preferred and then everyone in the future testing on the same test set with 150 examples? Why do the authors usually already provide the three splits?

Furthermore, when looking at these ML resource papers, I saw in a few instances that the test set is kept balanced with respect to the three classes, even though the original dataset was not and dev set is not made balanced. This is problematic in my case for my third class where there are only about 100 examples. If I make my test set to be 50-50-50 for class1-class2-class3, then there is only 50 examples of class3 left for train+dev! That is simply infeasible for the training set. None of the authors provide any sort of explanation why they split it like this, they just seem to say "here is our split". Is this done to discourage the model from just doing a majority-class prediction and thus make it challenging? Or because a dummy classifier would have a 60% accuracy? Still, with a metric like F1 and not accuracy, this does not seem like an issue...

FEVER dataset SciFact dataset CHEF dataset

When searching through Stack Overflow for similar questions, people were usually discouraged from splitting their Kaggle datasets into a test dataset that is balanced, with the argument that we want a classifier to work with data that resembles the real-world distribution and makes it ready for production.

To sum up:

  • Is is considered mandatory to provide the "official" train-dev-test split when introducing a new dataset in an ML publication?
  • If so, should the test set have a balanced class distribution and why?

1 Answer 1


It is not clear that splitting the data like this is a good idea. In particular, it has been noted that thousands of observations are usually required for such splitting to be stable. Since you only have $700$ observations, such a split is likely to present problems.


  1. From a statistical standpoint, no, you do not need to do an official train/validate/test split of your data.
  2. The venue hosting your data might require such a split, anyway, making it a requirement no matter what objections others might have. You are allowed for this requirement to be a dealbreaker, and they are allowed to tell you to host your data elsewhere if you do not want to play by their rules (no matter how bad you might think those rules are).
  • $\begingroup$ Imbalance is a separate issue, and you might want to head down the rabbit hole. Hopefully, after you do some reading, you will see why the natural class ratio is important for validation, whether you do that validation with an explicit holdout set or by some other method like bootstrap. $\endgroup$
    – Dave
    Apr 24, 2023 at 12:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.