I have constructed a novel ML (NLP) dataset for classification and labeled it with three classes. The dataset is rather small with about 700 examples, out of which the classes have about 400, 200, and 100 examples respectively. I would like to publish it and describe it in an official publication for a workshop or a conference.
When looking at related datasets and publication, I see that it is common for authors to publish the dataset already split into three chunks - train, dev, test dataset (see the images). It is also common in these papers to provide the performance of baseline models on the dataset. Considering the dataset's small size, I feel like doing a 5-fold cross-validation would be a good alternative for such a small dataset, rather than doing something like a split into 450-100-150 train-dev-test datasets and then evaluating only on the very small dataset with 150 examples. Still, I believe that for better replicability, doing an "official" split is preferred and then everyone in the future testing on the same test set with 150 examples? Why do the authors usually already provide the three splits?
Furthermore, when looking at these ML resource papers, I saw in a few instances that the test set is kept balanced with respect to the three classes, even though the original dataset was not and dev set is not made balanced. This is problematic in my case for my third class where there are only about 100 examples. If I make my test set to be 50-50-50 for class1-class2-class3, then there is only 50 examples of class3 left for train+dev! That is simply infeasible for the training set. None of the authors provide any sort of explanation why they split it like this, they just seem to say "here is our split". Is this done to discourage the model from just doing a majority-class prediction and thus make it challenging? Or because a dummy classifier would have a 60% accuracy? Still, with a metric like F1 and not accuracy, this does not seem like an issue...
When searching through Stack Overflow for similar questions, people were usually discouraged from splitting their Kaggle datasets into a test dataset that is balanced, with the argument that we want a classifier to work with data that resembles the real-world distribution and makes it ready for production.
To sum up:
- Is is considered mandatory to provide the "official" train-dev-test split when introducing a new dataset in an ML publication?
- If so, should the test set have a balanced class distribution and why?
