I have a series of classification tasks at hand and as a final step, I need to build a classifier for a particularly small dataset. This dataset has 9 features and the class is binary while having only 65 instances.
This is the procedure I followed:
- Splitting the 64 instances into a train and test set using a 90% / 10% ratio. Therefore, the train set has 57 instances and the remaining 7 instances will construct the test set.
- Performing model selection using a 5-fold cross-validation on the train set.
- Training the final model on the train set.
- Testing the trained model on the test set.
The test results are satisfactory at the first glance, but my only concern is that the test set contains very few instances, namely 7! Is that theoretically or practically a problem? Is this a particular situation in which one might consider performing a rather non-standard procedure? Would using an 80% / 20% ratio solve the potential problem?
Thanks in advance.