I have good an imbalanced very small dataset (58 instances) and whould like to create a multiclass classification model. I'd like to use cross-validation in order to make the most out of the data I have, and choose the model that would best fit my data.
By reading online I think I have understood how cross validation works, but I ma not sure about one thing.
Most pages, also on this site, claim that you should divide the data in train and test and than apply cross validation only on the train data.
Since I am using cross validation only for comparing models, why am I not allowed to train and test the performance with cross validation on the complete dataset, and then develop a model which is trained and tested on different data?
This is the steps I think it would make sense to follow:
- Divide in test and train dataset by using stratified sampling
- Evaluate many models by using cross validation with the whole dataset (each existing instance is used for both training and testing at separate times)
- Choose the algorithm that worked best (presumibly by using F1 score)
- Train the algorithm on only train data
- Test the algorithm on only test data
- Provide the results on the test data, but with more guaratees than without cross-validation (the test data is small, so good performance may be dictated by luck)
Where is my approach wrong?