I am testing combinations of preprocessing activities on a small data set (n=48, p=30). The script generates 3200 different versions of the original data and measures how they perform in a classification task against 5 classifiers. Classifier parameters are tuned with 10 times bootstrap.
I assume a good sampling scheme would be to repeat 10-fold cross-validation 10 times, but this is computationally too much, at least for my resources:
- 3200 times
- 10 * 10 (cross-validation)
- for each round tune and train the five models 10 * 5
The idea at this moment is to
- For each data version
- Tune the models' parameters with 10 times bootstrap sample (2/3 tuning set - 1/3 tuning test set)
- Train the models with new bootstrap sample given the parameters
- Test the learned models by taking 3/3 bootstrap repeated 100 times
This is a rather complicated sampling scheme and mixes training and test sets.
What would be a good yet computationally feasible sampling scheme in this situation?