We have 3000 samples for two classes, roughly 2000:1000.
Our plan is to train a classifier on the samples but first to set aside 30% randomly selected stratified samples as a "holdout data set" for a final test.
Then we want to experiment a lot with the remaining 2100 samples, i.e. feature engineering and tweaking of classifier settings. Here we plan to use cross validation at each "tweak" of the classifier (measuring accuracies etc.). Then when we think we are done, run a final test on the 900 "holdout" samples with the final classifier.
Is this a valid use of the "holdout data set" and cross validation methods?
I have seen cross validation used before, but not with a separate testing data set as described.