I am working on relating aesthetic scores of given images (about 17k training+validation samples and 280 image features) and getting best result using ensemble of CARTs. Beside achieveing a good prediction performance, I also have calculation time constraints and would like to select features using some background on the field. Some prediction performance loss is expected after the selection.
I am using ensemble feaure importances averaging MSE reduction at each tree and each split (MATLAB implementation). Doing so, I have some problems determining the exact method to use. Altough I have a seperate test set, I also need to report validation performance for model/feature set comparison purposes. Options I'm currently considering are;
Using the whole training set to train the model 20 times and average the resulting feature importances (repeated due to the bootstrap nature).
Separating 20% randomly selected validatation sets (without using the test set), train the model 20 times and average feature importances from each model.
Using whole samples (option 1) does not make complete sense since it will be using all the data and there is no validation set, rendering the feature selection somewhat biased.
On the other hand, with option 2 feature importances are calculated for different parts of the data (discarding the validation samples) better simulating the process but still there is a probability of using all samples when calculating and averaging importance, introducing bias.
Since it is not possible to "wrap" the heuristic and manual feature selection process into the validation method, considering the variance reduction and bootstrap nature of bagging method I "feel" that the first option will be equivalent to the second one.
Is the second option better in terms of introduced bias or the first one is OK?
When both options are used for feature selection can I report validation results of the reduced model or do I have to report only test set performance and validation set used during importance calculation is compromised?