Using bagged ensemble of regression trees, feature selection based on feature importance

Question

I am working on relating aesthetic scores of given images (about 17k training+validation samples and 280 image features) and getting best result using ensemble of CARTs. Beside achieveing a good prediction performance, I also have calculation time constraints and would like to select features using some background on the field. Some prediction performance loss is expected after the selection.

I am using ensemble feaure importances averaging MSE reduction at each tree and each split (MATLAB implementation). Doing so, I have some problems determining the exact method to use. Altough I have a seperate test set, I also need to report validation performance for model/feature set comparison purposes. Options I'm currently considering are;

1. Using the whole training set to train the model 20 times and average the resulting feature importances (repeated due to the bootstrap nature).

2. Separating 20% randomly selected validatation sets (without using the test set), train the model 20 times and average feature importances from each model.

Using whole samples (option 1) does not make complete sense since it will be using all the data and there is no validation set, rendering the feature selection somewhat biased.

On the other hand, with option 2 feature importances are calculated for different parts of the data (discarding the validation samples) better simulating the process but still there is a probability of using all samples when calculating and averaging importance, introducing bias.

Since it is not possible to "wrap" the heuristic and manual feature selection process into the validation method, considering the variance reduction and bootstrap nature of bagging method I "feel" that the first option will be equivalent to the second one.

Is the second option better in terms of introduced bias or the first one is OK?

When both options are used for feature selection can I report validation results of the reduced model or do I have to report only test set performance and validation set used during importance calculation is compromised?

Answer 1

In general, flexible approaches to statistical modeling that are the most accurate do not attempt to be parsimonious, and provide results that are not to be taken out of context. The "features selected" are really just examples, and you would be unlikely to find the same features in a new dataset. Use the overall predictions from the black box without trying to decode the black box. If you really need to decode it you can develop approximations using regression or single trees, but you will see in most cases that to provide an adequate approximation the second stage method needs to be complex (e.g., many nodes will be required for a single tree approximation). In short, black box methods do not result in a reduced model.