0
$\begingroup$

I am working on relating aesthetic scores of given images (about 17k training+validation samples and 280 image features) and getting best result using ensemble of CARTs. Beside achieveing a good prediction performance, I also have calculation time constraints and would like to select features using some background on the field. Some prediction performance loss is expected after the selection.

I am using ensemble feaure importances averaging MSE reduction at each tree and each split (MATLAB implementation). Doing so, I have some problems determining the exact method to use. Altough I have a seperate test set, I also need to report validation performance for model/feature set comparison purposes. Options I'm currently considering are;

  1. Using the whole training set to train the model 20 times and average the resulting feature importances (repeated due to the bootstrap nature).

  2. Separating 20% randomly selected validatation sets (without using the test set), train the model 20 times and average feature importances from each model.

Using whole samples (option 1) does not make complete sense since it will be using all the data and there is no validation set, rendering the feature selection somewhat biased.

On the other hand, with option 2 feature importances are calculated for different parts of the data (discarding the validation samples) better simulating the process but still there is a probability of using all samples when calculating and averaging importance, introducing bias.

Since it is not possible to "wrap" the heuristic and manual feature selection process into the validation method, considering the variance reduction and bootstrap nature of bagging method I "feel" that the first option will be equivalent to the second one.

Is the second option better in terms of introduced bias or the first one is OK?

When both options are used for feature selection can I report validation results of the reduced model or do I have to report only test set performance and validation set used during importance calculation is compromised?

$\endgroup$
3
$\begingroup$

In general, flexible approaches to statistical modeling that are the most accurate do not attempt to be parsimonious, and provide results that are not to be taken out of context. The "features selected" are really just examples, and you would be unlikely to find the same features in a new dataset. Use the overall predictions from the black box without trying to decode the black box. If you really need to decode it you can develop approximations using regression or single trees, but you will see in most cases that to provide an adequate approximation the second stage method needs to be complex (e.g., many nodes will be required for a single tree approximation). In short, black box methods do not result in a reduced model.

$\endgroup$
  • $\begingroup$ So you are telling eliminating features based on feature importance is not valid? In my case feature elimination is essential and I have to use some reference, such as feature importance. Wrapper methods proved to be highly unstable. $\endgroup$ – Ali Naci Erdem Jul 10 '15 at 13:35
  • 1
    $\begingroup$ Essentially yes. You would be going against how the estimation algorithm was devised. Parsimony is often your enemy in terms of predictive discrimination. If you do absolutely require parsimony it is best to use penalized maximum likelihood estimation so that the weights of the "selected" features are properly discounted for how hard it was to find those features (e.g., try elastic net). $\endgroup$ – Frank Harrell Jul 10 '15 at 13:38
  • $\begingroup$ When predictor importance is used then? And can you suggest any credible reading? $\endgroup$ – Ali Naci Erdem Jul 10 '15 at 13:42
  • 1
    $\begingroup$ Elsewhere on this site you'll find discussions about predictor importance. When using maximum likelihood, the log likelihood explained is the gold standard. Predictor importance is informative but to be taken with a grain of salt. You can also bootstrap the ranks of predictors in terms of explained variation or log likelihood to get confidence intervals for the ranking of importance. The very wide widths you will see expose the difficulty of the task. The upcoming 2nd edition of my book Regression Modeling Strategies has an example. $\endgroup$ – Frank Harrell Jul 10 '15 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.