Everything is in the title, does it makes sense to use feature selection before using random forest?


Yes it does and it is quite common. If you expect more than ~50% of your features not even are redundant but utterly useless. E.g. the randomForest package has the wrapper function rfcv() which will pretrain a randomForest and omit the least important variables. rfcv function refer to this chapter. Remember to embed feature selection + modeling in a outer cross-validation loop to avoid over optimistic results.

[edit below]

I could moderate "utterly useless". A single random forest will most often not as e.g. regression with lasso regularization completely ignore features, even if these (in simulated hindsight) were random features. Decision tree splits by features are chosen by local criteria in any of the thousands or millions of nodes and cannot later be undone. I do not advocate cutting features down to one superior selection, but it is for some data sets possible to achieve substantial increase in prediction performance (estimated by a repeated outer cross-validation) using this variable selection. A typical finding would be that keeping 100% of features or only few percent work less well, and then there can be a broad middle range with similar estimated prediction performance.

Perhaps a reasonable thumb rule: When one expect that lasso-like regularization would serve better than a ridge-like regularization for a given problem, then one could try pre-training a random forest and rank the features by the inner out-of-bag cross-validated variable importance and try drop some of the least important features. Variable importance quantifies how much the cross-validated model prediction decreases, when a given feature is permuted(values shuffled) after training, before prediction. One will never be certain if one specific feature should be included or not, but it likely much easier to predict by the top 5% features, than the bottom 5%.

From a practical point of view, computational run time could be lowered, and maybe some resources could be saved, if there is a fixed acquisition cost per feature.

  • 7
    $\begingroup$ The ability of data to tell you that a feature is useless is severely limited, and I hope the option to which you refer is integrated into the random forest algorithm. It would not be appropriate to do up-front deletion of features before sending the candidate features to the random forest algorithm. $\endgroup$ Mar 10 '16 at 13:47
  • 1
    $\begingroup$ @FrankHarrell, I have tried to elaborate my answer $\endgroup$ Mar 10 '16 at 22:02
  • 2
    $\begingroup$ I disagree that you choose different scoring rules for different purposes. An improper accuracy scoring rule leads to selection of the wrong features and giving them the wrong weights. More apparent is the arbitrariness in certain scoring rules. It is far better to choose an optimum predictive model and then using solid decision theory to make optimum decisions using that model. This is done by applying a utility function to the continuous predictions. $\endgroup$ Mar 11 '16 at 12:22
  • 1
    $\begingroup$ @FrankHarrell - can you give a detailed answer to this question? apparently you have some strong arguments against doing feature selection... $\endgroup$
    – ihadanny
    Mar 21 '16 at 12:30
  • 2
    $\begingroup$ The best way to learn about this is to do rigorous bootstrap internal validation of a procedure that tries to do feature selection vs. one that does not. Quite often the predictive discrimination (when measured using a proper accuracy scoring rule or even with the $c$-index (ROC area)) is better when feature selection is not attempted. Feature selection is almost always arbitrary. $\endgroup$ Mar 21 '16 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.