5
$\begingroup$

We have 10 features that is pre-selected from domain knowledge. We ran random forest with those features. one of the feature has zero feature importance. My question is:

  1. For those features that has zero importance in the random forest model, should I remove it and rerun the model?
  2. I did try that. When I remove the feature and rerun random forest, the importance of 7th important feature became zero, what should I do? Thanks a lot for any expert opinion...
Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

A more rigorous way to pursue this question is to apply the Boruta algorithm.

Boruta repeatedly measures feature importance from a random forest (or similar method) and then carries out statistical tests to screen out the features which are irrelevant. The procedure terminates when all features are either decisively relevant or decisively irrelevant.

There are several papers on this topic. Here's one. "The All Relevant Feature Selection using Random Forest" by Miron B. Kursa, Witold R. Rudnicki

Improve this answer
$\endgroup$
6
  • $\begingroup$ thank you very much for your input. We actually did try Boruta. We selected features that is confirmed in Boruta procedure, ran random forest and got same issue. I'm not sure what to do. If I leave the zero importance feature there, it'll be hard to defend our model later... @Sycorax $\endgroup$
    – zesla
    Nov 13, 2018 at 17:21
  • $\begingroup$ This doesn't really sound like a problem. Boruta is an "all relevant" feature selection method. It's conceivable that there are relevant features which are nonetheless less informative than others -- so much less informative that the model never chooses to split on them. Consider two features, one is highly relevant to the outcome and the second is correlated to the first but very noisy. The second feature is relevant, but the noise makes it much less informative. $\endgroup$
    – Sycorax ♦
    Nov 13, 2018 at 17:27
  • $\begingroup$ The issue is if I should leave the feature in my model. If the model does not choose it to split, then it sound reasonable to exclude it. But once I exclude one and rerun the model, I got another one with zero importance..... @Sycorax $\endgroup$
    – zesla
    Nov 13, 2018 at 17:34
  • $\begingroup$ Why does it matter either way? Including a feature that is never used isn't a problem -- the model is still making predictions using the random forest procedure. $\endgroup$
    – Sycorax ♦
    Nov 13, 2018 at 17:35
  • $\begingroup$ In our application, features selected is important for interpretation. If a feature is not used in model, how am I respond to people who ask why not exclude the feature not used. @Sycorax $\endgroup$
    – zesla
    Nov 13, 2018 at 18:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.