Scikit learn models gives weight to random variable? Should I remove features with less importance?

Question

I do some feature selection by removing correlated variables and backwards elimination. However, after all that is done as a test I threw in a random variable, and then trained logistic regression, random forest and XGBoost. All 3 models have the feature importance of the random feature as greater than 0. First, how can that be? Second, all models have it ranked toward the bottom, but it's not the lowest feature. Is this a valid step for another round of feature selection -i.e. remove all those who score below the random feature?

Answer 1

The boruta feature selection process is exactly designed to only retain features which are more important than randomized features, so your suggestion is on the right track.

Boruta works by creating a randomized version of every feature in your data set and then training a classifier, such as a random forest model. The features which consistently have a higher feature importance than the highest randomized feature are retained. This process is repeated to recursively eliminate the features which are consistently less informative than a randomized feature. By "consistently," I mean that a statistical test concludes to either include or exclude the feature. The procedure terminates when all features have been classified as either informative or uninformative.

Kursa, Miron B., and Witold R. Rudnicki. "Feature selection with the Boruta package." (2010).