I don't quite get how random forests feature selection works. I know this idea of looking at how much the tree nodes that use the feature reduce impurity (say Gini). I need a more detailed explanation for the case of one tree and then for random forests. Please, would you mind explaining it to me?


closed as too broad by Sycorax, kjetil b halvorsen, Michael Chernick, Carl, Peter Flom Nov 1 '17 at 12:46

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ What do you mean by feature selection? Are you asking how to down-select to some number of "important" features? Or are you asking how random forest chooses which features to split? Or something else? $\endgroup$ – Sycorax Oct 31 '17 at 16:48

A random forest is a collection of decision trees, so understanding how random forest feature selection works means understanding how it works in decision trees.

I found this explanation helpful (it's worth reading the whole blog post):

Random forest consists of a number of decision trees. Every node in the decision trees is a condition on a single feature, designed to split the dataset into two so that similar response values end up in the same set. The measure based on which the (locally) optimal condition is chosen is called impurity. For classification, it is typically either Gini impurity or information gain/entropy and for regression trees it is variance. Thus when training a tree, it can be computed how much each feature decreases the weighted impurity in a tree. For a forest, the impurity decrease from each feature can be averaged and the features are ranked according to this measure.

After a tree has been created, one can check the importance of a variable by looking at the difference in some measure (such as Gini impurity) when the feature is used compared to the case when the feature is not used.

How is the feature importance actually calculated? There are several ways.

Gini Importance or Mean Decrease in Impurity (MDI) calculates each feature importance as the sum over the number of splits (across all trees) that include the feature, proportionally to the number of samples it splits.

Permutation Importance or Mean Decrease in Accuracy (MDA) is assessed for each feature by removing the association between that feature and the target. This is achieved by randomly permuting the values of the feature and measuring the resulting increase in error. The influence of the correlated features is also removed.

This suggests that if a feature is used in more than one splitting, the sum of the reduction in Gini impurity is the determines its importance (when using Gini impurity).

  • 2
    $\begingroup$ The picture is clearer now. Thanks. The question I still have is considering one tree. If we take one tree, how can we measure the importance of a feature within it. I mean I get that we use a criterion to do so (e.g Gini impurity). What I don't get is if a feature is the criterion used for splitting a first time and then a second time, its importance is quantified as the sum of the reduction in Gini impurity or the sum divided by the number of times it's used as a criterion for splitting...? $\endgroup$ – chsafouane Oct 31 '17 at 15:28
  • 1
    $\begingroup$ Updated the answer. As far as I can tell, the scikit-learn implementation divides the sum of the total importance by the number of trees and nodes in the tree. See line 360 and line 1056 $\endgroup$ – Edgar H Oct 31 '17 at 16:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.