I don't quite get how random forests feature selection works. I know this idea of looking at how much the tree nodes that use the feature reduce impurity (say Gini). I need a more detailed explanation for the case of one tree and then for random forests. Please, would you mind explaining it to me?
closed as too broad by Sycorax, kjetil b halvorsen, Michael Chernick, Carl, Peter Flom♦ Nov 1 '17 at 12:46
Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
A random forest is a collection of decision trees, so understanding how random forest feature selection works means understanding how it works in decision trees.
I found this explanation helpful (it's worth reading the whole blog post):
Random forest consists of a number of decision trees. Every node in the decision trees is a condition on a single feature, designed to split the dataset into two so that similar response values end up in the same set. The measure based on which the (locally) optimal condition is chosen is called impurity. For classification, it is typically either Gini impurity or information gain/entropy and for regression trees it is variance. Thus when training a tree, it can be computed how much each feature decreases the weighted impurity in a tree. For a forest, the impurity decrease from each feature can be averaged and the features are ranked according to this measure.
After a tree has been created, one can check the importance of a variable by looking at the difference in some measure (such as Gini impurity) when the feature is used compared to the case when the feature is not used.
How is the feature importance actually calculated? There are several ways.
Gini Importance or Mean Decrease in Impurity (MDI) calculates each feature importance as the sum over the number of splits (across all trees) that include the feature, proportionally to the number of samples it splits.
Permutation Importance or Mean Decrease in Accuracy (MDA) is assessed for each feature by removing the association between that feature and the target. This is achieved by randomly permuting the values of the feature and measuring the resulting increase in error. The influence of the correlated features is also removed.
This suggests that if a feature is used in more than one splitting, the sum of the reduction in Gini impurity is the determines its importance (when using Gini impurity).