I have trained a regression forest using the R package ranger. Now I would like to discuss the variable importance measures of the included features. However, I am having difficulties understanding the exact definition of the different importance measures offered by ranger.
The documentation of the ranger function states the following about the argument 'importance':
Variable importance mode, one of ’none’, ’impurity’, ’impurity_corrected’, ’permutation’. The ’impurity’ measure is the Gini index for classification, the variance of the responses for regression and the sum of test statistics (see splitrule) for survival.
In my case, I trained a regression forest. Here is a reproducible example including all feature importance measures offered by ranger:
library(ranger)
## Estimate the same model with the three different importance measures offered by ranger
set.seed(1)
rg.iris_1 <- ranger(Sepal.Width ~ ., data = iris, importance = 'impurity')
set.seed(1)
rg.iris_2 <- ranger(Sepal.Width ~ ., data = iris, importance = 'impurity_corrected')
set.seed(1)
rg.iris_3 <- ranger(Sepal.Width ~ ., data = iris, importance = 'permutation')
## Store the importance measures in a table
varimp_table <- data.frame(impurity = rg.iris_1$variable.importance,
impurity_corrected = rg.iris_2$variable.importance,
perumtation = rg.iris_3$variable.importance)
This gives the following table:
And here are my questions:
- How can I interpret the three columns of this table?
- Are there any papers explaining what is meant by the different approaches?
- What are the pro's and con's of each measure?
Edit:
After trying to make sense of the code on github, this seems to be the part where the variable importance is calculated in the case of the given example:
double decrease = sum_left * sum_left / (double) n_left + sum_right * sum_right / (double) n_right;
and for example, sum_right seems to be defined here:
for (size_t pos = start_pos[nodeID]; pos < end_pos[nodeID]; ++pos) {
size_t sampleID = sampleIDs[pos];
double response = data->get_y(sampleID, 0);
double value = data->get_x(sampleID, varID);
size_t factorID = floor(value) - 1;
// If in right child, count
// In right child, if bitwise splitID at position factorID is 1
if ((splitID & (1ULL << factorID))) {
++n_right;
sum_right += response;
}
}
I never learned C++, but this seems as if the squared response is averaged and not the response variance, as it is stated in the description of the package. This wouldn't really make sense, though... would it?
Please help!
