4
$\begingroup$

I want to use the decrease in Gini impurity to rank features for my random forest classifier. I understand that the decrease in Gini impurity at one node is calculated as:

$$ \Delta i(n) = i(n) - p_li(n_l) - p_ri(n_r) $$

The overall decrease in Gini impurity is summed over all nodes and all trees for a given node ref. I don't quite understand if there is a link between decrease in Gini impurity and the prediction performance? That is, Gini impurity says which features are more important relative to others. But can I deduce how much individual features will affect prediction performance given the Gini impurity? I have read the following posts:

Gini decrease and Gini impurity of children nodes

What is the relationship between the GINI score and the log-likelihood ratio

$\endgroup$
  • $\begingroup$ (1) are you referring to random forests? If so you might want to be explicit. (2) what other measures of predictor importance are there for random forest? $\endgroup$ – charles Oct 30 '14 at 15:46
  • $\begingroup$ @charles: Thanks. to (2) there is also the decrease in accuracy for the obb sample.which have a more clear interpretation. $\endgroup$ – Cici Oct 30 '14 at 16:04
  • $\begingroup$ In response to your flag, this question is perfectly on-topic on CV, and it does not need to be migrated on data science.se. One way to attract attention is to offer a bounty or edit your question so that it get visible on the main page. $\endgroup$ – chl Nov 4 '14 at 19:10
2
$\begingroup$

I'm not sure there is a great answer to this questions. But maybe

As far as I know there were four measures of variable importance in the original Breiman paper, with only two making it into the randomForest package. But the permutation variable importance appears to be much more popular, likely because it is much easier to understand how this might produce intuitive measures of variable importance and how these measures of variable importance relate to predictive ability of the model. My understanding is that valSelRF, Boruta and conditional variable importance in party all use the permutation variable importance.

Variable importance are suggestive, but hard to make inferences from output. That might be why there are so many ways of calculating variable importance. the relaimpo package has six measures. Both the Boruta and relaimpo package vignettes are worth reading. Discuss subject at length. The relaimpo vignette especially emphasizes limitations of methods and the conflicting results you can obtain. The methods may be efficient at finding all-relevant feature, but often produce conflicting results when ranking feature. (You'll find the values also change when you change the cost function. )

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.