I am using random forests to try and determine variable importance as part of feature selection for a model I'm working on, and while I can get ranked variable importance by mean decrease in Gini from removal, I'm not sure what a good cutoff would be for including a variable in a model or not. For example, if the decrease in Gini from removing a variable is 30, is that variable worth including? Are there any good rules of thumb on this topic, or does it depend on the model?
$\begingroup$
$\endgroup$
4
-
$\begingroup$ Woudldn't the best threshold depend on the data. If you have only univariate data, then GINI will say "cut all but the information-holding variable". If your variables are all equally informative, wouldn't GINI say "keep them all"? $\endgroup$– EngrStudentCommented May 28, 2013 at 19:32
-
$\begingroup$ Certainly I could keep them all. There's not a lot of collinearity based upon VIF calculations, but there are a great many very low information variables and for calculation speed purposes I was trying to determine if there was a heuristic in the data science community for what lower bound information loss was acceptable. That's all. If I need to build the model with all of them, then so be it. $\endgroup$– Tom.RampleyCommented May 29, 2013 at 16:14
-
1$\begingroup$ Why not run it iteratively? You could run a "quick and dirty" compute to determine the bottom 10% of contributors, and cull them. Repeat until your cross-validation error starts changing meaningfully. At that point, step back a little and do an "in-depth" or "compute intensive" run. $\endgroup$– EngrStudentCommented May 29, 2013 at 16:31
-
1$\begingroup$ After doing a little more research, I went with the Boruta algorithm as implemented in the R package Boruta, and that helped a great deal (and conceptually is very cool as well). I ran VIF analysis as well, and that largely verified the Boruta output. So, problem somewhat solved. $\endgroup$– Tom.RampleyCommented May 30, 2013 at 15:52
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
You might want to use a permutation importance (mean decrease in accuracy) rather than Gini (decrease in node purity), if your random forest implementation supports it. (R's randomForest with the importance=TRUE option, for example.) Both of these measures are biased towards continuous variables and categorical variables with lots of levels (i.e. variables with many splitting opportunities) if you're working with a mixture of variable types.
Given the choice between these two (possibly biased) methods, permutation importance seems more interpretable to me. Oh, in R's randomForest, these values will be normalized unless you also add the option scale=FALSE.
It appears that Boruta uses permutation importance, and automates the inclusion and exclusion of variables based on that.
$\begingroup$
$\endgroup$
My solution to this problem has been to generate several random variables and add them to the feature set. Any feature that performs worse than the mean of the noise variables gets dropped.
