5
$\begingroup$

After creating a Random Forest object using randomForest with around 500 candidate variables, I used importance(object) to display IncNodePurity for each of the candidate variables in relation to the binary outcome of interest (Payment/No Payment).

I am aware that IncNodePurity is the total decrease in node impurities, measured by the Gini Index from splitting on the variable, averaged over all trees. What I don't know is what should be the cutoff for candidate variables to be retained after making use of randomForest for feature selection in regards to binary logistic regression models. For example, the smallest IncNodePurity among my 498 variables is 0.03, whereas the largest IncNodePurity is 96.68. In summary, I have one main question:

Is there a cutoff for IncNodePurity? If yes, what is it?

If no, how do you determine the cutoff? Do you simply take 10 candidate variables with the largest IncNodePurity if you want a model with only 10 predictor variables?

Any thoughts or references are greatly appreciated. Thanks!

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

I don't believe such a cutoff exists, although the variable importance plots can be informative. Carry out two experiments. Rerun the random forest and see how the list changes. Delete an observation and also observe. In my experience, the answer relates to what is the goal of feature selection. For example, why not use every variable to make predictions. The random forest can easily do that. We usually use feature selection for a reason, for example, seeking a rule using just a small number of features that can easily be measured in the future. In my case, the number was set by the technology I plan to use the diagnostic on. More importantly, if you are using feature selection, this has to be repeated at each iteration of cross-validation. Searching this site for "cross-validation", "feature selection", and "stepwise regression" will give you a start.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I am very familiar with feature selection and stepwise regression, although I would prefer to never use stepwise regression again, which is why I am interested in learning more about a possible cutoff that may or may not exist. $\endgroup$
    – Matt Reichenbach
    Commented Sep 5, 2013 at 12:27
  • 1
    $\begingroup$ Just to be clear, I referred to stepwise only because a lot of relevant issues are discussed on those questions through the critique of those techniques. In relation to the current question, I wanted to emphasize the importance of repeating feature selection when one is carrying out resampling. It sounds like that was already your plan. $\endgroup$
    – julieth
    Commented Sep 5, 2013 at 14:39
  • $\begingroup$ Thanks @julieth for the clarification as you are very correct! $\endgroup$
    – Matt Reichenbach
    Commented Sep 5, 2013 at 15:37
  • 1
    $\begingroup$ Try to have a look at the package Boruta to find that cutoff. Anyway, logistic regression is a highly biased model - you might not exploit the same information that random forest does on the selected set of variables. $\endgroup$
    – Simone
    Commented Sep 6, 2013 at 0:45
2
$\begingroup$

In my opinion you need to make a series of nested models. If the goal is to identify influential variables - then compare models sequentially with likelyhood ratio test (or other appropriate test). If the goal is to obtain predictive model - then compare cross-validation statistics, or make test set and evaluate predictivity with it.

Improve this answer
$\endgroup$
2
  • $\begingroup$ the goal, in my case, is to obtain a predictive model, can you provide more information on "cross-validation statistics"? $\endgroup$
    – Matt Reichenbach
    Commented Sep 4, 2013 at 17:55
  • 1
    $\begingroup$ This can be mean squared error obtained via cross-validation. Cross-validation is very useful trick, I'd suggest you to find its explanation in web - that would be more clear than explanations of mine. In R there is function cv.glm() in package boot that can help you. $\endgroup$
    – O_Devinyak
    Commented Sep 4, 2013 at 18:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.