1
$\begingroup$

I'm using the random forest classifier (RandomForestClassifier) from scikit-learn on a dataset with around 30 features, 3000 data points and 6 classes. I'm using leave-one-group out as well as leave-one-out cross-validation.

RandomForestClassifier provides directly the importances of the features through the feature_importances_ attribute. I think the importance scores are calculated by averaging the feature importances of each tree which are retrieved by looking at the impurity.

Currently I'm fitting a random forest on the whole dataset and then I'm looking at the feature importances. Another option would be to retrieve the feature importances on each training set of each split of the cross-validation procedure and then averaging the scores.

Which one is the correct way to go?

Second, how can I calculate if one (or several) features have significant more importance than others (p-value)?

$\endgroup$
  • $\begingroup$ What are you using the importances to deduce? $\endgroup$ – Matthew Drury Jul 23 '18 at 18:46
  • $\begingroup$ @MatthewDrury I would like to gain insight into the features (i.e. which feature is most important) and then I would like to choose the five most important features (feature selection). $\endgroup$ – machinery Jul 23 '18 at 18:50
  • $\begingroup$ There's generally no reason to do feature selection in a random forest, at least statistically. You shouldn't expect it to meaningfully improve the performance of the model (as long as you are properly using random forest). I'll also point out that your reasoning is a tautology: "I want to use feature importance metrics from a random forest to tell what features are most important." That sentence doesn't mean anything. I'm sorry to be blunt, but it has no conceptual content. WHY do you want to know that, what questions will you use it to answer? $\endgroup$ – Matthew Drury Jul 23 '18 at 19:41
  • 1
    $\begingroup$ @MatthewDrury I disagree. Let's say I have different groups of features (i.e. features related to different concepts). It would be insteresting to know if the top performing features are all from the same group for example. If that is the case one could focus on that group and derive other features. I think it is always good to know and interesting which features contribute the most. $\endgroup$ – machinery Jul 23 '18 at 20:00
  • $\begingroup$ @machinery you can't do that with RF feature importance, or at least not one shot of it. If one feature has been chosen, then another candidate feature is considered (also a good predictor). If the second doesn't add much information gain because of correlation with first feature, then it will be way down ranked in importance. Even though it might have been almost just as good as the first $\endgroup$ – jonnor Jul 25 '18 at 20:23
1
$\begingroup$

In Random forest, generally the feature importance is computed based on out-of-bag (OOB) error.

To compute the feature importance, the random forest model is created and then the OOB error is computed. This is followed by permuting (shuffling) a feature and then again the OOB error is computed. Like wise, all features are permuted one by one. Based on the increase (which is the score) in the OOB error, the feature importance is estimated. Higher this increase, higher the importance. These scores are then divided by the standard deviation of all the increases.

Apart from this, gini impurity measure can also used to estimate feature importance.

For more details refer this lecture note.

Which one is the correct way to go?

To compute the feature importance, you can use the complete dataset as the importance estimate is computed based on the OOB observations which are actually the left out observations during the bootstrap aggregation process for each tree in a Random Forest.

Second, how can I calculate if one (or several) features have significant more importance than others (p-value)?

Consider the feature importance values to compare the relevance of the features. To get p-value, statistical tests such as ANOVA (for parametric) or Kruskal-Wallis test (for non-parametric) could be used.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.