The extremely randomized trees classifier (scikitlearn) provides a (multivariate) feature importance measurement Ensemble methods/feature importance evaluation. For each feature, the classifier produces a statistical measurement (and the corresponding standard deviation) for how important the feature was for predicting the target variable. The basic use of this information is to create a "feature ranking" among the features (from high importance value to low).

The question that I have is if I can use this information to make a conclusion that my features are weak or strong for my training data? Has the size of the "importance value" a meaning in itself? For instance, the highest ranking feature has an importance value of 0.0494 with a standard deviation of 0.024. Can I make the conclusion that the features are weak because the "importance values" are very small?

  • $\begingroup$ I do not think that you can judge the feature as a weak one relative to the other featuers which are ranked lower. Also, In your example, 0.0494 means almost 5% of the samples is correctly classified by this feature !! $\endgroup$ – soufanom Nov 29 '12 at 3:38
  • 1
    $\begingroup$ @soufanom Thanks for the clarification of the meaning of the statistical "measurement of importance". Could you expand a little more about your first sentence. My understanding is that if a feature is ranked lower (lower feature importance value), it is less important for the classification of the target data. If a feature is ranked higher it is more important for the classification. In that sense, the "feature importance", can be used to compare one feature to the other and to create a ranking list. $\endgroup$ – user963386 Nov 29 '12 at 7:43
  • $\begingroup$ In your post, the feature as stated was ranked highest with the stated importance value. In decision trees, if the feature is ranked in a higher node (not necessarily a high score), then, it is of higher importance in this case. Thus, the node level is the actual ranker rather than the score itself. However, the score can be used to compare the features in the tree !! I hope that I have clarified your point !! $\endgroup$ – soufanom Nov 29 '12 at 11:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.