0
$\begingroup$

I use sklearn's random forest for classification (two categories).

The following code is used for "good" feature selection.

classifier = RandomForestClassifier()
classifier.fit(feature_vectors, target)
model = SelectFromModel(classifier, prefit=True, threshold=0.00001)
masks = model.get_support(True).tolist()

The features are binary, meaning either 1 (have) or 0 (not have).

For the result, I do not quite understand.

If any sample has Feature 1, it must be in category A. However, only very limited samples have Feature 1.

A lot of samples have Feature 2, both in category A and category B, almost equally.

For the result, I noticed that Feature 2 is selected, and Feature 1 is not.

For a good feature, shouldn't Feature 1 is better than Feature 2 ?

$\endgroup$

1 Answer 1

2
$\begingroup$

The short answer is: it depends on the rest of the feature set.

Unless you restrict your random forest to 'stumps' (i.e. trees of depth 1) then your algorithm will test combinations of features when it builds deeper trees. Feature 1 may perform very well in positively identifying $A$, but that leaves many observations for which you need other features. Feature 2 may not be informative on its own, but in combination with other features may perform well. So you can't really judge them on their own in most random forests.

There are at least couple of ways to tackle this:

  1. Try building a basic decision tree and see how your features interact. It may not be the model you'll use, but it can be a useful indicator.
  2. Inspect variable importance plots for your random forest. (I don't know exactly how to do this with Python as I mostly use R, but it seems that you can do this with scikit-learn.) This will indicate which features are most important (i.e. have the most predictive power).

In the end the question of what makes a 'good' feature is far from trivial in general, but this may help with answering it in this particular context.

NB. You might also find Max Kuhn's new book interesting: Feature Engineering and Selection is all about this topic, and the draft version is available online for free.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.