1
$\begingroup$

Having a dataset of 1000 data items, with 25 attributes (a mixture of numerical and categorical attributes), a binary label and using random forest classifier (it's a cost-sensitive learning task as data is a bit imbalanced, if that matters) I perform backward feature elimination which results in 20 attributes and a tangible performance decrease.

What are some potential reasons for this happening? Let's say I have done a somewhat good job in manual feature selection having the domain knowledge. Does this decrease mean that automatic feature selection might be inappropriate having manual feature selection done, or something might be wrong with the way I'm doing automatic feature selection? If the answer is dependent on data, then I'm looking for some potential reasons that might be causing this.

Improve this question
$\endgroup$
3
  • $\begingroup$ If X variables contain Y amount of information, then X-1 variables will contain Y-delta information. Depending on how much information was contained in the variables you dropped, you will see a dip in performance. $\endgroup$
    – Arun Jose
    Commented Jan 10, 2017 at 8:42
  • $\begingroup$ @ArunJose There seems to be a strong assumption in your comment that the X variables are an optimal set. One does automatic feature selection because they think some attributes are likely to be redundant. $\endgroup$
    – Aliweb
    Commented Jan 10, 2017 at 18:53
  • $\begingroup$ This has nothing to do with optimality, it's a very simple axiom. If there are redundant variables it simply implied Delta should be small or negligible. $\endgroup$
    – Arun Jose
    Commented Jan 11, 2017 at 0:46

1 Answer 1

Reset to default
3
$\begingroup$

Random Forest is already designed to be robust to features that are not informative of the response. By using backwards elimination you are not doing the algorithm any favors, but you are making more decisions. Each of these decisions has a chance to be influenced by noise, and hence be wrong. As you make more decisions, you build up risk of componding your errors.

Instead, just let the forest do its job. Give it all your features and trust the algorithm to make good decisions. If you really, really, really need to remove features for some outside concern (efficiency or implementability), then when all is said and done, you can use the feature importances to find which features were not used by the algorithm, remove them in one go, and then retrain for a final model.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.