Since decision tree don't use all the input features and select them in the process, is it useful to do feature selection before?

As I see it, choosing features will decrease computing time (and decrease overfitting risk on small dataset?), but as multiple weak features can perform better than strong ones, I may also have a worse prediction.

EDIT : Bonus question : Is there a way to select features before a decision tree, or should I let it do the work ?

  • $\begingroup$ The question is how can you select them before the decision tree? $\endgroup$ – Metariat Sep 19 '16 at 7:36
  • $\begingroup$ I was thinking using a feature selection technique, maybe LASSO or something else $\endgroup$ – CoMartel Sep 19 '16 at 7:46
  • 1
    $\begingroup$ Variables that are important in LASSO don't necessarily have the same relationship with the outcome as in decision tree. You can see a related question here: stats.stackexchange.com/questions/164048/… $\endgroup$ – Metariat Sep 19 '16 at 7:53
  • $\begingroup$ Ok, I see your point. The remaining question is : is there a way to select features before a decision tree, or should I let it do the work ? $\endgroup$ – CoMartel Sep 19 '16 at 8:11
  • $\begingroup$ In my personal experienc, I don't see any way to select features before building the tree. $\endgroup$ – Metariat Sep 19 '16 at 8:12

Decision Trees are pretty good at finding the most important features, they consider all features and create a split on the one that is separating class labels the best (in terms of entropy).

If you use Random Forests it's even better, because some implementations (like scikit-learn's) are capable of sampling the features and use only a subset of it. Also in general Random Forests are more robust than decision trees.

If you want, you can compute Information Gain before using a Decision Tree to see how much information a particular feature contains regarding the Label:



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.