2
$\begingroup$

I am recently learning machine learning and got to know about feature selection.

I am wondering if a wrapper method like "Recursive feature elimination" provided by scikit tests cross validation on all subset of features, isn't it the best way to select features compared to other feature selection methods. Such as, Removing features with low variance.

Because, "Recursive feature elimination" checks all subset of features in the training set.

If not, please explain the reason.

$\endgroup$
  • $\begingroup$ The problem with your statement is " isn't it the best way to select features". There really is no one size fits all. There have been debate after debate about multiple feature selection methodologies and the pitfalls in each of them. The only true way to find the right model is to test EVERY single possible model. When that is not possible, we reduce the search space. Recursive elimination is one such attempt at this. $\endgroup$ – Arun Jose Aug 30 '16 at 11:15
  • $\begingroup$ As you mentioned, the right model is to test every possible models. That is why I said, isn't Recursive elimination is the best one because it tests all possible subset of the features. Does it not test all possible models? $\endgroup$ – hitechnet Aug 31 '16 at 1:25
  • $\begingroup$ It does not test all models. It starts with your initial set and then keeps pruning variables in each iteration. Variables once pruned are not tested again at a later stage. Eg: a,b,c --> a,b --> a, here a,c never gets tested because c gets dropped in the initial step. $\endgroup$ – Arun Jose Aug 31 '16 at 10:41
2
$\begingroup$

I think you are confusing RFE with search algorithms that search over all possible subsets of features. RFE does something similar, but it does not check all possible combinations. At every step, RFE simply eliminates a certain number or a certain percentage of the lowest ranking features in your model, and retrains. This makes the assumption that these features are not important to begin with, so they can be eliminated. It then continues eliminating features as such, until your stop criterion is reached. Depending on your problem, the resulting features may or may not be the same ones you would obtain if you searched over all combinations of features. Most likely, they are not.

For many real world problems searching over all possible subsets of features is intractable. RFE becomes a good compromise.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.