I am new to ML and learning about feature selection and reduction. I have a few questions

q1) I see there are a lot of feature selection algorithms such as Lasso, RFE, Chi-squared, Genetic Algorithm, Stepwise forward and Stepwise backward etc.

How do we know which feature selection algorithm to use under which scenarios? Do we have any guidelines/rules on when to use which?

q2) Is there any difference between Feature selection and Feature reduction? If yes, can help me understand the difference?


1 Answer 1


I would generally advise not using feature selection unless selecting the informative features is an intrinsic goal of the analysis (i.e. to discover some useful knowledge about the system, such as identifying key genes from a gene expression analysis). If you just want to improve generalisation performance, feature selection usually makes things worse if you are using a modern machine learning method (e.g. SVMs) and regularisation (e.g. choosing the best value of C for an SVM) is normally a better approach.

It is easy to over-fit the feature selection criterion (Ambroise and MacLachlan, 2002), so I am generally rather skeptical about methods that aggressively search for a global minima (such as genetic algorithms).

The naive approach to (wrapper-based) feature selection essentially introduces one binary hyper-parameter per feature, that indicates whether it is in the model or not. This potentially introduces a lot of degrees of freedom to over-fit the feature selection criterion. I generally prefer methods like LASSO/L1 regularisation, which has a single, continuous hyper-parameter controlling the sparsity of the feature set. This imposes a structure on the feature selection and generally less of an opportunity for over-fitting.

I strongly recommend reading "An Introduction to Variable and Feature Selection" by Guyon and Elisseef, which is an excellent tutorial paper on feature selection.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.