While doing a classification I have to choose from the ocean of choices at every step like model selection, performance criteria selection and all. But the important two things I get confused most of the times are. 1. Model Selection 2. Feature Selection.

Can someone tell me that what is the criteria to select and important rules for feature selection?

For example We remove the multi-colinear variables that are highly co related. Which selection technique is better than the other. For example Chi_Square, ANOVA, Variance Threshold. Which one to use and when?

What are the basic rules of thumb if there are any?

  • $\begingroup$ Try searching this site for "feature selection", it's a very wide topic $\endgroup$ – Itamar Mushkin Dec 11 '19 at 9:54

The answer is of course quite complex, and the answer is, as often, it depends.

For model selection it depends on your constraints. Usually a good rule of thumb is starting with the easiest model. An example for classification would be logistic regression. If it works it's a win, if it doesn't it's a good benchmark for more powerful models. Ridge and Lasso are simply regularizations used on regression models. Ridge simply avoids overfitting, Lasso is also reducing the dimensionality of your problem, setting unimportant coefficients to 0. Therefore, if you need to have better model interpretability you can go with Lasso regularized linear models. Otherwise, if you only need performance, start looking into Ensemble methods (boosted machines, GAMs -which are closer to regression but still-, RFs) or SVMs.

For feature selection, again, it depends on the model. Lasso automatically selects important features, but you should still remove over-correlated features before. I usually compute a correlation matrix and see if there are any important dependencies that I can remove (usually via correlation threshold). Also, as some models might be affected by too many useless features, you can remove them in backwards steps by looking at their coefficients/importances. Finally, there's feature transformation methods such as PCA/SVD which can be useful in cases where you need to reduce the number of features and you don't need model interpretability

All this however is very field and problem dependent, so the best way is experimenting yourself over multiple datasets.

Edit: This is what I meant when I said removing features in backwards steps (RFE): https://towardsdatascience.com/feature-selection-in-python-recursive-feature-elimination-19f1c39b8d15


Not the answer you're looking for? Browse other questions tagged or ask your own question.