What are some good feature selection techniques for binary classification problems?

Question

I am performing binary classification on a data set with around 90k records and 28 features. I'd like to evaluate various models such logistics, SVM, Xgboost etc. via grid-search method to see which works best. I want to understand the best feature selection algorithms I can use to reduce the number of variables. I understand correlation heatmaps would be good to identify important variables in the case of regression problems. Is there any feature selection method that is more suitable for binary classification problems?

Answer 1

If you can compute the likelihood of your recordings for each model, I would recommend using model selection criteria, such as the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC). They allow to compare different models based on their capacity at explaining the observations while accounting for their complexity.

Classically used model selection criteria contain two terms:

1. A likelihood term, which measures the ability of the model to explain the observed data;
2. A regularization term, which penalizes the complexity of the model (i.e. its number of free parameters, or features in your case).

The point of the latter term is precisely to avoid overfitting and to allow that the selected model will generalize well to unobserved data, i.e. to avoid that the more complicated model (with the highest number of features) will be necessarily selected. This can be interpreted as a form of Occam's razor: if two models explain the data equally well, the simpler one should be favored.

An interesting reference on model selection criteria:

Burnham, K. P., & Anderson, D. R. (2004). Multimodel inference: understanding AIC and BIC in model selection. Sociological methods & research, 33(2), 261-304.

Alternatively, if you consider that you have enough observations to split them into a training set and a validation set, you could simply use cross-validation and see what is the ideal number of features to maximize the validation accuracy.