Feature selection with categorical variables

Question

I am working on the diamonds dataset. In it, we model the price of those diamonds based on several predictors.

Three of them are categorical variables (cut, color and clarity) and the rest are numerical.

I have identified the numerical variables that will make good predictors but I don't know how to do that for the categorical ones.

Specifically, I want to select only the categorical variables that will help the regression model, while keeping the rest out. Also, if several categorical variables are redundant (they give us the same information) I would like to keep the best and discard the rest.

How can I do that in this context?

Bonus question: is it ok to remove the least informative categories of a categorical variable and use the rest (as dummies)?

Answer 1

In my own opinion, feature selection is only appropriate when you are constrained by something else unrelated to the problem (e.g. your model must make predictions on a device with limited memory, hence you need a minimal data to run predictions on).

Feature selection is actually a bad idea if you want to interpret your model. Many studies have noted that feature selection techniques result in: bias away from the null, exaggerated precision, inaccurate or uninterpretable p−values due to inability to properly incorporate uncertainty in the selection process, and can fail to select the ”true” model with high confidence even when modelling assumptions are consistent with the true data generating process. If your goal is evaluation of a hypothesis, then you should select your features a priori based on what you think affects the outcome (either through previous studies or through intuition or something else).

Feature selection for prediction problems is also not needed. Rather than select features, penalization should be applied. This prevents an "in or out" mentality and allows features to be shrunk towards 0.

Lastly, picking categories from a categorical variable to keep/reject will likely add bias to the predictions/inference since if there is heterogeneity in the response, then the "left out" category (the case where all the selected categories are 0) is a heterogenous mix of outcomes.