Methods for feature elimination of categorical variables? I have a dataset with 56 columns, 4 numeric and the target variable which is also numeric. I am trying to eliminate some of the categorical variables from my model and wanted to get some understanding of the methodologies to do that.
I am currently working in R, which is somewhat new to me.
Any help would be great!
Edit: I would like to eliminate some variables to increase my models performance. While the model is not performing poorly, I believe the model can perform better than it is now. Some of the variables I can take a look at and know right off the bat that this is something that I can remove. But, I would like to show proof that this is not an informative predictors.
Edit2: Here is information about my dataset.
Out of the columns that are categorical, there are roughly 5 variables that have levels greater than 2. My goal is essentially to remove variables that are the least informative to my model. My model is a regression model to predict the sale price of something. Using accuracy as a metric, I want to have the most accurate model and remove predictors (both numeric and categorical) that are not conducive to the predicting accuracy of my model.
In removing categorical variables, I would be removing the entire variable, not particular levels.
 A: The group LASSO was designed specifically to allow groups of predictors to be selected or omitted together, so that is what you would want to use in your case to keep all levels of a categorical predictor together. There are several implementations in R. This, unlike stepwise selection, provides a principled way to decrease the number of predictors; you can choose the LASSO penalty (and thus the number of maintained predictors) by minimizing cross-validation error to get a good tradeoff between parsimony of predictors and performance while providing reasonable application to new data samples. It seems that you do not want to omit any of your numeric predictors, so make sure that the implementation you use allows for forced inclusion of chosen predictors.
There is one unavoidable difficulty with LASSO or ridge regression with categorical predictors: whether/how to normalize those predictors before doing the analysis, which applies a single penalty to all predictors. Normalization is clearly necessary with numeric predictor values: you don't want the penalization to depend on the scale of measurement (e.g., millimeters versus miles). So programs may default to scaling all predictors to 0 mean and unit standard deviation for calculations, then back-transforming to the original scales for reporting coefficient values.
But is that what you want with a binary categorical predictor, where the mean and SD necessarily depend on the class prevalence? Does such normalization even make sense with a multi-level categorical variable? Depending on the implementation, your results with a multi-level categorical predictor might even differ depending on your choice of reference level. So spend some time thinking about what you want to accomplish, the details of how your chosen LASSO implementation works, and whether or not to specify normalization for the categorical predictors.
A: Here are some ways to do this:
1) Lasso: Perform regularized regression with the lasso penalty. This will force most of your predictors to be 0.
2) Forward selection: At any point, greedily add the feature that adds most predictive value.
3) Backward selection: Start with all predictors and greedily remove the feature that if removed affects the prediction quality the least.
