How to "choose" categorical variables which have impact in a regression?

Question

I have a dataset of about 50K samples. I have approximately 90 columns which are all categorical and they're used to predict a price. There's no other continuous value. I'm trying to select "which of those columns" have a greater impact on the final response variable. Here's an example:

Suppose we're trying to estimate salaries of developers. The variables we have are:

• Role: Data Scientist
• Main Language: Python
• IDE Used: PyCharm
• Country: UK
• Years of experience: +5 years

How can you "measure" the impact of these variables on the final salary? How can you decide which variables to use for your regression and which ones to leave aside because they're not "relevant" to the final dependent variable?

Thanks!

Answer 1

The problem you are faced with here is variable selection. In the context of regression analysis there are a number of common approaches to this problem that could be implemented here:

• All-possible-models: This might be computationally infeasible with so many available variables, but it might be possible to implement if you can cut-down with some other first-pass method. In this approach the research fixed a value $m$ for the number of parameters in the model and finds $\min SSE(n)$ taken over all possible models with that number of parameters. The researcher then plots the function $n \mapsto \min SSE(n)$ and looks for an appropriate number of parameters that reduces the residual-sum-of-squares to a desired level. This can be done by formal methods using partial F-tests, but often the researcher will use graphical assessment. (In the latter case the researcher will usually look for a "kink" in the function indicating that the rate of decrease of the -sum-of-squares is rapidly diminishing, so that there is little value in adding more parameters.)

• Automated variable selection: This method is built into penalty models such as LASSO regression and Ridge regression. These models penalise the inclusion of additional variables through a penalty function that operates on the absolute values of the model parameters. Model-fitting occurs via minimisation of the penalised log-likelihood, and this automatically leads some of the estimated model parameters to be equal to zero (thereby removing the variable from the fitted model). These methods can be implemented on data sets with categorical variables (see related question here). They require specification of penalty weights, and the results depend on the penalties applied. Higher penalty weights lead to less variables being selected.

In this particular case I would recommend that you consider a penalty model such as the LASSO regression or Ridge Regression. These models will automatically perform your variable selection, which will depend on your penalty weights. You can use a sensitivity analysis to see how the number of variables selected depends on the penalty weights used, and how this affects the residual-sum-of-squares in the model.