I have a dataset of about 50K samples. I have approximately 90 columns which are all categorical and they're used to predict a price. There's no other continuous value. I'm trying to select "which of those columns" have a greater impact on the final response variable. Here's an example:

Suppose we're trying to estimate salaries of developers. The variables we have are:

  • Role: Data Scientist
  • Main Language: Python
  • IDE Used: PyCharm
  • Country: UK
  • Years of experience: +5 years

How can you "measure" the impact of these variables on the final salary? How can you decide which variables to use for your regression and which ones to leave aside because they're not "relevant" to the final dependent variable?


  • $\begingroup$ Do you mean you have 90 different predictors (I am not sure what you mean by a column). You can run a linear regression with 90 dummy variables although you really need to try to come up with some theory to limit them to a more manageable numbers. While methods such as stepwise regression exist to suggest what variables to add many are extremely critical of using such automatic approaches to select variables. From painful experience statistical approaches, including regression slope really don't tell you relative impact. $\endgroup$ – user54285 Mar 19 at 22:28

The problem you are faced with here is variable selection. In the context of regression analysis there are a number of common approaches to this problem that could be implemented here:

  • All-possible-models: This might be computationally infeasible with so many available variables, but it might be possible to implement if you can cut-down with some other first-pass method. In this approach the research fixed a value $m$ for the number of parameters in the model and finds $\min SSE(n)$ taken over all possible models with that number of parameters. The researcher then plots the function $n \mapsto \min SSE(n)$ and looks for an appropriate number of parameters that reduces the residual-sum-of-squares to a desired level. This can be done by formal methods using partial F-tests, but often the researcher will use graphical assessment. (In the latter case the researcher will usually look for a "kink" in the function indicating that the rate of decrease of the -sum-of-squares is rapidly diminishing, so that there is little value in adding more parameters.)

  • Automated variable selection: This method is built into penalty models such as LASSO regression and Ridge regression. These models penalise the inclusion of additional variables through a penalty function that operates on the absolute values of the model parameters. Model-fitting occurs via minimisation of the penalised log-likelihood, and this automatically leads some of the estimated model parameters to be equal to zero (thereby removing the variable from the fitted model). These methods can be implemented on data sets with categorical variables (see related question here). They require specification of penalty weights, and the results depend on the penalties applied. Higher penalty weights lead to less variables being selected.

In this particular case I would recommend that you consider a penalty model such as the LASSO regression or Ridge Regression. These models will automatically perform your variable selection, which will depend on your penalty weights. You can use a sensitivity analysis to see how the number of variables selected depends on the penalty weights used, and how this affects the residual-sum-of-squares in the model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.