Is it worthwhile to use LASSO for variable selection even if it is cumbersome?

Question

My goal is to understand which variables out of 10 actually explain my dependent variable for a voter/election research.
Now I have put all 10 variables into a multiple regression and have filtered out all that have p value over 0.1. Now I have a model with 3 variables, all p values at 0.014 or far below and r=0.8. It seems all fine.

Can't I just use this model? Or do I absolutely have to use lasso? Because I have tried to bring lasso to work for hours now. I can't do it (only experienced in excel). It's really frustrating all the errors R throws at me. If I have to use lasso, how can I do this the easiest? Is there any way to do LASSO or something similar with excel?

Answer 1

Variable selection is not compulsory. The idea that you have to throw away variables is wrong. Actually, unless there are strong reasons to throw away variables, don't do it! Use the full model and use the p-values to guide interpretation rather than throwing away information. An insignificant p-value doesn't mean that a variable should be removed, it only means that the data don't give you clear evidence that the coefficient is nonzero.

The Lasso mitigates some of the issues of doing variable selection by p-values, but as long as you don't feel the need to remove information, there's no need to do it, neither by Lasso.

Good reasons for selecting variables are:

1. The number of observations is critically low for the number of variables you have.

2. There are strong dependencies between your variables ($X^TX$ is close to singular) and certain information is represented by several variables

3. The model is used to predict future observations and you are happy to get rid of some variables because it may be costly to observe them in the future.

4. Using cross-validation and the like you find that a model with fewer variables predicts better (if this happens, mostly one of 1 and 2 is also in place, but there are some further situations in which the original set of variables contains a lot of noise).

Variable selection is not required for interpretation! Note that even if there is no evidence (high p-value) that a certain variable has a nonzero coefficient, this variable may still improve the predictive power of the model. Removing the variable will set its coefficient to zero - don't forget that the estimator for it's coefficient in the full model is the "best guess" of the coefficient value that you have, significant or not, and therefore also better than zero in the sense of least squares.

By the way, regarding Lasso vs. variable selection by p-values (backward/forward/stepwise selection - never throw away all variables with large p-values in one go anyway): There are well known issues with variable selection by p-values and Lasso is often better, but not always. I've seen a good number of examples in which a model selected by backward or forward selection did better predicting on independent data than the Lasso. If you have enough observations to properly assess prediction quality using cross-validation and the like, you can compare different approaches and pick the best rather than always using Lasso.

Answer 2

My goal is to understand which variables out of 10 actually explain my dependent variable for a voter/election research.

This is not variable selection!

With variable selection the goal is to reduce the size of a model

• Such that it is easier to work with, like less computation intensive or requires less future efforts in sampling.
• Such that it is less sensitive to noise. See 'bias variance' tradeoff.

Your goal is to understand a relationship. This can be done by making a plot and observe a straight line (or in the case of more variables the line is a plane and you look at statistics in a table instead of a figure because plotting 10-D is not easy in our 3-D physical world).

The role of the p-values is to filter the effects from the noise (like you would do in the 2-D, plot to tell if there is a straight line relationship, or just noise). The p-values indicate statistical significance. That means that you have observed an effect that is statistically measurable with your experiment. The other effects, with high p-value, might be present but are overshadowed by noise.

So in your case it seems fine to go with p-values. However, be aware of the pitfalls with p-values.

Answer 3

If its only the Feature Importance Aspect you are looking for , you can use RFE Method (Recursive Feature Elimination) as well.

But Lasso Regression takes care of Model not reaching Overfit or Underfit Situations as well as Feature Importance, by reducing all the Useless Features to Zero.

You can use Lasso Regression using GridSearchCV method, and then plot your Mean Train and Mean Test Errors accordingly, if you have different values of Regularization Parameters available. And for the best values of Errors, you can consider the value for the Regularization Parameter and fit your model with Train Data.

Answer 4

For any others that might be interested in the answer: I managed to do the LASSO with an excel plugin into excel 365. It did show significantly different results in comparison to stepwise. Also, I found out that I actually want to build a causal model not a variable selection. (different terminology). So yes LASSO seems to be worth the work.