Lasso Regression as Variable Selection

Question

Suppose we are initially given $p$ predictor variables. In lasso regression, we want to find estimates of the coefficients $\beta_1, \dots, \beta_p$ that minimize $\text{RSS}+ \lambda \sum_{j=1}^{p} |\beta_j|$ where $\text{RSS}$ is the residual sum of squares. So if $\lambda = 0$ then we just have regular ordinary least squares regression. As we increase $\lambda$, we are essentially assuming more structure in the data and so increase the bias and reduce the variance compared to OLS. Basically we are assuming a prior distribution for the coefficients. Now some of the coefficients can be $0$. So this is also a variable selection procedure.

Question. Can one first do a lasso regression and then some sort of stepwise selection technique for variable reduction? Does the order of performing these techniques matter?

Answer 1

LASSO doesn't just select among a set of predictors and perform a bias-variance tradeoff. In doing so it also penalizes the coefficients of the selected predictors in a way that minimizes overfitting. If you simply use LASSO to choose predictors and then go on your way without accounting for your having already used the outcomes to choose those predictors, then you can be losing that protection from overfitting. See this page for further details.

In general, stepwise selection of predictors is likely to provide overfitting and poor generalization to other samples of data; there is much discussion on this page with links to further discussion.

So if you have already selected a set of predictors with LASSO, it's best to stick with them and with their penalized regression coefficients.

Answer 2

Well, you can do that, in the sense that it is possible. But why would you want to?

LASSO regression works reasonably well. Stepwise does not.

Answer 3

Why do you want to do Variable selection after/before/together with Regularization? Lasso also find with lambda the best number of features to use in order to simplify your model and avoiding the overfitting.

Answer 4

First, I would not double up on variable selection methods. This does not make much sense. I would instead use the best method given your data and model framework, and go with that.

Second, there are numerous reasons to be very cautious regarding using LASSO as your preferred variable selection method.

There is a simple reason why not using LASSO for variable selection. It just does not work as well as advertised. This is due to its fitting algorithm that includes a penalty factor that penalizes the model against higher regression coefficients. It seems like a good idea, as people think it always reduces model overfitting, and improves predictions (on new data). In reality it very often does the opposite ... increase model under-fitting and weakens prediction accuracy. You can see many examples of that by searching the Internet for Images and searching specifically for "LASSO MSE graph." Whenever such graphs show the lowest MSE at the beginning of the X-axis, it shows a LASSO that has failed (increase model under-fitting).

The above unintended consequences are due to the penalty algorithm. Because of it LASSO has no way of distinguishing between a strong causal variable with predictive information and an associated high regression coefficient and a weak variable with no explanatory or predictive information value that has a low regression coefficient. Often, LASSO will prefer the weak variable over the strong causal variable. Also, it may at times even cause to shift the directional signs of variables (shifting from one direction that makes sense to an opposite direction that does not). You can see many examples of that by searching the Internet for Images and searching specifically for "LASSO coefficient path".