Challenges in interpretation of variable selection from LASSO and OLS [duplicate]

Question

I work as a consultant and I am often faced with variable selection and prediction problems.

For my clients, I run OLS and I am recently pushing for penalized methods which can handle variable selection better than OLS.

I am familiar with how the LASSO works It aims to shrink coefficients to zero and can identify the non - zero coefficients. By looking at the solution paths, I can understand how the variable selection occurs.

However, I work with non-statisticians. They are quite comfortable with OLS and I have also observed that explaining OLS to my clients is often simpler. My question is this:

I know this is not the way to do variable selection. Lets say my OLS outputs the following model. I have scaled my response and predictors.

$Y = 0.25~X_{1} + 0.95~X_{2} + 0.65~X_{3} +2 ~ X_{4} + 0.1 ~X_{5} $

My non-statistics friends say that we should select $X_{2}$ and $X_{4}$ in the model. They say this by observing two aspects:

1. Their coefficient values are high compared to the other variables
2. The model output also shows that these 2 coefficients are significant (by significant I mean p-value < 0.05).

In conclusion, they say the reduced model $Y = 0.95~X_{2} + 2~X_{4}$ is sufficient.

I don't know how to explain that this is not the right way to go about variable selection. In fact at times they also overlook the fact that $X_{1}$ is a significant variable. But they think it is not important because its coefficient value is not high in magnitude.

What I am unable to explain and I admit that at times I myself don't understand

1. I know how to interpret coefficients. For instance keeping all other variables constant, a unit change in $X_{4}$ will cause a 2-fold change in the response. But how can one conclude that $X_{4}$ is IMPORTANT basis its magnitude and the fact that its p-value is < 0.05?

Could someone help with this? If indeed $X_{2}$ and $X_{4}$ are IMPORTANT should I say that we should be comparing two models and do some kind of a lack of fit procedure to ascertain this claim.

1. $Y = \beta_{1}~X_{1} + \beta_{2}~X_{2} + \beta_{3}~X_{3} +\beta_{4}~X_{4} $
2. $Y = \beta_{2}~X_{2} + \beta_{4}~X_{4} $

Finally, if my understanding don't shrinkage methods like LASSO and ridge do variable selection keeping some end goal in mind like minimizing the mean absolute prediction error. OLS only minimizes the sum of squared error but how can one simply select variables in a model simply because their coefficients are high in magnitude.

If one only looks at coefficient values to select variables to enter the model, is one not ignoring the correlation structure of the predictor variables.

Answer 1

The next step in working with your clients or other non-statisticians would be to help them understand the assumptions and limitations of OLS, particularly in application to real-world data with correlated predictors.

One issue is omitted-variable bias. @Gung's answer on this Cross Validated page makes the point nicely. If, in a model, you add (or subtract) a variable that is correlated in the data sample with outcome and with other predictors, regression coefficients of other variables will change. It's important to try, intelligently, to keep all relevant predictors in a model. In fairness to your non-statistician colleagues, the leap to focus on high-magnitude coefficients (for standardized predictors) might make sense if there were no correlations among predictors. When was the last time you saw such a case in practice?

A second issue is that attempts to select variables automatically (which is what you do by picking the variables with the highest coefficients) end up with severe problems, as discussed extensively on this Cross Validated page. In particular, you may be finding the variables whose relation to outcome is strong in the present data sample but are unlikely to do so well in subsequent samples. @Gung again has a very helpful answer on that page that explains this problem of data-dredging much better than I can.

Third, if your clients want to use the model for prediction, then it's pretty clear that performance will be improved if you include as many variables as possible, even those that don't meet some criterion of statistical "significance." This Cross Validated page is one of several that cover this issue. You of course must then take care to avoid overfitting. I tend to agree with the answer there from @DikranMarsupial, who recommends using ridge regression to avoid overfitting. If you repeat LASSO on multiple bootstrap samples from the same data set you are likely to find many different sets of selected variables. That result might be harder to explain to your non-statistician colleagues than just saying you are down-weighting some variables to avoid overfitting, as with ridge regression.

With respect to your understanding of LASSO and ridge regression, these actually do minimize mean square error, not mean absolute prediction error, but subject to constraints on the magnitudes of the coefficients. The constraint on the coefficient magnitudes is why it's important to start with standardized predictors, so that differences of variable scales don't influence results. The difference is in the constraints on the coefficients: it's the sum of absolute magnitudes in LASSO, and the sum of their squares in ridge. LASSO thus selects a subset (and penalizes their coefficients to lower in magnitude than you would get in the corresponding OLS model), while ridge differentially weights all the predictors, tending to treat correlated predictors together. An Introduction to Statistical Learning is a good place for further details.