Challenges in interpretation of variable selection from LASSO and OLS [duplicate]

I work as a consultant and I am often faced with variable selection and prediction problems.

For my clients, I run OLS and I am recently pushing for penalized methods which can handle variable selection better than OLS.

I am familiar with how the LASSO works It aims to shrink coefficients to zero and can identify the non - zero coefficients. By looking at the solution paths, I can understand how the variable selection occurs.

However, I work with non-statisticians. They are quite comfortable with OLS and I have also observed that explaining OLS to my clients is often simpler. My question is this:

I know this is not the way to do variable selection. Lets say my OLS outputs the following model. I have scaled my response and predictors.

$Y = 0.25~X_{1} + 0.95~X_{2} + 0.65~X_{3} +2 ~ X_{4} + 0.1 ~X_{5}$

My non-statistics friends say that we should select $X_{2}$ and $X_{4}$ in the model. They say this by observing two aspects:

1. Their coefficient values are high compared to the other variables
2. The model output also shows that these 2 coefficients are significant (by significant I mean p-value < 0.05).

In conclusion, they say the reduced model $Y = 0.95~X_{2} + 2~X_{4}$ is sufficient.

I don't know how to explain that this is not the right way to go about variable selection. In fact at times they also overlook the fact that $X_{1}$ is a significant variable. But they think it is not important because its coefficient value is not high in magnitude.

What I am unable to explain and I admit that at times I myself don't understand

1. I know how to interpret coefficients. For instance keeping all other variables constant, a unit change in $X_{4}$ will cause a 2-fold change in the response. But how can one conclude that $X_{4}$ is IMPORTANT basis its magnitude and the fact that its p-value is < 0.05?

Could someone help with this? If indeed $X_{2}$ and $X_{4}$ are IMPORTANT should I say that we should be comparing two models and do some kind of a lack of fit procedure to ascertain this claim.

1. $Y = \beta_{1}~X_{1} + \beta_{2}~X_{2} + \beta_{3}~X_{3} +\beta_{4}~X_{4}$
2. $Y = \beta_{2}~X_{2} + \beta_{4}~X_{4}$

Finally, if my understanding don't shrinkage methods like LASSO and ridge do variable selection keeping some end goal in mind like minimizing the mean absolute prediction error. OLS only minimizes the sum of squared error but how can one simply select variables in a model simply because their coefficients are high in magnitude.

If one only looks at coefficient values to select variables to enter the model, is one not ignoring the correlation structure of the predictor variables.

marked as duplicate by Greenparker, John, gung♦ regression StackExchange.ready(function() { if (StackExchange.options.isMobile) return; $('.dupe-hammer-message-hover:not(.hover-bound)').each(function() { var$hover = $(this).addClass('hover-bound'),$msg = $hover.siblings('.dupe-hammer-message');$hover.hover( function() { $hover.showInfoMessage('', { messageElement:$msg.clone().show(), transient: false, position: { my: 'bottom left', at: 'top center', offsetTop: -7 }, dismissable: false, relativeToBody: true }); }, function() { StackExchange.helpers.removeMessages(); } ); }); }); Jun 19 '16 at 23:50

• If magnitude of the coefficient indicates a variable's importance, then merely changing the unit of measurement will make one more important. E.g. models $\text{Weight} = \text{Height (in cm)} + \text{Other Factors}$ v.s. $\text{Weight} = \text{Height (in m)} + \text{Other Factors}$, coefficient of height in the latter model will be 100 times of that in the former, does that make it more important? – Francis Jun 19 '16 at 13:52
• @Francis No because they are standardized beforehand. – WAF Feb 27 '18 at 11:08