# Interpretation of LASSO regression coefficients

I'm currently working on building a predictive model for a binary outcome on a dataset with ~300 variables and 800 observations. I've read much on this site about the problems associated with stepwise regression and why not to use it.

I've been reading into LASSO regression and its ability for feature selection and have been successful in implementing it with the use of the "caret" package and "glmnet".

I am able to extract the coefficient of the model with the optimal lambda and alpha from "caret"; however, I'm unfamiliar with how to interpret the coefficients.

• Are the LASSO coefficients interpreted in the same method as logistic regression?
• Would it be appropriate to use the features selected from LASSO in logistic regression?

EDIT

Interpretation of the coefficients, as in the exponentiated coefficients from the LASSO regression as the log odds for a 1 unit change in the coefficient while holding all other coefficients constant.

https://stats.idre.ucla.edu/other/mult-pkg/faq/general/faq-how-do-i-interpret-odds-ratios-in-logistic-regression/

• Can you fill in a little what you mean by "interpreted in the same way as logistic regression"? I'd be very useful to know exactly what interpretations you'd like to generalize. Commented Jul 27, 2016 at 19:04
• @Matthew Drury - Thank you so much for taking the time in assisting me, as my coursework never gone over LASSO. In general, from what I was taught during my graduate courses, the exponentiated coefficients from a logistic regression yields the log odds of a 1 unit increase in the coefficient while holding all the other coefficients constant. Commented Jul 27, 2016 at 19:16
• In "caret" you select $\alpha$ and $\lambda$. Where does $\alpha$ come from? Is it probably a hyperparameter of an elastic net (the relative weight of LASSO versus ridge penalty) (in which case you would actually be using elastic net rather than LASSO)? Commented Jul 27, 2016 at 19:33
• As far as I can tell, significance testing for coefficients has not been introduced in most LASSO implementations. So could a difference not be that while we can determine statistically significant variables in OLS, we cannot do so with LASSO except making a weaker statement that the LASSO coefficients of corresponding variables selected are the "important" variables to consider? Commented Jul 27, 2016 at 20:30

Are the LASSO coefficients interpreted in the same method as logistic regression?

Let me rephrase: Are the LASSO coefficients interpreted in the same way as, for example, OLS maximum likelihood coefficients in a logistic regression?

LASSO (a penalized estimation method) aims at estimating the same quantities (model coefficients) as, say, OLS maximum likelihood (an unpenalized method). The model is the same, and the interpretation remains the same. The numerical values from LASSO will normally differ from those from OLS maximum likelihood: some will be closer to zero, others will be exactly zero. If a sensible amount of penalization has been applied, the LASSO estimates will lie closer to the true values than the OLS maximum likelihood estimates, which is a desirable result.

Would it be appropriate to use the features selected from LASSO in logistic regression?

There is no inherent problem with that, but you could use LASSO not only for feature selection but also for coefficient estimation. As I mention above, LASSO estimates may be more accurate than, say, OLS maximum likelihood estimates.

• Thank you so much for this response! Makes alot of sense! Please excuse my limited knowledge in this matter. As you have mentioned in another comment that I may be using elastic net rather than LASSO via caret as it chooses the optimal lambda and alpha. Would the same apply in regards to the coefficients? Commented Jul 27, 2016 at 20:10
• Yes, it would. The basic logic remains the same. Commented Jul 28, 2016 at 5:42
• You write "interpretation remains the same". Could you help me understand this point? It seems to be me that the interpretation of OLS coefficients in a multiple regression setting relies on partial regression plots. However, this property does not hold for lasso coefficients, leading me to believe the interpretation would be different. Commented Oct 22, 2017 at 13:37
• @Ben, If we assume an underlying statistical model, we can estimate its parameters in different ways, two popular ones being OLS and lasso. The estimated coefficients target the same targets, and both have some estimation error (which, if squared, can be decomposed into bias and variance), so in this sense their interpretation is the same. Now of course the methods are not the same, so you get different estimated coefficient values. If you care about the methods and their algebraic and geometrical interpretations, than these are not the same. But subject-matter interpretations are the same. Commented Oct 22, 2017 at 18:17
• @Ben, I think we can separate the theoretical properties of the estimator from the subject-matter interpretation of the estimates as long as the estimands (the targets) are the same. Of course, the point estimates are not the end of the story; standard errors and confidence intervals play a role, too. But once we have point estimates and their standard errors / confidence intervals, we need no longer care how these estimates were obtained when interpreting them from the subject-matter perspective. Commented Oct 22, 2017 at 19:01