11
$\begingroup$

I'm currently working on building a predictive model for a binary outcome on a dataset with ~300 variables and 800 observations. I've read much on this site about the problems associated with stepwise regression and why not to use it.

I've been reading into LASSO regression and its ability for feature selection and have been successful in implementing it with the use of the "caret" package and "glmnet".

I am able to extract the coefficient of the model with the optimal lambda and alpha from "caret"; however, I'm unfamiliar with how to interpret the coefficients.

  • Are the LASSO coefficients interpreted in the same method as logistic regression?
  • Would it be appropriate to use the features selected from LASSO in logistic regression?

EDIT

Interpretation of the coefficients, as in the exponentiated coefficients from the LASSO regression as the log odds for a 1 unit change in the coefficient while holding all other coefficients constant.

https://stats.idre.ucla.edu/other/mult-pkg/faq/general/faq-how-do-i-interpret-odds-ratios-in-logistic-regression/

$\endgroup$
  • $\begingroup$ Can you fill in a little what you mean by "interpreted in the same way as logistic regression"? I'd be very useful to know exactly what interpretations you'd like to generalize. $\endgroup$ – Matthew Drury Jul 27 '16 at 19:04
  • 1
    $\begingroup$ @Matthew Drury - Thank you so much for taking the time in assisting me, as my coursework never gone over LASSO. In general, from what I was taught during my graduate courses, the exponentiated coefficients from a logistic regression yields the log odds of a 1 unit increase in the coefficient while holding all the other coefficients constant. $\endgroup$ – Michael Luu Jul 27 '16 at 19:16
  • 1
    $\begingroup$ In "caret" you select $\alpha$ and $\lambda$. Where does $\alpha$ come from? Is it probably a hyperparameter of an elastic net (the relative weight of LASSO versus ridge penalty) (in which case you would actually be using elastic net rather than LASSO)? $\endgroup$ – Richard Hardy Jul 27 '16 at 19:33
  • $\begingroup$ As far as I can tell, significance testing for coefficients has not been introduced in most LASSO implementations. So could a difference not be that while we can determine statistically significant variables in OLS, we cannot do so with LASSO except making a weaker statement that the LASSO coefficients of corresponding variables selected are the "important" variables to consider? $\endgroup$ – godspeed Jul 27 '16 at 20:30
12
$\begingroup$

Are the LASSO coefficients interpreted in the same method as logistic regression?

Let me rephrase: Are the LASSO coefficients interpreted in the same way as, for example, OLS maximum likelihood coefficients in a logistic regression?

LASSO (a penalized estimation method) aims at estimating the same quantities (model coefficients) as, say, OLS maximum likelihood (an unpenalized method). The model is the same, and the interpretation remains the same. The numerical values from LASSO will normally differ from those from OLS maximum likelihood: some will be closer to zero, others will be exactly zero. If a sensible amount of penalization has been applied, the LASSO estimates will lie closer to the true values than the OLS maximum likelihood estimates, which is a desirable result.

Would it be appropriate to use the features selected from LASSO in logistic regression?

There is no inherent problem with that, but you could use LASSO not only for feature selection but also for coefficient estimation. As I mention above, LASSO estimates may be more accurate than, say, OLS maximum likelihood estimates.

$\endgroup$
  • $\begingroup$ Thank you so much for this response! Makes alot of sense! Please excuse my limited knowledge in this matter. As you have mentioned in another comment that I may be using elastic net rather than LASSO via caret as it chooses the optimal lambda and alpha. Would the same apply in regards to the coefficients? $\endgroup$ – Michael Luu Jul 27 '16 at 20:10
  • $\begingroup$ Yes, it would. The basic logic remains the same. $\endgroup$ – Richard Hardy Jul 28 '16 at 5:42
  • $\begingroup$ You write "interpretation remains the same". Could you help me understand this point? It seems to be me that the interpretation of OLS coefficients in a multiple regression setting relies on partial regression plots. However, this property does not hold for lasso coefficients, leading me to believe the interpretation would be different. $\endgroup$ – user795305 Oct 22 '17 at 13:37
  • 1
    $\begingroup$ @Ben, If we assume an underlying statistical model, we can estimate its parameters in different ways, two popular ones being OLS and lasso. The estimated coefficients target the same targets, and both have some estimation error (which, if squared, can be decomposed into bias and variance), so in this sense their interpretation is the same. Now of course the methods are not the same, so you get different estimated coefficient values. If you care about the methods and their algebraic and geometrical interpretations, than these are not the same. But subject-matter interpretations are the same. $\endgroup$ – Richard Hardy Oct 22 '17 at 18:17
  • $\begingroup$ @RichardHardy Ah, okay, I think I better understand what you're saying. It's certainly true that lasso may beat OLS in estimation error, but, at the end of the day, like you say, these are just estimators for the same target. Would any estimator be interpreted in the same way that OLS is interpreted? For instance, would the (nonrandom) estimator $(1, \dots, p)^T$ be interpreted that way? or the estimator with iid uniform(0,1) entries? (etc) It does seem (to me) that properties of the estimator need to be directly used in its interpretation, and even subject-matter interpretations would change. $\endgroup$ – user795305 Oct 22 '17 at 18:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.