I was wondering about the ethics of using lasso regression for variable selection and then simply entering the selected variables into a standard regression.

Is it kosher to do this?

  • 1
    $\begingroup$ probably not because Lasso has an additional L1 constraint compared to the "standard" regression. Both will produce different estimates $\endgroup$ – user3119750 Oct 19 '17 at 19:30
  • $\begingroup$ Yes I suspected as much but was interested in why. I just wondered whether the shrinkage towards 0 in a lasso, in its own way, produces another form of bias. I guess I'm interested in which estimates are more 'accurate' (acknowledging the difficulty with notions like accuracy in statistics) $\endgroup$ – llewmills Oct 19 '17 at 19:42
  • $\begingroup$ What's your goal with the analysis? Why do the LASSO estimates not meet this goal? $\endgroup$ – Matthew Drury Oct 19 '17 at 20:07
  • $\begingroup$ @Matthew Drury my goal is to select the model, from a list of 17 predictors, that best predicts the outcome variable. My boss chose the variables, from a much larger number, that were the most theoretically sound; however 17 seems like too large a number to appear like anything other than fishing. if I am going to fish, I want to do it ethically (i.e. no stepwise regression). But my boss wants p-values and the glmnet package in R doesn't seem to supply any. I wondered if it was ok to use glmnet to supply me with variables, then lm() to run the analysis. I guessed not but wanted to check. $\endgroup$ – llewmills Oct 19 '17 at 21:39
  • $\begingroup$ Do you know what your boss intends to do with the p-values? If your goal is predictive power, p-values have nothing to say on that issue, so your boss's request is not inline with the business problem. Here are some simple examples to make that point: stats.stackexchange.com/questions/291210/… $\endgroup$ – Matthew Drury Oct 19 '17 at 23:52

This is not kosher, but if you do it anyway, I won't tell anyone.

The reason this is frowned upon is because you are performing model selection (that's the second S in LASSO), and in model selection you are reusing your data to figure out the best model. I'm hoping someone else can give you a better explanation mathematically, because I don't think I can. You are simply messing with the conditionality that got you to $\hat\beta^{LASSO}$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.