1
$\begingroup$

I read this post Why use Lasso estimates over OLS estimates on the Lasso-identified subset of variables? . It says the LASSO shrinkage causes the estimates of the non-zero coefficients to be biased towards zero. So using OLS after LASSO selection is a recommended method (two-stage LASSO).

Is there any similar rule applying to elastic net? I understand that EN is the combination of ridge + LASSO. Say if I select the number of variables > the number of samples, because some of them are correlated. Then if I apply ridge regression on the selected variables, ridge will also bias the coefficients towards zero, but OLS is not applicable in this case.

So how to resolve this problem?

|cite|improve this question
$\endgroup$
  • $\begingroup$ See this thread: stats.stackexchange.com/questions/326427 -- some parts of my answer there are relevant. I used ridge after elastic net (ridge+lasso) as an attempt to make something like "relaxed elastic net". A commenter here stats.stackexchange.com/questions/386408/… mentioned that this paper arxiv.org/abs/0809.1777 also used a similar procedure. $\endgroup$ – amoeba says Reinstate Monica Jan 22 '19 at 8:08
  • $\begingroup$ @amoeba Thx! I did read your answers to those questions. But my concern is, in the de-biasing step, if we you Ridge, there will still be bias caused by the L2 regularization, so it seems that the de-biasing is not fully working (only L1 part is removed). I have searched a lot but didn't find anything related to that. Do you have any comments on this? Why we concerned about L1 but not L2 biases. $\endgroup$ – Vickyyy Jan 22 '19 at 16:08
  • $\begingroup$ IMHO bias (shrinkage) is good, not bad: e.g. ridge regression (I mean ridge itself, without any elastic net) introduces bias but this improves generalization performance. $\endgroup$ – amoeba says Reinstate Monica Jan 22 '19 at 17:32
  • $\begingroup$ @amoeba I guess my concern is RR actually penalizes all the directions with the same penalty magnitude (the prior believes that the model parameters are of the same size), but if the underlying system actually uses some key principal components (PCs) to adjust the output, RR in this case will biases the results in an incorrect way (PCR is more reasonable/or maybe PLS in this case). But I didn't find reference regarding this point. So does this mean that the "incorrect" biases of RR is not a concern? $\endgroup$ – Vickyyy Jan 22 '19 at 18:28
  • $\begingroup$ No, this is incorrect: RR biases towards high-variance PCs, very similarly to PCR actually. See here: stats.stackexchange.com/questions/81395 $\endgroup$ – amoeba says Reinstate Monica Jan 22 '19 at 19:29
0
$\begingroup$

The adaptive LASSO and the Elastic Net (EN) improve the LASSO in different ways. The adaptive LASSO controls the bias by shrinking larger parameters less, while the EN handles collinearity by incorporating the ridge penalty. The EN can be also extended to shrink parameters by different amounts - the adaptive EN is a combination of the adaptive LASSO and the EN and enjoys the good properties of both methods. Zou & Zhang (2009) suggest fitting the EN in the first stage and using those coefficients to calculate the weights for the adaptive EN.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.