1
$\begingroup$

I read this post Why use Lasso estimates over OLS estimates on the Lasso-identified subset of variables? . It says the LASSO shrinkage causes the estimates of the non-zero coefficients to be biased towards zero. So using OLS after LASSO selection is a recommended method (two-stage LASSO).

Is there any similar rule applying to elastic net? I understand that EN is the combination of ridge + LASSO. Say if I select the number of variables > the number of samples, because some of them are correlated. Then if I apply ridge regression on the selected variables, ridge will also bias the coefficients towards zero, but OLS is not applicable in this case.

So how to resolve this problem?

Improve this question
$\endgroup$
5
  • $\begingroup$ See this thread: stats.stackexchange.com/questions/326427 -- some parts of my answer there are relevant. I used ridge after elastic net (ridge+lasso) as an attempt to make something like "relaxed elastic net". A commenter here stats.stackexchange.com/questions/386408/… mentioned that this paper arxiv.org/abs/0809.1777 also used a similar procedure. $\endgroup$ – amoeba Jan 22 '19 at 8:08
  • $\begingroup$ @amoeba Thx! I did read your answers to those questions. But my concern is, in the de-biasing step, if we you Ridge, there will still be bias caused by the L2 regularization, so it seems that the de-biasing is not fully working (only L1 part is removed). I have searched a lot but didn't find anything related to that. Do you have any comments on this? Why we concerned about L1 but not L2 biases. $\endgroup$ – Vickyyy Jan 22 '19 at 16:08
  • $\begingroup$ IMHO bias (shrinkage) is good, not bad: e.g. ridge regression (I mean ridge itself, without any elastic net) introduces bias but this improves generalization performance. $\endgroup$ – amoeba Jan 22 '19 at 17:32
  • $\begingroup$ @amoeba I guess my concern is RR actually penalizes all the directions with the same penalty magnitude (the prior believes that the model parameters are of the same size), but if the underlying system actually uses some key principal components (PCs) to adjust the output, RR in this case will biases the results in an incorrect way (PCR is more reasonable/or maybe PLS in this case). But I didn't find reference regarding this point. So does this mean that the "incorrect" biases of RR is not a concern? $\endgroup$ – Vickyyy Jan 22 '19 at 18:28
  • $\begingroup$ No, this is incorrect: RR biases towards high-variance PCs, very similarly to PCR actually. See here: stats.stackexchange.com/questions/81395 $\endgroup$ – amoeba Jan 22 '19 at 19:29
0
$\begingroup$

The adaptive LASSO and the Elastic Net (EN) improve the LASSO in different ways. The adaptive LASSO controls the bias by shrinking larger parameters less, while the EN handles collinearity by incorporating the ridge penalty. The EN can be also extended to shrink parameters by different amounts - the adaptive EN is a combination of the adaptive LASSO and the EN and enjoys the good properties of both methods. Zou & Zhang (2009) suggest fitting the EN in the first stage and using those coefficients to calculate the weights for the adaptive EN.

Improve this answer
$\endgroup$
0
$\begingroup$

The answer is it depends on what you try to achieve. Post-LASSO OLS estimator is well understood, see e.g. [1]. What has to be said is that you can never be sure that the selected covariates are 'true' covariates - LASSO, like any other estimator (e.g. adaptive LASSO) does mistakes. Oracle properties of adaptive LASSO are derived under very strong assumptions on the covariates - the so-called beta-min condition - which rarely, if at all, happens that the data is such in practice. That being said, the prediction should improve by using post-LASSO OLS or be at least as good as only using LASSO. This is because you reduce the shrinkage bias as you pointed out.

For inference, i.e. stating whether coefficients are significant or not, using post-LASSO OLS is not correct as the results are biased due to omitted variable(s) which LASSO kicks-out mistakenly (as already discussed).

The elastic net is much less understood from the theory point of view, but you may expect similar behavior of this estimator. Therefore, you may try post elastic net OLS and see if it improves your predictions. The same for adaptive versions, although from a theoretical point of view, they are not superior to standard LASSO/elastic net.

Hope this helps!

References:

[1] Belloni, A., & Chernozhukov, V. (2013). Least squares after model selection in high-dimensional sparse models. Bernoulli, 19 (2), 521-547. https://projecteuclid.org/download/pdfview_1/euclid.bj/1363192037

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.