2
$\begingroup$

So, I understand why simple linear or logistic regression will have infinite solutions in this case (good answers here and here). But while LASSO will only select n features, Elastic net does not have this limitation. This answer explains how regularization limits the potential solutions to a problem so that building a model can be possible. Is the same concept true of Elastic Net? If regularization limits the possible solutions, then how is the "final" solution chosen from that space?

$\endgroup$
5
  • 2
    $\begingroup$ Have a look at web.stanford.edu/~hastie/TALKS/enet_talk.pdf $\endgroup$ Commented May 21, 2019 at 19:33
  • $\begingroup$ @GabrielRomon I looked through this but perhaps the section I need went over my head... where in here do they address the problem of selecting more predictors than observations? $\endgroup$ Commented Jun 11, 2019 at 15:36
  • $\begingroup$ A lot is said on page 9. For the details you definitely want to check the original paper. If something in the paper is unclear, don't hesitate to ask. $\endgroup$ Commented Jun 11, 2019 at 16:17
  • $\begingroup$ With penalization the effective p is much lower than the apparent p. $\endgroup$ Commented Sep 30, 2023 at 15:44
  • $\begingroup$ Near duplicate: stats.stackexchange.com/questions/274225/…. $\endgroup$
    – whuber
    Commented Sep 30, 2023 at 16:44

1 Answer 1

2
$\begingroup$

One way to look at this is that (as long as $\lambda_2\neq 0$) the L2 penalty is equivalent to adding $p$ examples: $$ \Vert y - X \beta \Vert^2 + \lambda_2 \Vert \beta \Vert^2 = \Vert \tilde y - \tilde X \beta \Vert^2 $$ with $$ \tilde X = \begin{bmatrix}X\\ \sqrt{\lambda_2} I_{p\times p}\end{bmatrix} \quad \tilde y = \begin{bmatrix}y\\ 0_{p\times 1}\end{bmatrix}. $$ So, in general, $n\times p$ ridge regression is equivalent to $(n+p)\times p$ non-regularised regression. Similarly $n\times p$ elastic-net regression is equivalent to $(n+p)\times p$ Lasso regression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.