According to Zou et al. 2005, the elastic estimator has the follow formulation.

$$ \beta_{elastic} = \arg \min_{\beta} 1/2n\|Y - X\beta\|_2^2 + \alpha \lambda\|z\|_1 + (1-\alpha)\lambda/2 \|z\|_2^2$$

I understand that the elastic net takes the advantage of both the ridge and lasso regression in the penalty term. But I don't understand why the $l_1$-norm and $l_2$-squared norm should interacts linearly via $\alpha$. Does it make more sense to use: $$ \beta_{new} = \arg \min_{\beta} 1/2n\|Y - X\beta\|_2^2 + \alpha \lambda\|z\|_1 + (1-\alpha)\lambda \|z\|_2$$

p.s. I understand that adding the coefficient $\frac{1}{2}$ is a equivalent formulation, but I don't understand why we combine a norm and a squared-norm for the elastic net penalty.

For ridge regression, the following two formulations are equivalent via their constrained formulations, or KKT condition:

$$ \beta_{ridge1} = \arg \min_{\beta} 1/2n\|Y-X\beta\|_2^2 + \lambda\|\beta\|_2^2 $$

$$ \beta_{ridge2} = \arg \min_{\beta} 1/2n\|Y-X\beta\|_2^2 + \lambda\|\beta\|_2 $$

My question is that why we use $\beta_{ridge1}$ instead of $\beta_{ridge2}$ for the elastic net penalty?

  • $\begingroup$ The penalty term in the second formulation is a sum of the two norm, instead of the sum of a norm and a squared norm. And the links do not fully address my question. $\endgroup$ – Yang Guo Jul 9 '17 at 20:34
  • $\begingroup$ @gung: $\beta_{ridge1}$ uses squared 2-norm for the penalty term, while $\beta_{ridge2}$ uses 2-norm for the penalty term. $\endgroup$ – Yang Guo Jul 11 '17 at 23:42
  • 1
    $\begingroup$ @Ben: Thank you very much. That link answers my question about the use of the squared-norm and the norm in the elastic net penalty term. Also, I need to point out that p-norm is always convex, and both problem are convex. See: math.stackexchange.com/questions/535825/… $\endgroup$ – Yang Guo Jul 11 '17 at 23:47
  • $\begingroup$ Ah, of course, my mistake! Thank you! It follows immediately from the triangle inequality that $\beta \mapsto \|\beta\|_2$ is convex. (I was accidentally imagining the function $\beta \mapsto \sqrt{\|\beta\|_2}$ in my comment.) $\endgroup$ – user795305 Jul 12 '17 at 0:01
  • $\begingroup$ You're right; I see it now. I don't know why I missed it before. $\endgroup$ – gung - Reinstate Monica Jul 12 '17 at 0:12

Thanks for @Ben 's comment.

This link shows the $\beta_{ridge1}$ and $\beta_{ridge2}$ are equivalent. The only difference for the proof that $\widehat{\beta}_{elastic} = \widehat{\beta}_{new}$to work for the elastic is to change both $\alpha$ and $\lambda$.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.