2
$\begingroup$

According to Zou et al. 2005, the elastic estimator has the follow formulation.

$$ \beta_{elastic} = \arg \min_{\beta} 1/2n\|Y - X\beta\|_2^2 + \alpha \lambda\|z\|_1 + (1-\alpha)\lambda/2 \|z\|_2^2$$

I understand that the elastic net takes the advantage of both the ridge and lasso regression in the penalty term. But I don't understand why the $l_1$-norm and $l_2$-squared norm should interacts linearly via $\alpha$. Does it make more sense to use: $$ \beta_{new} = \arg \min_{\beta} 1/2n\|Y - X\beta\|_2^2 + \alpha \lambda\|z\|_1 + (1-\alpha)\lambda \|z\|_2$$

p.s. I understand that adding the coefficient $\frac{1}{2}$ is a equivalent formulation, but I don't understand why we combine a norm and a squared-norm for the elastic net penalty.

For ridge regression, the following two formulations are equivalent via their constrained formulations, or KKT condition:

$$ \beta_{ridge1} = \arg \min_{\beta} 1/2n\|Y-X\beta\|_2^2 + \lambda\|\beta\|_2^2 $$

$$ \beta_{ridge2} = \arg \min_{\beta} 1/2n\|Y-X\beta\|_2^2 + \lambda\|\beta\|_2 $$

My question is that why we use $\beta_{ridge1}$ instead of $\beta_{ridge2}$ for the elastic net penalty?

Improve this question
$\endgroup$
5
  • $\begingroup$ The penalty term in the second formulation is a sum of the two norm, instead of the sum of a norm and a squared norm. And the links do not fully address my question. $\endgroup$
    – Yang Guo
    Commented Jul 9, 2017 at 20:34
  • $\begingroup$ @gung: $\beta_{ridge1}$ uses squared 2-norm for the penalty term, while $\beta_{ridge2}$ uses 2-norm for the penalty term. $\endgroup$
    – Yang Guo
    Commented Jul 11, 2017 at 23:42
  • 1
    $\begingroup$ @Ben: Thank you very much. That link answers my question about the use of the squared-norm and the norm in the elastic net penalty term. Also, I need to point out that p-norm is always convex, and both problem are convex. See: math.stackexchange.com/questions/535825/… $\endgroup$
    – Yang Guo
    Commented Jul 11, 2017 at 23:47
  • $\begingroup$ Ah, of course, my mistake! Thank you! It follows immediately from the triangle inequality that $\beta \mapsto \|\beta\|_2$ is convex. (I was accidentally imagining the function $\beta \mapsto \sqrt{\|\beta\|_2}$ in my comment.) $\endgroup$
    – user795305
    Commented Jul 12, 2017 at 0:01
  • $\begingroup$ You're right; I see it now. I don't know why I missed it before. $\endgroup$
    – gung - Reinstate Monica
    Commented Jul 12, 2017 at 0:12

1 Answer 1

Reset to default
1
$\begingroup$

Thanks for @Ben 's comment.

This link shows the $\beta_{ridge1}$ and $\beta_{ridge2}$ are equivalent. The only difference for the proof that $\widehat{\beta}_{elastic} = \widehat{\beta}_{new}$to work for the elastic is to change both $\alpha$ and $\lambda$.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.