3
$\begingroup$

I've been reading the manual about GLM, specifically the one on the elastic net in this link, and they come up with the following function to minimize: $$\frac{1}{2n}||Xw-y||^2_{2}+\alpha \rho ||w||_1+\frac{\alpha(1-\rho)}{2}||w||_2^2,\ \ \ \ \ \ \ \ \ \ \ (1)$$ where $n$ is the number of samples and $w$ are the coefficients of the regression. However, obviating the division by $2n$ (which I find unnecessary), I'm confused by the second part. In Zou & Hastie (2005) the RSS (Residual Sum of Squares) of the elastic net is defined as: $$RSS=||Xw-y||^2_{2}+\lambda \left(\alpha ||w||_1+(1-\alpha)||w||_2^2\right),\ \ \ (2)$$ and I really don't see the similarity between criterions (1) and (2) (like, the relations between the $\rho$ in eq. (1) and $\alpha$ of eq. (2), and the relation between $\alpha$ in eq.(1) and $\lambda$ in eq. (2)): am I missing something here?

$\endgroup$
3
$\begingroup$

The term $\frac{1}{2n}$ is there for convenience to obtain a first and second derivative of $RSS$ that looks nicer.

Regarding the different formulations, I don't think it makes a big difference how you define the weights of the L1 and L2 penalty, and indeed different formulations exist. You can see that both are equal. Let's start with equation (1): $$ \alpha \rho ||w||_1 + \frac{\alpha(1 - \rho)}{2} ||w||_2^2 = \alpha \rho ||w||_1 + \frac{\alpha - \alpha \rho}{2} ||w||_2^2 $$

And equation (2) becomes: $$ \lambda ( \alpha ||w||_1 + (1 - \alpha)||w||_2^2) = \lambda \alpha ||w||_1 + \lambda (1 - \alpha)||w||_2^2 = \lambda \alpha ||w||_1 + (\lambda - \lambda \alpha)||w||_2^2 $$

In essence, $\alpha$ of (1) is the same as $\lambda$ in (2) and $\rho$ of (1) is the same as $\alpha$ in (2). The $\frac{1}{2}$ part of (1) was added to obtain a nicer looking derivative, because $$ \frac{d}{dw_j} \left( \frac{\alpha(1 - \rho)}{2} ||w||_2^2 \right) = \alpha(1 - \rho) w_j . $$ In fact they also use it in another paper by Friedman, J., Hastie, T., & Tibshirani, R. (2010). Regularization Paths for Generalized Linear Models via Coordinate Descent. Journal of Statistical Software, 33(1), 1-22.

$\endgroup$
2
  • $\begingroup$ Sorry for insisting in the point, but the $1/2$ still bothers me. Suppose I have values for $\alpha$ and $\lambda$ in eq. (2), and a set of coefficients $\beta$: how would I obtain from them $\alpha$, $\rho$ and the coefficients on eq. (1)? Are the coefficients the same? $\endgroup$ – Néstor Jul 25 '12 at 9:24
  • $\begingroup$ The parameters $\alpha, \rho, \lambda$ for both formulations would differ. What I wanted to point out is that it does not matter much, because they are constant and you end up with one weight for $||w||_1$ and one for $||w|_2^2$, therefore I prefer to formulate the penalty as $\lambda_1 ||w||_1 + \lambda_2 ||w||_2^2$. The principal of finding a solution to $\arg \min_w RSS$ stays the same for all formulations. $\endgroup$ – sebp Jul 25 '12 at 10:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.