I've been reading the manual about GLM, specifically the one on the elastic net in this link, and they come up with the following function to minimize: $$\frac{1}{2n}||Xw-y||^2_{2}+\alpha \rho ||w||_1+\frac{\alpha(1-\rho)}{2}||w||_2^2,\ \ \ \ \ \ \ \ \ \ \ (1)$$ where $n$ is the number of samples and $w$ are the coefficients of the regression. However, obviating the division by $2n$ (which I find unnecessary), I'm confused by the second part. In Zou & Hastie (2005) the RSS (Residual Sum of Squares) of the elastic net is defined as: $$RSS=||Xw-y||^2_{2}+\lambda \left(\alpha ||w||_1+(1-\alpha)||w||_2^2\right),\ \ \ (2)$$ and I really don't see the similarity between criterions (1) and (2) (like, the relations between the $\rho$ in eq. (1) and $\alpha$ of eq. (2), and the relation between $\alpha$ in eq.(1) and $\lambda$ in eq. (2)): am I missing something here?
1 Answer
The term $\frac{1}{2n}$ is there for convenience to obtain a first and second derivative of $RSS$ that looks nicer.
Regarding the different formulations, I don't think it makes a big difference how you define the weights of the L1 and L2 penalty, and indeed different formulations exist. You can see that both are equal. Let's start with equation (1): $$ \alpha \rho ||w||_1 + \frac{\alpha(1 - \rho)}{2} ||w||_2^2 = \alpha \rho ||w||_1 + \frac{\alpha - \alpha \rho}{2} ||w||_2^2 $$
And equation (2) becomes: $$ \lambda ( \alpha ||w||_1 + (1 - \alpha)||w||_2^2) = \lambda \alpha ||w||_1 + \lambda (1 - \alpha)||w||_2^2 = \lambda \alpha ||w||_1 + (\lambda - \lambda \alpha)||w||_2^2 $$
In essence, $\alpha$ of (1) is the same as $\lambda$ in (2) and $\rho$ of (1) is the same as $\alpha$ in (2). The $\frac{1}{2}$ part of (1) was added to obtain a nicer looking derivative, because $$ \frac{d}{dw_j} \left( \frac{\alpha(1 - \rho)}{2} ||w||_2^2 \right) = \alpha(1 - \rho) w_j . $$ In fact they also use it in another paper by Friedman, J., Hastie, T., & Tibshirani, R. (2010). Regularization Paths for Generalized Linear Models via Coordinate Descent. Journal of Statistical Software, 33(1), 1-22.
-
$\begingroup$ Sorry for insisting in the point, but the $1/2$ still bothers me. Suppose I have values for $\alpha$ and $\lambda$ in eq. (2), and a set of coefficients $\beta$: how would I obtain from them $\alpha$, $\rho$ and the coefficients on eq. (1)? Are the coefficients the same? $\endgroup$– NéstorCommented Jul 25, 2012 at 9:24
-
$\begingroup$ The parameters $\alpha, \rho, \lambda$ for both formulations would differ. What I wanted to point out is that it does not matter much, because they are constant and you end up with one weight for $||w||_1$ and one for $||w|_2^2$, therefore I prefer to formulate the penalty as $\lambda_1 ||w||_1 + \lambda_2 ||w||_2^2$. The principal of finding a solution to $\arg \min_w RSS$ stays the same for all formulations. $\endgroup$– sebpCommented Jul 25, 2012 at 10:25