1/2 on lagrangian equation from lasso [duplicate]

Question

I´ve read this fantastic book The elements of statistical learning and I have a question about the lasso equation for the Lasso problem in its Lagrangian form:

$\hat{\beta}_{lasso} = argmin \{ \frac{1}{2} \sum_{i=1}^{N}(y_i -\beta_0 -\sum_{j=1}^{p} x_{ij}\beta_{j})^2 + \lambda \sum_{j=1}^{p} |\beta_j| \}$

I don´t know why $\frac{1}{2}$ is necessary for lasso, however for ridge it doesn't.

$\hat{\beta}_{ridge} = argmin \{\sum_{i=1}^{N}(y_i -\beta_0 -\sum_{j=1}^{p} x_{ij}\beta_{j})^2 + \lambda \sum_{j=1}^{p} \beta_j^2 \}$

References

• Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1, pp. 241-249). New York: Springer series in statistics.

Answer 1

There is nothing "necessary" about the factor of $\frac{1}{2}$. It is often used, as a matter of convenience, for quadratic objectives of the form $\frac{1}{2}x^TQx + g^Tx$ so that the matrix $Q$ winds up being the Hessian of the objective function.

In this case, the authors were not consistent between these two problems. The factor of $\frac{1}{2}$ can be absorbed in (adjustment made to) $\lambda$ and result in an equivalent problem, i.e., having the same argmin (although not the same optimal objective value).

Answer 2

The factor $\frac{1}{2}$ is quite obviously of no practical importance and is just a rescaling. To see this just multiply the objective function with $2$, then lasso solves obviously also the equivalent problem $$\beta_{lasso} \in \arg\min\{\sum_{i=1}^n (y_i-\beta_0 - \sum_{j=1}^p x_{ij}\beta_j)^2 + \lambda^* \sum_{j=1}^p |\beta_j|\}$$ where $\lambda^* = 2 \lambda \geq 0$. Since lasso is a convex optimization problem, the solutions to the problems will be identical, moreover there is a one-to-one relationship between $\lambda^*$ and $\lambda$. finally, both equivalent minimization problems translate to the same constrained minimization problem (just with different $\lambda$'s): $$\min_{\beta}\sum_{i=1}^n (y_i-\beta_0 - \sum_{j=1}^p x_{ij}\beta_j)^2 \qquad s.t. \qquad \sum_{j=1}^p |\beta_j| \leq t.$$

The factor $\frac{1}{2}$ ist just introduced for convenience, i.e. to simplify the writing within the theoretical analysis of the lasso. for example, the KKT conditions are then nicely "scaled", otherwise you would carry the factor $2$ from the derivative of the quadratic sum with you during your whole analysis.