7
$\begingroup$

This question already has an answer here:

I´ve read this fantastic book The elements of statistical learning and I have a question about the lasso equation for the Lasso problem in its Lagrangian form:

$\hat{\beta}_{lasso} = argmin \{ \frac{1}{2} \sum_{i=1}^{N}(y_i -\beta_0 -\sum_{j=1}^{p} x_{ij}\beta_{j})^2 + \lambda \sum_{j=1}^{p} |\beta_j| \}$

I don´t know why $\frac{1}{2}$ is necessary for lasso, however for ridge it doesn't.

$\hat{\beta}_{ridge} = argmin \{\sum_{i=1}^{N}(y_i -\beta_0 -\sum_{j=1}^{p} x_{ij}\beta_{j})^2 + \lambda \sum_{j=1}^{p} \beta_j^2 \}$

References

  • Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1, pp. 241-249). New York: Springer series in statistics.
$\endgroup$

marked as duplicate by gung, whuber Jun 26 '17 at 21:51

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

10
$\begingroup$

There is nothing "necessary" about the factor of $\frac{1}{2}$. It is often used, as a matter of convenience, for quadratic objectives of the form $\frac{1}{2}x^TQx + g^Tx$ so that the matrix $Q$ winds up being the Hessian of the objective function.

In this case, the authors were not consistent between these two problems. The factor of $\frac{1}{2}$ can be absorbed in (adjustment made to) $\lambda$ and result in an equivalent problem, i.e., having the same argmin (although not the same optimal objective value).

$\endgroup$
  • 5
    $\begingroup$ More broadly, the OP should be aware that are numerous ways to formulate what's essentially the same optimization problem. The equivalence of minimizing $f(x)$ or $\frac{1}{2} f(x)$ is a simple such case. Another example is the equivalence between maximizing the likelihood function and the logarithm of the likelihood function (since the logarithm is a strictly monotonic function). There are all kinds of situations where an equivalent optimization problem is more convenient. $\endgroup$ – Matthew Gunn Jun 26 '17 at 16:01
3
$\begingroup$

The factor $\frac{1}{2}$ is quite obviously of no practical importance and is just a rescaling. To see this just multiply the objective function with $2$, then lasso solves obviously also the equivalent problem $$\beta_{lasso} \in \arg\min\{\sum_{i=1}^n (y_i-\beta_0 - \sum_{j=1}^p x_{ij}\beta_j)^2 + \lambda^* \sum_{j=1}^p |\beta_j|\}$$ where $\lambda^* = 2 \lambda \geq 0$. Since lasso is a convex optimization problem, the solutions to the problems will be identical, moreover there is a one-to-one relationship between $\lambda^*$ and $\lambda$. finally, both equivalent minimization problems translate to the same constrained minimization problem (just with different $\lambda$'s): $$\min_{\beta}\sum_{i=1}^n (y_i-\beta_0 - \sum_{j=1}^p x_{ij}\beta_j)^2 \qquad s.t. \qquad \sum_{j=1}^p |\beta_j| \leq t.$$

The factor $\frac{1}{2}$ ist just introduced for convenience, i.e. to simplify the writing within the theoretical analysis of the lasso. for example, the KKT conditions are then nicely "scaled", otherwise you would carry the factor $2$ from the derivative of the quadratic sum with you during your whole analysis.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.