# Is there a closed form solution for L2-norm regularized linear regression (not ridge regression)

Consider the penalized linear regression problem:
$$\text{minimize}_\beta \,\,(y-X\beta)^T(y-X\beta)+\lambda \sqrt{\sum \beta_i^2}$$ Without the square root this problem becomes ridge regression. Note that this is not the LASSO problem which may be expressed as:
$$\text{minimize}_\beta \,\,(y-X\beta)^T(y-X\beta)+\lambda \sum \sqrt{ \beta_i^2}$$ This is also a special case of group LASSO when all coefficients are within one group. Is there a closed form solution to this problem?

• This is a different problem. That question asks about the solution for L1 norm regularization i.e. lasso. My question is about a different regularization. Commented Oct 6, 2015 at 2:49
• The penalty term is $\sqrt{\sum{\beta_i^2}}$, or the Euclidean norm, which is not equivalent to $\sum|\beta_i|$ when there are more than one element in $\beta$. (also see my edited question for clarification). Commented Oct 6, 2015 at 2:54
• @user777, the sum is inside the square root in OP's case, but the square root is inside the sum for LASSO. Commented Oct 6, 2015 at 5:32
• I realize my mistake now. Thanks for clarifying your question.
– Sycorax
Commented Oct 6, 2015 at 13:20

## 1 Answer

You will get the ridge regression solutions, but parametrised differently in terms of the penalty parameter $\lambda$. This holds more generally for convex loss functions.

If $L$ is a convex, differentiable function of $\beta$ let $\beta(\lambda)$ denote the unique minimiser of the strictly convex function $$h(\beta) = L(\beta) + \lambda \|\beta\|_2^2$$ for $\lambda > 0$. Let, furthermore, $s(\lambda) = \|\beta(\lambda)\|_2$.

Consider now the function $$g(\beta) = L(\beta) + 2 \lambda s(\lambda) \|\beta\|_2.$$ Its Jacobian is $$Dg(\beta) = DL(\beta) + 2 \lambda s(\lambda) \frac{\beta}{\|\beta\|_2}.$$ If we plug in $\beta(\lambda)$ we find that $$Dg(\beta(\lambda)) = DL(\beta(\lambda)) + 2 \lambda \beta(\lambda) = Dh(\beta(\lambda) = 0,$$ because $\beta(\lambda)$ is a stationary point of $h$. Since $g$ is still convex this shows that $\beta(\lambda)$ is a global minimiser of $g$.

It is possible that $\lambda \mapsto \lambda s(\lambda)$ does not map $(0, \infty)$ onto $(0,\infty)$, thus there can be choices of the penalty parameter $-$ when the $\|\cdot\|_2$-penalty and not the $\|\cdot\|_2^2$-penalty is used $-$ that give minimisers that are not of the form $\beta(\lambda)$ for any $\lambda > 0$. With the squared error loss (yielding ridge regression) this will be the case for large choices of the penalty parameter, where the $\|\cdot\|_2$-penalty will give the zero solution.