5
$\begingroup$

Consider the penalized linear regression problem:
$$ \text{minimize}_\beta \,\,(y-X\beta)^T(y-X\beta)+\lambda \sqrt{\sum \beta_i^2} $$ Without the square root this problem becomes ridge regression. Note that this is not the LASSO problem which may be expressed as:
$$ \text{minimize}_\beta \,\,(y-X\beta)^T(y-X\beta)+\lambda \sum \sqrt{ \beta_i^2} $$ This is also a special case of group LASSO when all coefficients are within one group. Is there a closed form solution to this problem?

$\endgroup$
  • $\begingroup$ This is a different problem. That question asks about the solution for L1 norm regularization i.e. lasso. My question is about a different regularization. $\endgroup$ – Z. Li Oct 6 '15 at 2:49
  • 2
    $\begingroup$ The penalty term is $\sqrt{\sum{\beta_i^2}}$, or the Euclidean norm, which is not equivalent to $\sum|\beta_i|$ when there are more than one element in $\beta$. (also see my edited question for clarification). $\endgroup$ – Z. Li Oct 6 '15 at 2:54
  • $\begingroup$ @user777, the sum is inside the square root in OP's case, but the square root is inside the sum for LASSO. $\endgroup$ – Matt Krause Oct 6 '15 at 5:32
  • $\begingroup$ I realize my mistake now. Thanks for clarifying your question. $\endgroup$ – Sycorax Oct 6 '15 at 13:20
8
$\begingroup$

You will get the ridge regression solutions, but parametrised differently in terms of the penalty parameter $\lambda$. This holds more generally for convex loss functions.

If $L$ is a convex, differentiable function of $\beta$ let $\beta(\lambda)$ denote the unique minimiser of the strictly convex function $$h(\beta) = L(\beta) + \lambda \|\beta\|_2^2$$ for $\lambda > 0$. Let, furthermore, $s(\lambda) = \|\beta(\lambda)\|_2$.

Consider now the function $$g(\beta) = L(\beta) + 2 \lambda s(\lambda) \|\beta\|_2.$$ Its Jacobian is $$Dg(\beta) = DL(\beta) + 2 \lambda s(\lambda) \frac{\beta}{\|\beta\|_2}.$$ If we plug in $\beta(\lambda)$ we find that $$Dg(\beta(\lambda)) = DL(\beta(\lambda)) + 2 \lambda \beta(\lambda) = Dh(\beta(\lambda) = 0,$$ because $\beta(\lambda)$ is a stationary point of $h$. Since $g$ is still convex this shows that $\beta(\lambda)$ is a global minimiser of $g$.

It is possible that $\lambda \mapsto \lambda s(\lambda)$ does not map $(0, \infty)$ onto $(0,\infty)$, thus there can be choices of the penalty parameter $-$ when the $\|\cdot\|_2$-penalty and not the $\|\cdot\|_2^2$-penalty is used $-$ that give minimisers that are not of the form $\beta(\lambda)$ for any $\lambda > 0$. With the squared error loss (yielding ridge regression) this will be the case for large choices of the penalty parameter, where the $\|\cdot\|_2$-penalty will give the zero solution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.