in Rigid regression, we generally set a positive Lambda for regularization to get a less Residual. Why cant we have a negative Lambda in a regularization if we can benefit from it?

enter image description here

  • $\begingroup$ We do not set a positive $\lambda$ to get smaller residuals; any positive $\lambda$ will produce larger residuals than setting $\lambda = 0$. $\endgroup$ – jbowman Nov 17 '19 at 3:05
  • $\begingroup$ Also see understanding negative ridge regression $\endgroup$ – user20160 Nov 17 '19 at 3:14

There are two (equivalent) formulations of ridge regression (I mention both because I'm not sure which version you're referring to). If $\mathbf{X} \in \mathbb{R}^{n \times p}$ is the design matrix and $\mathbf{y} \in \mathbb{R}^n$ is the target vector, then the two formulations are the following ($\|\cdot\|_2$ denotes the $L^2$ norm).

  1. The constrained optimization form: $$ \hat{\boldsymbol{\theta}}_{\text{ridge}} = \operatorname*{arg\,min}_{\substack{\boldsymbol{\theta} \in \mathbb{R}^p \\ \|\boldsymbol{\theta}\|_2 \leq \lambda}} \left\|\mathbf{X}\boldsymbol{\theta} - \mathbf{y}\right\|_2^2. $$ Here, if $\lambda < 0$, then the constraint $\|\boldsymbol{\theta}\|_2 \leq \lambda$ never holds, so the optimization problem is ill-defined.

  2. The Lagrangian form: $$ \hat{\boldsymbol{\theta}}_{\text{ridge}} = \operatorname*{arg\,min}_{\boldsymbol{\theta} \in \mathbb{R}^p} \left( \left\|\mathbf{X}\boldsymbol{\theta} - \mathbf{y}\right\|_2^2 + \lambda \|\boldsymbol{\theta}\|_2^2\right) $$ (this $\lambda$ is not the same as the $\lambda$ in the first formulation, but they are related). Here, if $\lambda < 0$, then the optimization problem encourages the $L^2$ norm of the vector $\boldsymbol{\theta}$ to get as large as possible, which goes against the point of regularization.

Thus, in both formulations of ridge regressions, choosing a negative $\lambda$ leads to undesirable results.

  • 2
    $\begingroup$ In the lagrangian form with negative $\lambda$, there is no minimum because the norm of $\theta$ can grow without bound. But, apparently people have used variants of 'negative ridge regression' (e.g. the paper referenced in this thread) $\endgroup$ – user20160 Nov 17 '19 at 3:32
  • $\begingroup$ @user20160 interesting! I hadn't heard of negative ridge regression, thanks for the link! $\endgroup$ – Artem Mavrin Nov 17 '19 at 3:37

Not the answer you're looking for? Browse other questions tagged or ask your own question.