Skip to main content
2 of 6
added 73 characters in body
amoeba
  • 107.2k
  • 36
  • 321
  • 346

There does not seem to be a lot of literature, but here is a geometric illustration of what is going on with negative ridge.

I will consider estimators of the form $$\hat{\boldsymbol\beta}_\lambda = (\mathbf X^\top \mathbf X + \lambda \mathbf I)^{-1}\mathbf X^\top\mathbf y$$ arising from the loss function $$\mathcal L_\lambda = \|\mathbf y - \mathbf X\boldsymbol\beta\|^2 + \lambda \|\boldsymbol\beta\|^2.$$ Here is a rather standard illustration of what happens in a two-dimensional case with $\lambda\in[0,\infty)$:

enter image description here

Now consider what happens when $\lambda\in(-\infty, -s^2_\max)$, where $s_\mathrm{max}$ is the largest singular value of $\mathbf X$. For very large negative lambdas, $\hat{\boldsymbol\beta}_\lambda$ is of course close to zero. When lambda approaches $-s^2_\max$, the term $(\mathbf X^\top \mathbf X + \lambda \mathbf I)$ gets one singular value approaching zero, meaning that the inverse has one singular value going to minus infinity. This singular value corresponds to the first principal component of $\mathbf X$, so in the limit one gets $\hat{\boldsymbol\beta}_\lambda$ pointing in the direction of PC1 but with absolute value growing to infinity.

What is really nice, is that one can draw it on the same figure in the same way: betas are given by points where circles touch the ellipses from the inside. I do not really understand why this works (the loss function interpretation seems to have been lost when $\lambda<0$), but it does:

enter image description here

When $\lambda\in(-s^2_\mathrm{min},0]$, a similar logic applies, allowing to continue the ridge path on the other side of the OLS estimator. Now the circles touch the ellipses from the outside. In the limit, betas approach the PC2 direction (but it happens far outside this sketch):

enter image description here


The $\lambda\in(-\infty, -s^2_\max)$ range has an interpretation in terms of constrained ridge regression, see The limit of "unit-variance" ridge regression estimator when $\lambda\to\infty$. This is related to what is known in the chemometrics literature as "continuum regression", see my answer in the linked thread. I am currently not aware of any natural loss function interpretation for the $\lambda\in(-s^2_\mathrm{min},0]$ range.

amoeba
  • 107.2k
  • 36
  • 321
  • 346