Why does ridge regression apply a non-monotone transformation to the singular values of the design matrix?

Per Wikipedia, Ridge Regression is equivalent to transforming the singular values $$\sigma_i$$ of the design matrix to $$\frac{\sigma_i^2 + \alpha^2}{\sigma_i}$$, where $$\alpha$$ is (in Wikipedia's notation) the Ridge Regression parameter. The interesting thing is that the transformation $$f(\sigma) = \frac{\sigma^2 + \alpha^2}{\sigma}$$ is not a monotone transformation. It blows up as $$\sigma_i \to 0$$ and has an oblique asymptote as $$\sigma_i \to \infty$$. So the small and large singular vectors (since we will end up taking the reciprocals of the "singular values") and we're left with a Goldilocks situation where the stuff in the middle carries the most weight. Is this the right way to do things or merely how the algebra works out? Do there exist other forms of regularization (presumably not as algebraically clean as Ridge Regression) that are monotone, such as letting $$f(\sigma) = \sigma + \alpha$$?

• $x$ doesn't appear in any of your functions, so presumably you mean $\sigma_i$. And do you mean $f(\sigma_i)$?
– Sycorax
Feb 11 at 22:03
• Yes @Sycorax, fixed. Thanks. Feb 12 at 14:22