Per Wikipedia, Ridge Regression is equivalent to transforming the singular values $\sigma_i$ of the design matrix to $\frac{\sigma_i^2 + \alpha^2}{\sigma_i}$, where $\alpha$ is (in Wikipedia's notation) the Ridge Regression parameter. The interesting thing is that the transformation $f(\sigma) = \frac{\sigma^2 + \alpha^2}{\sigma}$ is not a monotone transformation. It blows up as $\sigma_i \to 0$ and has an oblique asymptote as $\sigma_i \to \infty$. So the small and large singular vectors (since we will end up taking the reciprocals of the "singular values") and we're left with a Goldilocks situation where the stuff in the middle carries the most weight. Is this the right way to do things or merely how the algebra works out? Do there exist other forms of regularization (presumably not as algebraically clean as Ridge Regression) that are monotone, such as letting $f(\sigma) = \sigma + \alpha$?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.