Per Wikipedia, Ridge Regression is equivalent to transforming the singular values $\sigma_i$ of the design matrix to $\frac{\sigma_i^2 + \alpha^2}{\sigma_i}$, where $\alpha$ is (in Wikipedia's notation) the Ridge Regression parameter. The interesting thing is that the transformation $f(\sigma) = \frac{\sigma^2 + \alpha^2}{\sigma}$ is not a monotone transformation. It blows up as $\sigma_i \to 0$ and has an oblique asymptote as $\sigma_i \to \infty$. So the small and large singular vectors (since we will end up taking the reciprocals of the "singular values") and we're left with a Goldilocks situation where the stuff in the middle carries the most weight. Is this the right way to do things or merely how the algebra works out? Do there exist other forms of regularization (presumably not as algebraically clean as Ridge Regression) that are monotone, such as letting $f(\sigma) = \sigma + \alpha$?

  • 1
    $\begingroup$ $x$ doesn't appear in any of your functions, so presumably you mean $\sigma_i$. And do you mean $f(\sigma_i)$? $\endgroup$
    – Sycorax
    Feb 11 at 22:03
  • $\begingroup$ Yes @Sycorax, fixed. Thanks. $\endgroup$
    – kuzzooroo
    Feb 12 at 14:22


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.