For Ridge Regression, I've seen the statement that the solutions aren't equivariant to scaling of the inputs so we typically preprocess the response and regressors by standardizing.

What does "equivariant to scaling of the inputs" mean? Is OLS equivariant to scaling of inputs?

  • 1
    $\begingroup$ See the characterization of ridge regression at stats.stackexchange.com/a/164546/919 and consider what happens to $X_{*}$ after standardizing $X$ (the original model matrix). $\endgroup$
    – whuber
    Jul 16, 2020 at 22:02
  • $\begingroup$ @whuber I will take a look, but you mention "standardize," which is well known as subtracting the mean and dividing by the SD. Is "scaling" equivalent to "standardizing?" $\endgroup$ Jul 16, 2020 at 22:24
  • $\begingroup$ Scaling typically means standardizing without the preliminary recentering. It doesn't matter, though: the point is that when you alter the columns of $X$ you don't alter the artificial data that Ridge Regression tacks on to them. Thus, the Ridge Regression solution depends on exactly what numbers are used to express your data, whereas the OLS solution does not. $\endgroup$
    – whuber
    Jul 17, 2020 at 14:41
  • $\begingroup$ @whuber Ah so my understanding is for OLS, if you scale the input X, you can simply just scale the estimators by the inverse of the scaling of X. However, you can't do this with Ridge because if you look at the solution for ridge, there's a $+ \lambda I$ term in the matrix you're inverting. If the scaling was also applied to the regularization term, then ridge regression would be equivariant. $\endgroup$ Jul 17, 2020 at 16:53
  • $\begingroup$ Yes, that's right. Another way to put it is that the strength of the regularization, as reflected in the magnitude of $\lambda$ relative to the magnitude of each column, depends on how you scale each column. $\endgroup$
    – whuber
    Jul 17, 2020 at 17:48


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.