3
$\begingroup$

Why is using centered or uncentered data equivalent in ridge regression? In other words, given two ridge regression problems: \begin{equation} (b',c')=\operatorname*{argmin}_{b,c}\Big[ { \sum_i^{m} (y_i - c - b^Tx_i)^2 + \lambda b^Tb}\Big] \end{equation}

$$(b'',c'')=\operatorname*{argmin}_{b,c} \Big[{ \sum_i^{m} (y_i - c - b^T(x_i - \bar{x}))^2 + \lambda b^Tb} \Big]$$ where $\bar{x}$ is the mean of the input data, why does $(b',c')$ correspond to $(b'',c'')$?

I'm writing a piece of code where this thing holds numerically, I was wondering what is the mathematical explanation.

$\endgroup$
  • $\begingroup$ can we see your code? $\endgroup$ – TPArrow Nov 27 '15 at 17:21
  • $\begingroup$ Unfortunately the code is for an university assignment, I'm not so happy about posting it on the internet, I'd rather keep it theoretical. If it may help, I have found this sentence: 'If we center the columns of $x$, then the intercept estimate ends up just being $c= \overline{y}$' here stat.cmu.edu/~ryantibs/datamining/lectures/16-modr1-marked.pdf I still don't get it though. $\endgroup$ – Puzzle Nov 27 '15 at 17:28
  • $\begingroup$ that's correct, the b term just depends on the variance of x. if x is not centered only c changes. remember that the b term is representing the change in y with a unit change in x $\endgroup$ – seanv507 Nov 27 '15 at 17:28
  • $\begingroup$ @seanv507 thank you. Could you please elaborate a bit? How can you say that b only depends on the variance of x? $\endgroup$ – Puzzle Nov 27 '15 at 17:32
  • $\begingroup$ sorry, I meant it doesn't depend on the mean. in the univariate case, its the covariance of (x,y)/variance of x, but as I say its easier to think of it as the change in y with one unit change in each component of x (in multivariate case) $\endgroup$ – seanv507 Nov 27 '15 at 17:51
3
$\begingroup$

$f(b,c):=\sum_i^m(y_i-c-b^Tx_i)^2+\lambda b^T b$ is equivalent to $g(d,e):=\sum_i^m(y_i-e-d^T (x_i-\bar x))^2+\lambda d^T d$ under the change of variables $d=b,e=c+b^T \bar x$

ie $f(b,c)=g(b,c+b^T\bar x)$.

Therefore they have the same minimisers [same constraints on (b,c) vs (d,e)]. But this change of variables corresponds to using centred or uncentred data.

It should be noted that this only works when the regularisation is not on the constant term. Although regularisation is typically performed as above, some software also penalises the constant/bias term.

$\endgroup$
  • $\begingroup$ This answer looks very much unfinished; did you intend it to be like that? In its current form it's more of a comment. Would be great if you could extend it to a proper answer. $\endgroup$ – amoeba Nov 29 '15 at 22:09
  • $\begingroup$ @amoeba - hope this is better:) $\endgroup$ – seanv507 Nov 30 '15 at 16:25
  • 1
    $\begingroup$ +1. Yes, thanks. I would add that this only works because the intercept $c$ is not penalized. $\endgroup$ – amoeba Nov 30 '15 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.