3
$\begingroup$

Why is using centered or uncentered data equivalent in ridge regression? In other words, given two ridge regression problems: \begin{equation} (b',c')=\operatorname*{argmin}_{b,c}\Big[ { \sum_i^{m} (y_i - c - b^Tx_i)^2 + \lambda b^Tb}\Big] \end{equation}

$$(b'',c'')=\operatorname*{argmin}_{b,c} \Big[{ \sum_i^{m} (y_i - c - b^T(x_i - \bar{x}))^2 + \lambda b^Tb} \Big]$$ where $\bar{x}$ is the mean of the input data, why does $(b',c')$ correspond to $(b'',c'')$?

I'm writing a piece of code where this thing holds numerically, I was wondering what is the mathematical explanation.

Improve this question
$\endgroup$
8
  • $\begingroup$ can we see your code? $\endgroup$
    – TPArrow
    Commented Nov 27, 2015 at 17:21
  • $\begingroup$ Unfortunately the code is for an university assignment, I'm not so happy about posting it on the internet, I'd rather keep it theoretical. If it may help, I have found this sentence: 'If we center the columns of $x$, then the intercept estimate ends up just being $c= \overline{y}$' here stat.cmu.edu/~ryantibs/datamining/lectures/16-modr1-marked.pdf I still don't get it though. $\endgroup$
    – Puzzle
    Commented Nov 27, 2015 at 17:28
  • $\begingroup$ that's correct, the b term just depends on the variance of x. if x is not centered only c changes. remember that the b term is representing the change in y with a unit change in x $\endgroup$
    – seanv507
    Commented Nov 27, 2015 at 17:28
  • $\begingroup$ @seanv507 thank you. Could you please elaborate a bit? How can you say that b only depends on the variance of x? $\endgroup$
    – Puzzle
    Commented Nov 27, 2015 at 17:32
  • $\begingroup$ sorry, I meant it doesn't depend on the mean. in the univariate case, its the covariance of (x,y)/variance of x, but as I say its easier to think of it as the change in y with one unit change in each component of x (in multivariate case) $\endgroup$
    – seanv507
    Commented Nov 27, 2015 at 17:51

1 Answer 1

Reset to default
5
$\begingroup$

$f(b,c):=\sum_i^m(y_i-c-b^Tx_i)^2+\lambda b^T b$ is equivalent to $g(d,e):=\sum_i^m(y_i-e-d^T (x_i-\bar x))^2+\lambda d^T d$ under the change of variables $d=b,e=c+b^T \bar x$

ie $f(b,c)=g(b,c+b^T\bar x)$.

Therefore they have the same minimisers [same constraints on (b,c) vs (d,e)]. But this change of variables corresponds to using centred or uncentred data.

It should be noted that this only works when the regularisation is not on the constant term. Although regularisation is typically performed as above, some software also penalises the constant/bias term.

Improve this answer
$\endgroup$
3
  • $\begingroup$ This answer looks very much unfinished; did you intend it to be like that? In its current form it's more of a comment. Would be great if you could extend it to a proper answer. $\endgroup$
    – amoeba
    Commented Nov 29, 2015 at 22:09
  • $\begingroup$ @amoeba - hope this is better:) $\endgroup$
    – seanv507
    Commented Nov 30, 2015 at 16:25
  • 2
    $\begingroup$ +1. Yes, thanks. I would add that this only works because the intercept $c$ is not penalized. $\endgroup$
    – amoeba
    Commented Nov 30, 2015 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.