So let's say we ran a Ridge or Lasso regression on $Y \sim X$, and get coefficient $\beta_X$. Now if we duplicate the $X$, and call it $Z$, and then run the same regression on: $Y \sim X + Z$. How will the coefficients change in both the Ridge case and the Lasso case?

So it's clear that Lasso will push one towards zero, but which one? Is it just dependent on how you initialize your random weights when doing gradient descent? What about for ridge? This was mentioned in a lecture once where the instructor said "clearly the two new coefficients: $\beta'_X + \beta'_Z = \beta_X$ but I don't see why? Also can we tell anything about the values of the individual beta's?

Can someone provide some clearer guidance on how to tackle this problem and what the answer will be? I've tried looking this up online but couldn't get anywhere


$\newcommand{\x}{\mathbf x}$$\newcommand{\one}{\mathbf 1}$$\newcommand{\X}{\mathbf X}$@kjetil b halvorsen's linked answer explains what's happening, but here's an algebraic answer just for the case of ridge regression (since there's a closed form for the solution).

Suppose we have $X\in\mathbb R^{n\times (p+k)}$ as our feature matrix where $$ X = (\underbrace{\x \mid \x \mid \dots \mid \x}_{k\text{ times}} \mid Z) $$ with $Z\in\mathbb R^{n\times p}$ containing whatever other columns we may have. Note that $X\beta = XP^TP\beta$ where $P$ is a permutation matrix so this shows how it's safe to assume a particular ordering of the columns of $X$ so long as we correspondingly permute the elements of $\beta$. I'll use $\X = (\x\mid \dots\mid \x) \in \mathbb R^{n\times k}$.

We have $$ \hat\beta_R = (X^TX + \lambda I_{p+k})^{-1}X^Ty \\ = \left[\begin{array}{c|c} \x^T\x \one_k\one_k^T + \lambda I_k & \X^TZ \\ \hline Z^T\X & Z^TZ + \lambda I_p \end{array}\right]^{-1} \left[\begin{array}{c}\X^Ty \\ \hline Z^Ty\end{array}\right]. $$ I'll invert that matrix as a 2x2 block matrix (and we know it's always invertible). Letting $$ \left[\begin{array}{c|c} \x^T\x \one_k\one_k^T + \lambda I_k & \X^TZ \\ \hline Z^T\X & Z^TZ + \lambda I_p \end{array}\right] = \begin{bmatrix} A & B \\ C & D\end{bmatrix} $$ we have $$ \left[\begin{array}{cc} A & B \\ C & D\end{array}\right]^{-1} = \left[\begin{array}{c|c} (A-BD^{-1}C)^{-1} & -(A-BD^{-1}C)^{-1}BD^{-1} \\\hline \cdot & \cdot\end{array}\right] $$ where I've marked the lower row of blocks with $\cdot$ since we don't need those for the $k$ coefficients of $\x$. Letting $H_Z = Z(Z^TZ + \lambda I)^{-1}Z^T$ be the hat matrix for a ridge regression just on $Z$, we can show that $$ BD^{-1}C = \X^TH_Z \X = \x^T H_Z \x \one\one^T $$ and $$ A = \x^T\x\one\one^T + \lambda I_k $$ so $$ A - BD^{-1}C = \x^T(I-H_Z)\x\one\one^T + \lambda I_k. $$ Then $$ BD^{-1} = \X^T Z(Z^TZ + \lambda I_p)^{-1} $$ so all together we get $$ \hat\beta_{R;k} = \left(\x^T(I-H_Z)\x\one\one^T + \lambda I_k\right)^{-1}\x^T (I-H_Z) y \one. $$ Factoring out the scalar $\x^T(I-H_Z)\x$ this can be written as $$ \hat\beta_{R;k} = \frac{\x^T(I-H_Z)y}{\x^T(I-H_Z)\x}\left(\one\one^T + \nu I_k\right)^{-1}\one $$ where for convenience I'm letting $$ \nu = \frac{\lambda}{\x^T(I-H_Z)\x}. $$ This inverse can be done explicitly via Sherman-Morrison so $$ \left(\one\one^T + \nu I\right)^{-1}\one = \nu^{-1}(I - (\nu + k)^{-1}\one\one^T)\one\\ = \frac{1}{\nu + k}\one. $$

This means $$ \hat\beta_{R;k} = (\nu + k)^{-1}\frac{\x^T(I-H_Z)y}{\x^T(I-H_Z)\x}\one \\ = \frac{\x^T(I-H_Z)y}{k\x^T(I-H_Z)\x + \lambda}\one $$ where I've substituted back in what $\nu$ is equal to. This shows that each coefficient of $\x$ is the same thing, and that it is a shrunken form of what we'd get if $\x$ wasn't copied (if there was just one $\x$, like in the standard setting, the $k$ would disappear but the rest would be the same).

This also shows what the sum of these coefficients is: $$ \hat\beta_{R;k} ^T\one = \frac{\x^T(I-H_Z)y}{k\x^T(I-H_Z)\x + \lambda}\one^T\one \\ = \frac{\x^T(I-H_Z)y}{\x^T(I-H_Z)\x + \lambda/k} $$ so the coefficients don't quite sum to what we'd have if $\x$ wasn't copied, but rather their sum is what we'd get if we just had one $\x$ but divided the regularization parameter by $k$.

| cite | improve this answer | |
  • 2
    $\begingroup$ +1 -- it's great to see this worked out so carefully. Maybe you could have spared yourself a lot of algebra--and perhaps obtain a little more insight--by using the characterization of ridge regression at stats.stackexchange.com/a/164546/919 as a starting point, which leads immediately to your final observation. $\endgroup$ – whuber Jul 20 at 18:38
  • 1
    $\begingroup$ @whuber thank you, and good point! I will try to update soon as I have time with that argument $\endgroup$ – jld Jul 21 at 13:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.