Does ridge regression with a duplicate column actually halve the coefficient?

Question

The following is an excerpt from "Ridge Regularization: An Essential Concept in Data Science" by Trevor Hastie.

How does one actually prove this? It seems to depend on understanding how the singular values and right singular vectors change when duplicating a column, but it doesn't seem to be clear exactly how those change (e.g., https://math.stackexchange.com/questions/244399/how-does-the-left-singular-value-decomposition-change-when-one-duplicates-a-colu)

Answer 1

I won't offer a formal proof but this motivation for it in the "two copies of the variable" case could be turned into one (and a similar argument for more copies).

Imagine you have a regression model with some predictor $X_1$ with coefficient $\hat{\beta}_1$ and potentially other predictors ($V_1, V_2,...$, say), if needed. Let $X_2=X_1$ and add it to the model.

A weighted mixture of these two predictors $w X_1 + (1-w)X_2$ as a predictor would reproduce the fit of the original model (with the same coefficient); some other mixing proportion would result in a worse fit. This takes care of the contribution to the ridge loss function from the fit itself - that kind of mixture with any $w$ would do for minimizing the squared error loss term.

Now consider that if you add both terms to the model independently, the $w$ and $1-w$ will impact the estimated coefficients (look at it not as $\beta_1 (w X_1)$ but as $(\beta_1 w) X_1$).

Everything else being equal, to minimize the ridge penalty, we're minimizing $\hat{\beta_1}^2(w^2 +(1-w)^2)$ over $w$ where $\hat{\beta_1}^2$ is again the coefficient from the original model. Which choice of $w$ will minimize the ridge penalty?

Answer 2

We can see that it is not literally true by contradiction.

Say we have three predictors $X_1$, $X_2$ and $X_3$. Let us assume that $X_1 = X_2 = X_3 = X$. Consider the three fit ridge models

$$ \begin{align*} &\beta_1 X + \beta_2 X + \beta_3 X \\ & \beta_{12} X + \beta_3^{'} X \\ & \beta_{123} X \end{align*} $$

Taken literally, the claim in ESL implies each of the following:

$$ \begin{align*} &\beta_1 = \beta_2 = \frac{1}{2} \beta_{12}\\ & \beta_{12} = \beta_3^{'} = \frac{1}{2} \beta_{123}\\ &\beta_1 = \beta_2 = \beta_3 = \frac{1}{3} \beta_{123} \end{align*} $$

These are easily seen to be inconsistent. Writing everything in terms of $\beta_{123}$ we have $\beta_{12} = \frac{2}{3} \beta_{123}$ from the first equation and $\beta_{12} = \frac{1}{2} \beta_{123}$ from the second equation.

So the precise claim in ESL is false. I think the intuition provided by @Gleb_b's answer is what they were trying to get at. There is some qualitative/intuitive content here but not a precise theorem.

Answer 3

A simulation based counterexample:

import numpy as np
from sklearn.linear_model import Ridge

from numpy.random import RandomState

prng1 = RandomState(4243)
prng2 = RandomState(1258)
prng3 = RandomState(3991)

nobs = 5
x = prng1.normal(size = nobs)
X = np.ones((nobs,4))
X[:,1] = x
X[:,2] = x
X[:,3] = prng2.normal(size = nobs)
y = 3 + np.dot(X, [0, 2, 1, 4]) + prng3.normal(size = nobs)

model1 = Ridge(alpha = 1, fit_intercept = False)
model1.fit(X,y)

model2 = Ridge(alpha = 1, fit_intercept = False)
model2.fit(X[:,[0,1,3]], y)

print(model1.coef_)
print(model2.coef_)

Results:

[2.59857301 0.50722255 0.50722255 1.7140635 ]
[2.66713959 0.7987741  1.55787553]

Note that 0.5 is not half of 0.79.