# Linear Regression with $L_2$: Different penalty strengths yield the same parameters?

Suppose I have model 1: $$Y=aX$$ where $X$ is $n\times1$, consisting of a single feature. suppose I fit this model with $L_2$ penalty with coefficient $\lambda=1$ : $$\underset{a}{\min} \sum_{i=1}^{n}(y_i-ax_i)^2 +a^2$$ and I find that $a=b$.

Now consider model 2 : $Y=a'X'+a''X''$ where $X',X''$ are just duplicates of $X$ so $X'=X''=X$. If we fit this model with $L_2$ penalty using $\lambda=1$, $$\underset{a',a'' }{\min} \sum_{i=1}^{n}(y_i-a'x'_i-a''x''_i)^2 +a'^2 +a''^2$$

We find that $a'=a''=\frac b2$.

If we set $\lambda=2$, we get $$\underset{a',a'' }{\min} \sum_{i=1}^{n}(y_i-a'x'_i-a''x''_i)^2 +2(a'^2 +a''^2) = \min_{a' } \sum_{i=1}^{n}(y_i-2a'x_i)^2 +4a'^2$$ since $X'=X''$ and $a'=a''$. Letting $c=2a'$, this reduces to $$\underset{c}{\min} \sum_{i=1}^{n}(y_i-cx_i)^2 +c^2$$ which is the same optimization as in model 1, which yields $c=b$ and thus $a'=a''=\frac b2$ - same parameters even though the penalty strength is different. This seems to conflict with the analytic solution for Ridge regression: $$a_{ridge} = (X^TX+\lambda I)^{-1}X^TY$$ where different $\lambda$ lead to different parameters.

How do you reconcile this?

The mistake is in the claim that $a' = a'' = \frac{b}{2}$ would be optimal for model 2 with penalty $\lambda = 1$. This is not true, because the $L_2$ penalty with constant $\lambda$ penalizes one coefficient with value $a$ more than two coefficients summing to $a$.
If we look at the minimization problem of model 2 $$\underset{a',a'' }{\min} \sum_{i=1}^{n}(y_i-a'x'_i-a''x''_i)^2 +a'^2 +a''^2$$ and substitute $x_i'=x_i''=x_i,~a'=a''=\frac{a}{2}$, we get $$\underset{a }{\min} \sum_{i=1}^{n}(y_i-ax_i)^2 +\left(\frac{a}{2}\right)^2 +\left(\frac{a}{2}\right)^2,$$ that is, $$\underset{a }{\min} \sum_{i=1}^{n}(y_i-ax_i)^2 +\frac{a^2}{2}.$$ That is, the penalty for $a = a'+a''$ is smaller in the model formulation 2 than in the model formulation 1 when $\lambda$ is kept constant. Therefore, the optimal $a'+a''$ will be greater than the optimal $a$ (and thus $a' \neq \frac{b}{2}$).
More generally, in model 1, the penalty term with a general $\lambda_1$ is $\lambda_1 a^2$, while in model 2 $$\lambda_2 \left[(a')^2 + (a'')^2\right] = \lambda_2 \left[\left(\frac{a}{2}\right)^2 + \left(\frac{a}{2}\right)^2\right] = \frac{\lambda_2}{2} a^2.$$
From this we conclude that to obtain equivalent result from model 2, one needs to double the penalty term $\lambda$ (which corresponds to your derivations for model 2 with $\lambda=2$ being equivalent to model 1 with $\lambda=1$).
For further intuition, note that with equal $\lambda$s the objective functions would obtain equal values if one set $a'=a, a''=0$ in model 2. However, in model 2 the penalty can be decreased by spreading the responsibility to from $a'$ to $a''$, which then allows one to increase the sum $a'+a''$.