2
$\begingroup$

Suppose I have model 1: $$Y=aX$$ where $X$ is $n\times1$, consisting of a single feature. suppose I fit this model with $L_2$ penalty with coefficient $\lambda=1$ : $$ \underset{a}{\min} \sum_{i=1}^{n}(y_i-ax_i)^2 +a^2$$ and I find that $a=b$.

Now consider model 2 : $Y=a'X'+a''X''$ where $X',X''$ are just duplicates of $X$ so $X'=X''=X$. If we fit this model with $L_2$ penalty using $\lambda=1$, $$ \underset{a',a'' }{\min} \sum_{i=1}^{n}(y_i-a'x'_i-a''x''_i)^2 +a'^2 +a''^2$$

We find that $a'=a''=\frac b2$.

If we set $\lambda=2$, we get $$\underset{a',a'' }{\min} \sum_{i=1}^{n}(y_i-a'x'_i-a''x''_i)^2 +2(a'^2 +a''^2) = \min_{a' } \sum_{i=1}^{n}(y_i-2a'x_i)^2 +4a'^2$$ since $X'=X''$ and $a'=a''$. Letting $c=2a'$, this reduces to $$\underset{c}{\min} \sum_{i=1}^{n}(y_i-cx_i)^2 +c^2$$ which is the same optimization as in model 1, which yields $c=b$ and thus $a'=a''=\frac b2$ - same parameters even though the penalty strength is different. This seems to conflict with the analytic solution for Ridge regression: $$a_{ridge} = (X^TX+\lambda I)^{-1}X^TY$$ where different $\lambda$ lead to different parameters.

How do you reconcile this?

$\endgroup$

1 Answer 1

3
$\begingroup$

The mistake is in the claim that $a' = a'' = \frac{b}{2}$ would be optimal for model 2 with penalty $\lambda = 1$. This is not true, because the $L_2$ penalty with constant $\lambda$ penalizes one coefficient with value $a$ more than two coefficients summing to $a$.

If we look at the minimization problem of model 2 \begin{equation} \underset{a',a'' }{\min} \sum_{i=1}^{n}(y_i-a'x'_i-a''x''_i)^2 +a'^2 +a''^2 \end{equation} and substitute $x_i'=x_i''=x_i,~a'=a''=\frac{a}{2}$, we get \begin{equation} \underset{a }{\min} \sum_{i=1}^{n}(y_i-ax_i)^2 +\left(\frac{a}{2}\right)^2 +\left(\frac{a}{2}\right)^2, \end{equation} that is, \begin{equation} \underset{a }{\min} \sum_{i=1}^{n}(y_i-ax_i)^2 +\frac{a^2}{2}. \end{equation} That is, the penalty for $a = a'+a''$ is smaller in the model formulation 2 than in the model formulation 1 when $\lambda$ is kept constant. Therefore, the optimal $a'+a''$ will be greater than the optimal $a$ (and thus $a' \neq \frac{b}{2}$).

More generally, in model 1, the penalty term with a general $\lambda_1$ is $\lambda_1 a^2$, while in model 2 \begin{equation} \lambda_2 \left[(a')^2 + (a'')^2\right] = \lambda_2 \left[\left(\frac{a}{2}\right)^2 + \left(\frac{a}{2}\right)^2\right] = \frac{\lambda_2}{2} a^2. \end{equation}

From this we conclude that to obtain equivalent result from model 2, one needs to double the penalty term $\lambda$ (which corresponds to your derivations for model 2 with $\lambda=2$ being equivalent to model 1 with $\lambda=1$).

For further intuition, note that with equal $\lambda$s the objective functions would obtain equal values if one set $a'=a, a''=0$ in model 2. However, in model 2 the penalty can be decreased by spreading the responsibility to from $a'$ to $a''$, which then allows one to increase the sum $a'+a''$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.