Suppose we have two equations

$$ Y=\alpha_1X_1+\alpha_3X_3 $$


$$ Y=\beta_1X_1+\beta_2X_2+\beta_3X_3 $$

Suppose further that $X_1=X_2$, then would it usually be the case that $\hat{\alpha_1}$ or $\left(\hat{\beta_1} + \hat{\beta_2}\right)$ is larger, if we were using ridge regression, such that we tried to minimize the criterion, (where RSS is the usual residual sum of squares for linear regression)

$$ RSS + \lambda\sum_{j=1}^p \beta_j^2 $$

My simulations show that $\left(\hat{\beta_1} + \hat{\beta_2}\right)$ is larger, and I have tried to prove this by letting

\begin{align} Y&=\beta_1X_1+\beta_2X_2+\beta_3X_3 \\ &= \left(\beta_1+\beta_2\right)X_1+\beta_3X_3 \end{align}

In this case, structurally $\left(\beta_1+\beta_2\right) = \alpha_1$, and so it seems they would be equal. Additionally, would it be the case $\hat{\alpha_1}+\hat{\beta_2}$ would be positive or negative?

  • 2
    $\begingroup$ Clarifying question: this is for a fixed value of the regularization parameter $\lambda$? $\endgroup$ – Matthew Drury May 3 at 18:21
  • $\begingroup$ The penalty in the second model is smaller than in the first. You can deduce the rest, provided by "larger" you mean larger in absolute value. $\endgroup$ – whuber May 3 at 20:51
  • $\begingroup$ @whuber It seems the implication is that the $RSS$ in both models are the same, why would that be? $\endgroup$ – user321627 May 10 at 13:12
  • $\begingroup$ Let me invert the question: how could it possibly not be? Whenever you can achieve a particular RSS in the first model, setting $\beta_1=\beta_2=\alpha_1/2$ in the second model achieves that RSS in the second model, showing the minimal RSS in the second model is no greater than the minimal RSS in the first model. Whenever you can achieve a particular RSS in the second model, setting $\alpha_1=\beta_1+\beta_2$ produces that RSS in the first model. Thus, the minimal RSSes in these models must be identical. $\endgroup$ – whuber May 10 at 14:30

Your own reasoning here is flawed because ---although it is true that the respective coefficients appear in the model equation in the same way--- the penalties applying to the fit are different for the two cases. Ultimately, we need to figure out which method penalises analogous coefficients more heavily. I'm going to try to give you an intuitive answer, appealing to a comparison where we first optimise the "split" between the two coefficients in the sum. (I'm also going to assume that you intended for your model equations to contain error terms, so that these are (non-deterministic) statistical models.)

Reframing the problem: Since you have stipulated that $X_1=X_2$, let's simplify your model expressions to:

$$\begin{matrix} \text{Model } 1 & & & Y = \alpha_1 X_1 + \alpha_3 X_3 + \varepsilon, \quad \quad \quad \ \\[8pt] \text{Model } 2 & & & Y = (\beta_1+\beta_2) X_1 + \beta_3 X_3 + \varepsilon_*. \\[6pt] \end{matrix}$$

Note that this is a case where the second model is non-identifiable, because we have split the identifiable parameter $\alpha_1$ into the sum $\beta_1+\beta_2$ containing the non-identifiable parameters $\beta_1$ and $\beta_2$. Under ridge regression we penalise proportionately to the norm of the coefficient vector, so the question here hinges on whether we will reduce or increase this norm (and thus the penalty) by splitting the identifiable parameter into a sum of non-identifiable parameters. To check this, we will first examine the optimal "split" in the sum $\beta_1+\beta_2$ and then compare the optimisations.

Formal analysis: Since the residual-sum-of-squares in Model 2 depends on $\beta_1$ and $\beta_2$ only through their sum, let us first inquire as to how we can form these parameters to minimise their penalty, conditional on a particular sum. This is the constrained optimisation problem:

$$\text{Minimise } \beta_1^2 + \beta_2^2 \quad \quad \text{subject to} \quad \quad \beta_1 + \beta_2 = \text{const}.$$

Some simple calculus shows that the minimum occurs when $\beta_1 = \beta_2$. Now, using this as a partial requirement for the overall optimisation, consider the following three objective functions for fitting the models:

$$\begin{matrix} F_1(\alpha_3, \alpha_3) = \text{RSS}_1 + \lambda (\alpha_1^2 + \alpha_3^2), \quad \ \ \\[8pt] F_2(\beta_1, \beta_3) = \text{RSS}_2 + \lambda (2\beta_1^2 + \beta_3^2), \quad \\[8pt] F_2^*(\beta_1, \beta_3) = \text{RSS}_2 + \lambda ((2\beta_1)^2 + \beta_3^2). \\[6pt] \end{matrix}$$

The functions $F_1$ and $F_2$ are the actual objective functions for the models. The function $F_2^*$ is the minimisation problem that would apply if we fit Model 2 but treated $\beta_1 + \beta_2$ as a single coefficient in the penalty function (i.e., as if it were fitted like Model 1). Since $(2\beta_1)^2 = 4 \beta_1^2 \geqslant 2 \beta_1^2$, we can instantly see that the latter objective has a higher penalty for any value $\beta_1 \neq 0$. This means that Model 1 penalises the coefficient on the first term more heavily and so the coefficient will tend to be smaller in magnitude under this model --- i.e., we have:

$$|\hat{\alpha}_1| \leqslant |\hat{\beta}_1 + \hat{\beta}_2|,$$

and this inequality will tend to be strict for cases where the true coefficient is non-zero. (The exact conditions for a strict inequality are complicated, but roughly speaking this will occur when the RSS is continuously differentiable in $\beta_1$, with a maximum that occurs when $\beta_1 \neq 0$. In this case the higher penalty will push the optima for $\beta_1$ downward in magnitude in a continuous way.) Thus, we see that the magnitude of the slope coefficient for $X_1$ will tend to be smaller under Model 1 than under Model 2.

  • $\begingroup$ Thank you. I am confused how comparing $F_2(\beta_1, \beta_3)$ and $F_2^*(\beta_1, \beta_3)$ leads to knowledge of $F_1(\alpha_1, \alpha_3)$. That is, how the penalty under $F^*_2$ being greater than the penalty under $F_2$ leads to the penalty of $F^*_2$ being greater than that of $F_1$. It seems like $F_2$ is almost serving as a proxy for $F_1$. Why is that? Additionally, if we didn't use the partial requirement of $\beta_1 = \beta_2$, is it still showable through the fact $|\beta^2_1+\beta^2_2| < |(\beta_1+\beta_2)^2|$? $\endgroup$ – user321627 May 7 at 6:34

Because of the constraint of Rigde Regression, assuming your are obtaining the same constraints to solve with the same value then:

$$ RSS + \lambda\sum_{j=1}^3 \beta_j^2 = RSS + \lambda\sum_{j=1}^2 \alpha_j^2 $$ $$ \beta_1^2 + \beta_2^2 = \alpha_1^2 $$ Given than $X_1 = X_2 \implies \beta_1 = \beta_2$. This is theoretically, notice that in practice this means multicollinearity and this may affect the way the algorithms perform numerically. $$ 2\beta_1^2= \alpha_1^2 $$ $$ \beta_1= \frac{\alpha_1 }{\sqrt{2}} $$ That means that the coefficients are related. If the coefficient are related how is $\beta_1 + \beta_2$? $$ \beta_1 + \beta_2 = 2\beta_1= 2 \frac{\alpha_1 }{\sqrt{2}} = \sqrt{2}\alpha_1 \geq \alpha_1 $$

So that confirms the simulations results furthermore if $\alpha_1 \geq 0$.

  • 1
    $\begingroup$ How do you obtain "$2\beta_1^2 = \alpha_1^2$" and, later, that $\beta_1=\beta_2$? Although that's correct, it does not immediately follow -- and I suspect that's why this question is being asked. $\endgroup$ – whuber May 5 at 22:35
  • $\begingroup$ Because $X_1 = X_2$. Place comments above. $\endgroup$ – Rafael Valero May 6 at 11:37
  • 1
    $\begingroup$ That's too glib. The equality of the $X_i$ does not directly imply the equality of their coefficients. The RSS is minimized for all $\beta_1,\beta_2$ satisfying $\beta_1+\beta_2=\alpha_1.$ You have to demonstrate that among this space of solutions, the one that minimizes the ridge penalty is $\beta_1=\beta_2=\alpha_1/2.$ $\endgroup$ – whuber May 6 at 11:54
  • 1
    $\begingroup$ This is an interesting attempt, and you may have the kernel of a good answer here, but I agree with whuber that it is presently insufficiently developed to demonstrate what you are trying to demonstrate. In particular, the parameter equations you give would not generally hold in the solution-space for the two models. $\endgroup$ – Ben May 7 at 6:23
  • $\begingroup$ In the first equation, why is it the RSS for both models are the same and $\beta_3^2 = \alpha_3^2$? $\endgroup$ – user321627 May 7 at 10:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.