2
$\begingroup$

I am working through the ISLR book have made a solid effort at answering Question 5 (Chapter 6), but for some reason I am having some real difficulty wrapping my head around the final step. I really hope someone can help as i've been on this question for hours now!

For a tl;dr please refer to the section titled "The part I'm stuck on:"

 

The question:

It is well-known that ridge regression tends to give similar coefficient values to correlated variables, whereas the lasso may give quite different coefficient values to correlated variables. We will now explore this property in a very simple setting.

Suppose that:

  • $n = 2$
  • $p = 2$
  • $x_{11} = x_{12}$
  • $x_{21} = x_{22}$

Furthermore, suppose that $y_1 + y_2 = 0$ and $x_{11} + x_{21} = 0$ and $x_{11} + x_{22} = 0$, so that the estimate for the intercept in a least squares, ridge regression, or lasso model is zero: $\hat{\beta_0} = 0$.

  • (a) Write out the ridge regression optimization problem in this setting

  • (b) Argue that the ridge coefficient estimates satisfy $\hat{\beta_1} =\hat{\beta_2}$

 

The part I'm stuck on:

I'm stuck on the final stage of part (b). I have the following:

$$\hat{\beta_1} = \frac{2 y_1 x_{11} - 2 x_{11}^2 \hat{\beta_2}}{\lambda + 2x_{11}^2} \\ \hat{\beta_2} = \frac{2 y_1 x_{11} - 2 x_{11}^2 \hat{\beta_1}}{\lambda + 2x_{11}^2}$$

I have looked around for other people attempting this question, and they arrive at the same stage (the answer here can be simplified to mine), then they all say:

Symmetry in these expressions suggests that $\hat{\beta_1} = \hat{\beta_2}$

I don't understand why the equations above show equality. Can someone please help my wrap my head around what is presumably a very simple step?

In my mind, we can simplify this even more to:

$$\beta_1 = C + K \beta_2 \\ \beta_2 = C + K \beta_1$$

In which case... I definitely don't see why they are equal? I feel like i'm being really stupid here! I give my full answer below in case mistakes were made earlier in the question.

 

My FULL answer:

Answer to (a):

We have $X = \begin{bmatrix} x_{11} & x_{12} \\ x_{21} & x_{22} \end{bmatrix} = \begin{bmatrix} x_{11} & x_{11} \\ x_{22} & x_{22} \end{bmatrix}$, so the $p = 2$ predictors ($x_1$ & $x_2$) are perfectly correlated.

We know that the ridge coefficient estimates $\hat{\beta}_{\lambda}^R$ are the values that minimize:

$$\sum_{i = 1}^{n} \left( y_i - \beta_0 - \sum_{j = 1}^{p} \beta_j x_{ij} \right)^2 + \lambda \sum_{j = 1}^{p} \beta_j^2$$

Plugging in the specific example, this means for each value of $\lambda$, ridge optimization will select $\hat{\beta}_{\lambda}^R = \begin{pmatrix} \hat{\beta_1} \\ \hat{\beta_2} \end{pmatrix}$ that minimizes the quantity:

$$\sum_{i = 1}^{2} \left( y_i - \beta_0 - \sum_{j = 1}^{2} \beta_j x_{ij} \right)^2 + \lambda \sum_{j = 1}^{2} \beta_j^2 \\ = ( y_1 - \beta_0 - \beta_1 x_{11} - \beta_2 x_{12})^2 + ( y_2 - \beta_0 - \beta_1 x_{21} - \beta_2 x_{22})^2 + \lambda (\beta_1^2 + \beta_2^2)$$

 

Answer to (b):

Let $f(\hat{\beta_1}, \hat{\beta_2}) = ( y_1 - \hat{\beta_0} - \hat{\beta_1} x_{11} - \hat{\beta_2} x_{12})^2 + ( y_2 - \hat{\beta_0} - \hat{\beta_1} x_{21} - \hat{\beta_2} x_{22})^2 + \lambda \left( \hat{\beta_1}^2 + \hat{\beta_2}^2 \right)$.

We therefore have:

$$\begin{align*} f(\hat{\beta_1}, \hat{\beta_2}) & = ( y_1 - \hat{\beta_0} - \hat{\beta_1} x_{11} - \hat{\beta_2} x_{12})^2 + ( y_2 - \hat{\beta_0} - \hat{\beta_1} x_{21} - \hat{\beta_2} x_{22})^2 + \lambda \left( \hat{\beta_1}^2 + \hat{\beta_2}^2 \right) \\ & = ( y_1 - \hat{\beta_1} x_{11} - \hat{\beta_2} x_{12})^2 + ( y_2 - \hat{\beta_1} x_{21} - \hat{\beta_2} x_{22})^2 + \lambda \left( \hat{\beta_1}^2 + \hat{\beta_2}^2 \right) && \text{(since } \hat{\beta_0} = 0 \text{)} \\ & = ( y_1 - \hat{\beta_1} x_{11} - \hat{\beta_2} x_{11})^2 + ( -y_1 + \hat{\beta_1} x_{11} + \hat{\beta_2} x_{11})^2 + \lambda \left( \hat{\beta_1}^2 + \hat{\beta_2}^2 \right) && \text{(since } x_{11} = x_{12} = - x_{21} = -x_{22}, \,\,\, y_2 = -y_1 \text{)} \\ & = ( y_1 - \hat{\beta_1} x_{11} - \hat{\beta_2} x_{11})^2 + (-1)^2(y_1 - \hat{\beta_1} x_{11} - \hat{\beta_2} x_{11})^2 + \lambda \left( \hat{\beta_1}^2 + \hat{\beta_2}^2 \right) \\ & = 2( y_1 - \hat{\beta_1} x_{11} - \hat{\beta_2} x_{11})^2 + \lambda \left( \hat{\beta_1}^2 + \hat{\beta_2}^2 \right) \\ & = 2(y_1^2 - 2 y_1 x_{11} \hat{\beta_1} - 2 y_1 x_{11} \hat{\beta_2} + 2 x_{11}^2 \hat{\beta_1} \hat{\beta_2} + x_{11}^2 \hat{\beta_1}^2 + x_{11}^2 \hat{\beta_2}^2) + \lambda \left( \hat{\beta_1}^2 + \hat{\beta_2}^2 \right) \\ & = 2y_1^2 - 4 y_1 x_{11} \hat{\beta_1} - 4 y_1 x_{11} \hat{\beta_2} + 4 x_{11}^2 \hat{\beta_1} \hat{\beta_2} + 2 x_{11}^2 \hat{\beta_1}^2 + 2 x_{11}^2 \hat{\beta_2}^2 + \lambda \hat{\beta_1}^2 + \lambda \hat{\beta_2}^2 \\ \end{align*}$$

To find the $\hat{\beta_1}$ and $\hat{\beta_2}$ that minimize the above function, we partially differentiate w.r.t $\hat{\beta_1}$ & $\hat{\beta_2}$ and set these equal to zero:

$$\frac{\partial f(\hat{\beta_1}, \hat{\beta_2})}{\partial \hat{\beta_1}} = -4 y_1 x_{11} + 4 x_{11}^2 \hat{\beta_2} + 4 x_{11}^2 \hat{\beta_1} + 2 \lambda \hat{\beta_1} = 0 \\ \begin{align*} & \implies \hat{\beta_1}(\lambda + 2x_{11}^2) = 2 y_1 x_{11} - 2 x_{11}^2 \hat{\beta_2} \\ & \implies \hat{\beta_1} = \frac{2 y_1 x_{11} - 2 x_{11}^2 \hat{\beta_2}}{\lambda + 2x_{11}^2} \end{align*}$$

$$\frac{\partial f(\hat{\beta_1}, \hat{\beta_2})}{\partial \hat{\beta_2}} = -4 y_1 x_{11} + 4 x_{11}^2 \hat{\beta_1} + 4 x_{11}^2 \hat{\beta_2} + 2 \lambda \hat{\beta_2} = 0 \\ \begin{align*} & \implies \hat{\beta_2}(\lambda + 2x_{11}^2) = 2 y_1 x_{11} - 2 x_{11}^2 \hat{\beta_1} \\ & \implies \hat{\beta_2} = \frac{2 y_1 x_{11} - 2 x_{11}^2 \hat{\beta_1}}{\lambda + 2x_{11}^2} \end{align*}$$

Here is where I get stuck.

$\endgroup$

1 Answer 1

3
$\begingroup$

I agree with your simplification: \begin{align*} \beta_1&=C+K\beta_2\\ \beta_2&=C+K\beta_1\\ \\ \beta_1&=C+K(C+K\beta_1)\\ \beta_1\big(1-K^2\big)&=C(1+K)\\ \beta_1&=\frac{C(1+K)}{1-K^2}. \end{align*} You might be able to simplify to $$\beta_1=\frac{C}{1-K}$$ if $K\not=-1.$ But here's the thing: if you perform the same operations for $\beta_2,$ you get the same result. Hence, they are equal.

$\endgroup$
3
  • $\begingroup$ Damn, I was along the right lines but just couldn't put the last bits together! I tried to give you a +1 but i don't have the rep count for it. Thanks for the help though Adrian $\endgroup$ Apr 20, 2020 at 17:08
  • $\begingroup$ You're very welcome! If you can't upvote, you should at least be able to accept as the answer, if you think it is the answer, of course. $\endgroup$ Apr 20, 2020 at 17:10
  • $\begingroup$ Done it - yep I agree with the answer :) thanks again. $\endgroup$ Apr 20, 2020 at 17:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.