I am working through the ISLR book have made a solid effort at answering Question 5 (Chapter 6), but for some reason I am having some real difficulty wrapping my head around the final step. I really hope someone can help as i've been on this question for hours now!
For a tl;dr please refer to the section titled "The part I'm stuck on:"
The question:
It is well-known that ridge regression tends to give similar coefficient values to correlated variables, whereas the lasso may give quite different coefficient values to correlated variables. We will now explore this property in a very simple setting.
Suppose that:
- $n = 2$
- $p = 2$
- $x_{11} = x_{12}$
- $x_{21} = x_{22}$
Furthermore, suppose that $y_1 + y_2 = 0$ and $x_{11} + x_{21} = 0$ and $x_{11} + x_{22} = 0$, so that the estimate for the intercept in a least squares, ridge regression, or lasso model is zero: $\hat{\beta_0} = 0$.
(a) Write out the ridge regression optimization problem in this setting
(b) Argue that the ridge coefficient estimates satisfy $\hat{\beta_1} =\hat{\beta_2}$
The part I'm stuck on:
I'm stuck on the final stage of part (b). I have the following:
$$\hat{\beta_1} = \frac{2 y_1 x_{11} - 2 x_{11}^2 \hat{\beta_2}}{\lambda + 2x_{11}^2} \\ \hat{\beta_2} = \frac{2 y_1 x_{11} - 2 x_{11}^2 \hat{\beta_1}}{\lambda + 2x_{11}^2}$$
I have looked around for other people attempting this question, and they arrive at the same stage (the answer here can be simplified to mine), then they all say:
Symmetry in these expressions suggests that $\hat{\beta_1} = \hat{\beta_2}$
I don't understand why the equations above show equality. Can someone please help my wrap my head around what is presumably a very simple step?
In my mind, we can simplify this even more to:
$$\beta_1 = C + K \beta_2 \\ \beta_2 = C + K \beta_1$$
In which case... I definitely don't see why they are equal? I feel like i'm being really stupid here! I give my full answer below in case mistakes were made earlier in the question.
My FULL answer:
Answer to (a):
We have $X = \begin{bmatrix} x_{11} & x_{12} \\ x_{21} & x_{22} \end{bmatrix} = \begin{bmatrix} x_{11} & x_{11} \\ x_{22} & x_{22} \end{bmatrix}$, so the $p = 2$ predictors ($x_1$ & $x_2$) are perfectly correlated.
We know that the ridge coefficient estimates $\hat{\beta}_{\lambda}^R$ are the values that minimize:
$$\sum_{i = 1}^{n} \left( y_i - \beta_0 - \sum_{j = 1}^{p} \beta_j x_{ij} \right)^2 + \lambda \sum_{j = 1}^{p} \beta_j^2$$
Plugging in the specific example, this means for each value of $\lambda$, ridge optimization will select $\hat{\beta}_{\lambda}^R = \begin{pmatrix} \hat{\beta_1} \\ \hat{\beta_2} \end{pmatrix}$ that minimizes the quantity:
$$\sum_{i = 1}^{2} \left( y_i - \beta_0 - \sum_{j = 1}^{2} \beta_j x_{ij} \right)^2 + \lambda \sum_{j = 1}^{2} \beta_j^2 \\ = ( y_1 - \beta_0 - \beta_1 x_{11} - \beta_2 x_{12})^2 + ( y_2 - \beta_0 - \beta_1 x_{21} - \beta_2 x_{22})^2 + \lambda (\beta_1^2 + \beta_2^2)$$
Answer to (b):
Let $f(\hat{\beta_1}, \hat{\beta_2}) = ( y_1 - \hat{\beta_0} - \hat{\beta_1} x_{11} - \hat{\beta_2} x_{12})^2 + ( y_2 - \hat{\beta_0} - \hat{\beta_1} x_{21} - \hat{\beta_2} x_{22})^2 + \lambda \left( \hat{\beta_1}^2 + \hat{\beta_2}^2 \right)$.
We therefore have:
$$\begin{align*} f(\hat{\beta_1}, \hat{\beta_2}) & = ( y_1 - \hat{\beta_0} - \hat{\beta_1} x_{11} - \hat{\beta_2} x_{12})^2 + ( y_2 - \hat{\beta_0} - \hat{\beta_1} x_{21} - \hat{\beta_2} x_{22})^2 + \lambda \left( \hat{\beta_1}^2 + \hat{\beta_2}^2 \right) \\ & = ( y_1 - \hat{\beta_1} x_{11} - \hat{\beta_2} x_{12})^2 + ( y_2 - \hat{\beta_1} x_{21} - \hat{\beta_2} x_{22})^2 + \lambda \left( \hat{\beta_1}^2 + \hat{\beta_2}^2 \right) && \text{(since } \hat{\beta_0} = 0 \text{)} \\ & = ( y_1 - \hat{\beta_1} x_{11} - \hat{\beta_2} x_{11})^2 + ( -y_1 + \hat{\beta_1} x_{11} + \hat{\beta_2} x_{11})^2 + \lambda \left( \hat{\beta_1}^2 + \hat{\beta_2}^2 \right) && \text{(since } x_{11} = x_{12} = - x_{21} = -x_{22}, \,\,\, y_2 = -y_1 \text{)} \\ & = ( y_1 - \hat{\beta_1} x_{11} - \hat{\beta_2} x_{11})^2 + (-1)^2(y_1 - \hat{\beta_1} x_{11} - \hat{\beta_2} x_{11})^2 + \lambda \left( \hat{\beta_1}^2 + \hat{\beta_2}^2 \right) \\ & = 2( y_1 - \hat{\beta_1} x_{11} - \hat{\beta_2} x_{11})^2 + \lambda \left( \hat{\beta_1}^2 + \hat{\beta_2}^2 \right) \\ & = 2(y_1^2 - 2 y_1 x_{11} \hat{\beta_1} - 2 y_1 x_{11} \hat{\beta_2} + 2 x_{11}^2 \hat{\beta_1} \hat{\beta_2} + x_{11}^2 \hat{\beta_1}^2 + x_{11}^2 \hat{\beta_2}^2) + \lambda \left( \hat{\beta_1}^2 + \hat{\beta_2}^2 \right) \\ & = 2y_1^2 - 4 y_1 x_{11} \hat{\beta_1} - 4 y_1 x_{11} \hat{\beta_2} + 4 x_{11}^2 \hat{\beta_1} \hat{\beta_2} + 2 x_{11}^2 \hat{\beta_1}^2 + 2 x_{11}^2 \hat{\beta_2}^2 + \lambda \hat{\beta_1}^2 + \lambda \hat{\beta_2}^2 \\ \end{align*}$$
To find the $\hat{\beta_1}$ and $\hat{\beta_2}$ that minimize the above function, we partially differentiate w.r.t $\hat{\beta_1}$ & $\hat{\beta_2}$ and set these equal to zero:
$$\frac{\partial f(\hat{\beta_1}, \hat{\beta_2})}{\partial \hat{\beta_1}} = -4 y_1 x_{11} + 4 x_{11}^2 \hat{\beta_2} + 4 x_{11}^2 \hat{\beta_1} + 2 \lambda \hat{\beta_1} = 0 \\ \begin{align*} & \implies \hat{\beta_1}(\lambda + 2x_{11}^2) = 2 y_1 x_{11} - 2 x_{11}^2 \hat{\beta_2} \\ & \implies \hat{\beta_1} = \frac{2 y_1 x_{11} - 2 x_{11}^2 \hat{\beta_2}}{\lambda + 2x_{11}^2} \end{align*}$$
$$\frac{\partial f(\hat{\beta_1}, \hat{\beta_2})}{\partial \hat{\beta_2}} = -4 y_1 x_{11} + 4 x_{11}^2 \hat{\beta_1} + 4 x_{11}^2 \hat{\beta_2} + 2 \lambda \hat{\beta_2} = 0 \\ \begin{align*} & \implies \hat{\beta_2}(\lambda + 2x_{11}^2) = 2 y_1 x_{11} - 2 x_{11}^2 \hat{\beta_1} \\ & \implies \hat{\beta_2} = \frac{2 y_1 x_{11} - 2 x_{11}^2 \hat{\beta_1}}{\lambda + 2x_{11}^2} \end{align*}$$
Here is where I get stuck.
