# Interpreting Special Case for Ridge Regression and the Lasso

The below text is from Statistical Learning Page no.225

Consider a case with $n = p$, and $\mathbf{X}$ a diagonal matrix with 1’s on the diagonal and 0’s in all oﬀ-diagonal elements. To simplify the problem further, assume also that we are performing regression without an intercept. With these assumptions, the usual least squares problem simpliﬁes to ﬁnding $\beta_1,\ldots,\beta_p$ that minimize $$\sum_{j=1}^p(y_j−β_j)^2$$

In this case, the least squares solution is given by $$\hatβ_j = y_j$$

And in this setting, ridge regression amounts to ﬁnding $\beta_1,\ldots,\beta_p$ such that $$\sum_{j=1}^p(y_j-β_j)^2+λ \sum_{j=1}^p β_j^2$$

is minimized, and the lasso amounts to ﬁnding the coeﬃcients such that $$\sum_{j=1}^p(y_j-β_j)^2+λ \sum_{j=1}^p |β_j|$$

Up to this it is comprehensible to me but I am not able to understand the below text. Can anyone explain how the results shown below were derived ?

is minimized. One can show that in this setting, the ridge regression estimates take the form $$\hatβ^R_j = \frac{y_j}{1 + λ}$$

and the lasso estimates take the form $$\hatβ^L_j =y_j −\frac{λ}{2} \hspace{1cm} if \hspace{.3cm} y_j > \frac{λ}{2}$$ $$\hatβ^L_j =y_j +\frac{λ}{2} \hspace{1cm} if \hspace{.3cm} y_j <−λ/2$$ $$\hatβ^L_j = 0 \hspace{1cm} if \hspace{1cm} |y_j|≤\frac{λ}{2}$$

For ridge regression, the problem is to minimize $$r(\underline{\beta})=\sum_{j=1}^{p}\left(y_{j}-\beta_{j}\right) ^{2}+\lambda\sum_{j=1}^{p}\beta_{j}^2=\sum_{j=1}^{p}\left[ \left( y_{j}-\beta_{j}\right) ^{2}+\lambda \beta_{j}^2\right],$$ where $\underline{\beta}=(\beta_1,\beta_2,\ldots,\beta_p)$. Now, this equation is additively separable, $$r(\underline{\beta}) =\sum_{j=1}^{p}r(\beta_{j})$$ so the derivative with respect to $\beta_j$ is $$\frac{\partial}{\partial\beta_j}r(\underline{\beta})=\frac{d}{d\beta_j}r(\beta_{j}).$$ Thus, minimizing with respect to $\underline{\beta}$ is equivalent to $p$ component-wise minimizations with respect to $\beta_{j}$ for $j=1,2,\ldots,p$. So, $$\frac{d}{d\beta_j}r(\beta_{j})=\frac{d}{d\beta_j}\left[\left(y_{j}-\beta_{j}\right) ^{2}+\lambda \beta_{j}^2\right] =\frac{d}{d\beta_j}\left[y_{j}^2-2y_j\beta_{j}+(1+\lambda)\beta_{j}^2\right]=-2y_j+2(1+\lambda)\beta_j.$$ Setting this to zero provides the minimum, $$2(1+\lambda)\hat{\beta}_j^R-2y_j=0\Leftrightarrow\hat{\beta}_j^R=\frac{y_j}{1+\lambda}.$$

Similarly, the LASSO problem minimizes the additively separable function $$l(\underline{\beta})=\sum_{j=1}^{p}\left(y_{j}-\beta_{j}\right) ^{2}+\lambda\sum_{j=1}^{p}\left\vert\beta_{j}\right\vert=\sum_{j=1}^{p}\left[ \left( y_{j}-\beta_{j}\right) ^{2}+\lambda\left\vert \beta_{j}\right\vert \right].$$ Thus, for $j=1,2,\ldots,p$, we must find the derivatives $$\frac{d}{d\beta_j}l(\beta_{j})=\frac{d}{d\beta_j}\left[\left(y_{j}-\beta_{j}\right) ^{2}+\lambda\left\vert \beta_{j}\right\vert \right]=\frac{d}{d\beta_j}\left[y_{j}^2-2y_j\beta_{j}+\beta_{j}^2+\lambda\left\vert \beta_{j}\right\vert\right].$$ Because of the $-\beta_{j}y_{j}$ term in the objective function, we choose $\beta_{j}$ to have the same sign as $y_{j}$ to preserve the formation of the problem.

1. Suppose that $y_{j}>0$, then for $j=1,2,\ldots,p$, we must minimize $$l(\beta_{j}) =y_{j}^{2}-2y_{j}\beta_{j}+\beta_{j}^{2}+\lambda\beta_{j},$$ since $\left\vert \beta_{j}\right\vert =\beta_{j}$ when $\beta_{j}\geq0$. The derivative is $$l^{\prime}(\beta_{j})=-2y_{j}+2\beta_{j}+\lambda=2\left[\beta_{j}-\left( y_{j}-\frac{\lambda}{2}\right)\right].$$

a. If $\left\vert y_{j}\right\vert \leq\frac{\lambda}{2}$ then $-\left(y_{j}-\frac{\lambda}{2}\right) >0$ so that $l^{\prime}\left( \beta_{j}\right) >0$ for all $\beta_{j}\geq0$. Thus $l(\beta_{j})$ is strictly increasing for all $\beta_{j}\geq0$ and $\hat{\beta}_{j}^{L}=0$.

b. If $\left\vert y_{j}\right\vert >\frac{\lambda}{2}$ then $-\left( y_{j}-\frac{\lambda}{2}\right) \leq0$ and setting $l^{\prime}\left( \beta_{j}\right) =0$ gives the solution $$\hat{\beta}_{j}^{L}=y_{j}-\frac{\lambda}{2} \textrm{ if }y_j>\frac{\lambda}{2}.$$

2. Similarly, for $y_{j}<0$ we must minimize $$l(\beta_{j}) =y_{j}^{2}-2y_{j}\beta_{j}+\beta_{j}^{2}-\lambda\beta_{j},$$ since $\left\vert \beta_{j}\right\vert =-\beta_{j}$ when $\beta _{j}\leq0$. The derivative is $$l^{\prime}\left( \beta_{j}\right) =-2y_{j}+2\beta_{j}-\lambda=2\left[\beta_{j}-\left( y_{j}+\frac{\lambda}{2}\right)\right].$$

a. If $\left\vert y_{j}\right\vert\leq\frac{\lambda}{2}$ then $-\left( y_{j}+\frac{\lambda}{2}\right) <0$ so that $l^{\prime}\left(\beta_{j}\right) <0$ for all $\beta_{j}\leq0$. Thus $l(\beta_{j})$ is strictly decreasing for all $\beta_{j}\leq0$ and $\hat{\beta}_{j}^{L}=0$.

b. If $\left\vert y_{j}\right\vert >\frac{\lambda}{2}$ then $-\left( y_{j}+\frac{\lambda}{2}\right) \geq0$ and setting $l^{\prime}\left( \beta_{j}\right) =0$ gives the solution $$\hat{\beta}_{j}^{L}=y_{j}+\frac{\lambda}{2} \textrm{ if }y_j<-\frac{\lambda}{2}.$$

From 1a and 2a, $$\hat{\beta}_{j}^{L}=0 \textrm{ if }\left\vert y_{j}\right\vert\leq\frac{\lambda}{2}.$$

• thanks a lot for taking out the time and providing the solution. I don't understand one thing i.e.∂∂βjr(β−)=ddβjr(βj).How partial derivative becomes equal to full derivative. (Is it a rule or something else) If its a rule can you provide me a source where i can learn more about this . Apr 12, 2016 at 7:47
• That equation follows from the one just before it. Suppose that r(β1,β2)=r(β1)+r(β2)=2β1+3β2. Then ∂/∂β1 r(β1,β2) = d/dβ1 r(β1) = 2. r(β1) is a function of β1 alone - thus the partial derivative is the full derivative. Make sense? Apr 12, 2016 at 8:11
• sorry i didn't get it. Apr 12, 2016 at 8:31
• Not sure how else I can explain.From my example above, ∂/∂β2 r(β1,β2) = ∂/∂β2 (2β1 + 3β2) = 3. d/dβ2 r(β2) = d/dβ2 (3β2) = 3. Apr 12, 2016 at 9:03
• Apr 12, 2016 at 9:07