2
$\begingroup$

The below text is from Statistical Learning Page no.225

Consider a case with $n = p$, and $\mathbf{X}$ a diagonal matrix with 1’s on the diagonal and 0’s in all off-diagonal elements. To simplify the problem further, assume also that we are performing regression without an intercept. With these assumptions, the usual least squares problem simplifies to finding $\beta_1,\ldots,\beta_p$ that minimize $$\sum_{j=1}^p(y_j−β_j)^2$$

In this case, the least squares solution is given by $$\hatβ_j = y_j$$

And in this setting, ridge regression amounts to finding $\beta_1,\ldots,\beta_p$ such that $$\sum_{j=1}^p(y_j-β_j)^2+λ \sum_{j=1}^p β_j^2$$

is minimized, and the lasso amounts to finding the coefficients such that $$\sum_{j=1}^p(y_j-β_j)^2+λ \sum_{j=1}^p |β_j|$$

Up to this it is comprehensible to me but I am not able to understand the below text. Can anyone explain how the results shown below were derived ?

is minimized. One can show that in this setting, the ridge regression estimates take the form $$\hatβ^R_j = \frac{y_j}{1 + λ} $$

and the lasso estimates take the form $$\hatβ^L_j =y_j −\frac{λ}{2} \hspace{1cm} if \hspace{.3cm} y_j > \frac{λ}{2}$$ $$\hatβ^L_j =y_j +\frac{λ}{2} \hspace{1cm} if \hspace{.3cm} y_j <−λ/2$$ $$\hatβ^L_j = 0 \hspace{1cm} if \hspace{1cm} |y_j|≤\frac{λ}{2}$$

$\endgroup$

1 Answer 1

5
$\begingroup$

For ridge regression, the problem is to minimize $$r(\underline{\beta})=\sum_{j=1}^{p}\left(y_{j}-\beta_{j}\right) ^{2}+\lambda\sum_{j=1}^{p}\beta_{j}^2=\sum_{j=1}^{p}\left[ \left( y_{j}-\beta_{j}\right) ^{2}+\lambda \beta_{j}^2\right],$$ where $\underline{\beta}=(\beta_1,\beta_2,\ldots,\beta_p)$. Now, this equation is additively separable, $$r(\underline{\beta}) =\sum_{j=1}^{p}r(\beta_{j})$$ so the derivative with respect to $\beta_j$ is $$\frac{\partial}{\partial\beta_j}r(\underline{\beta})=\frac{d}{d\beta_j}r(\beta_{j}).$$ Thus, minimizing with respect to $\underline{\beta}$ is equivalent to $p$ component-wise minimizations with respect to $\beta_{j}$ for $j=1,2,\ldots,p$. So, $$\frac{d}{d\beta_j}r(\beta_{j})=\frac{d}{d\beta_j}\left[\left(y_{j}-\beta_{j}\right) ^{2}+\lambda \beta_{j}^2\right] =\frac{d}{d\beta_j}\left[y_{j}^2-2y_j\beta_{j}+(1+\lambda)\beta_{j}^2\right]=-2y_j+2(1+\lambda)\beta_j.$$ Setting this to zero provides the minimum, $$2(1+\lambda)\hat{\beta}_j^R-2y_j=0\Leftrightarrow\hat{\beta}_j^R=\frac{y_j}{1+\lambda}.$$

Similarly, the LASSO problem minimizes the additively separable function $$l(\underline{\beta})=\sum_{j=1}^{p}\left(y_{j}-\beta_{j}\right) ^{2}+\lambda\sum_{j=1}^{p}\left\vert\beta_{j}\right\vert=\sum_{j=1}^{p}\left[ \left( y_{j}-\beta_{j}\right) ^{2}+\lambda\left\vert \beta_{j}\right\vert \right].$$ Thus, for $j=1,2,\ldots,p$, we must find the derivatives $$\frac{d}{d\beta_j}l(\beta_{j})=\frac{d}{d\beta_j}\left[\left(y_{j}-\beta_{j}\right) ^{2}+\lambda\left\vert \beta_{j}\right\vert \right]=\frac{d}{d\beta_j}\left[y_{j}^2-2y_j\beta_{j}+\beta_{j}^2+\lambda\left\vert \beta_{j}\right\vert\right].$$ Because of the $-\beta_{j}y_{j}$ term in the objective function, we choose $\beta_{j}$ to have the same sign as $y_{j}$ to preserve the formation of the problem.

  1. Suppose that $y_{j}>0$, then for $j=1,2,\ldots,p$, we must minimize $$l(\beta_{j}) =y_{j}^{2}-2y_{j}\beta_{j}+\beta_{j}^{2}+\lambda\beta_{j},$$ since $\left\vert \beta_{j}\right\vert =\beta_{j}$ when $\beta_{j}\geq0$. The derivative is $$l^{\prime}(\beta_{j})=-2y_{j}+2\beta_{j}+\lambda=2\left[\beta_{j}-\left( y_{j}-\frac{\lambda}{2}\right)\right].$$

    a. If $\left\vert y_{j}\right\vert \leq\frac{\lambda}{2}$ then $-\left(y_{j}-\frac{\lambda}{2}\right) >0$ so that $l^{\prime}\left( \beta_{j}\right) >0$ for all $\beta_{j}\geq0$. Thus $l(\beta_{j})$ is strictly increasing for all $\beta_{j}\geq0$ and $\hat{\beta}_{j}^{L}=0$.

    b. If $\left\vert y_{j}\right\vert >\frac{\lambda}{2}$ then $-\left( y_{j}-\frac{\lambda}{2}\right) \leq0$ and setting $l^{\prime}\left( \beta_{j}\right) =0$ gives the solution $$\hat{\beta}_{j}^{L}=y_{j}-\frac{\lambda}{2} \textrm{ if }y_j>\frac{\lambda}{2}.$$

  2. Similarly, for $y_{j}<0$ we must minimize $$l(\beta_{j}) =y_{j}^{2}-2y_{j}\beta_{j}+\beta_{j}^{2}-\lambda\beta_{j},$$ since $\left\vert \beta_{j}\right\vert =-\beta_{j}$ when $\beta _{j}\leq0$. The derivative is $$l^{\prime}\left( \beta_{j}\right) =-2y_{j}+2\beta_{j}-\lambda=2\left[\beta_{j}-\left( y_{j}+\frac{\lambda}{2}\right)\right].$$

    a. If $\left\vert y_{j}\right\vert\leq\frac{\lambda}{2}$ then $-\left( y_{j}+\frac{\lambda}{2}\right) <0$ so that $l^{\prime}\left(\beta_{j}\right) <0$ for all $\beta_{j}\leq0$. Thus $l(\beta_{j})$ is strictly decreasing for all $\beta_{j}\leq0$ and $\hat{\beta}_{j}^{L}=0$.

    b. If $\left\vert y_{j}\right\vert >\frac{\lambda}{2}$ then $-\left( y_{j}+\frac{\lambda}{2}\right) \geq0$ and setting $l^{\prime}\left( \beta_{j}\right) =0$ gives the solution $$\hat{\beta}_{j}^{L}=y_{j}+\frac{\lambda}{2} \textrm{ if }y_j<-\frac{\lambda}{2}.$$

From 1a and 2a, $$\hat{\beta}_{j}^{L}=0 \textrm{ if }\left\vert y_{j}\right\vert\leq\frac{\lambda}{2}.$$

$\endgroup$
5
  • $\begingroup$ thanks a lot for taking out the time and providing the solution. I don't understand one thing i.e.∂∂βjr(β−)=ddβjr(βj).How partial derivative becomes equal to full derivative. (Is it a rule or something else) If its a rule can you provide me a source where i can learn more about this . $\endgroup$
    – learner
    Apr 12, 2016 at 7:47
  • $\begingroup$ That equation follows from the one just before it. Suppose that r(β1,β2)=r(β1)+r(β2)=2β1+3β2. Then ∂/∂β1 r(β1,β2) = d/dβ1 r(β1) = 2. r(β1) is a function of β1 alone - thus the partial derivative is the full derivative. Make sense? $\endgroup$
    – StatGrrl
    Apr 12, 2016 at 8:11
  • $\begingroup$ sorry i didn't get it. $\endgroup$
    – learner
    Apr 12, 2016 at 8:31
  • $\begingroup$ Not sure how else I can explain.From my example above, ∂/∂β2 r(β1,β2) = ∂/∂β2 (2β1 + 3β2) = 3. d/dβ2 r(β2) = d/dβ2 (3β2) = 3. $\endgroup$
    – StatGrrl
    Apr 12, 2016 at 9:03
  • $\begingroup$ See calculus.subwiki.org/wiki/Additively_separable_function $\endgroup$
    – StatGrrl
    Apr 12, 2016 at 9:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.