Can someone please explain me how the optimization of 6.12 leads to 6.14 and the optimization of 6.13 leads to 6.15?

enter image description here


2 Answers 2


For the first equation, it's the result of zero gradient; $$ \begin{aligned} S &= \sum_{j=1}^p (y_j-\beta_j)^2 +\lambda\sum_{j=1}^p\beta_j^2\\ \end{aligned} $$ at extrema, $$ \begin{aligned} \frac{\partial S}{\partial \beta_j} &=0\\ -2(y_j -\beta_j) +2\lambda\beta_j &= 0\\ \beta_j &= \frac{y_j}{1+\lambda}. \end{aligned} $$

I think you should be able to derive the other expression using the same technique shown above and use the fact that $$ \vert \beta_j \vert = \begin{cases} \beta_j \ \text{if} \ \beta_j > 0\\ -\beta_j \ \text{if} \ \beta_j < 0\end{cases}. $$


The main technique has two steps: 1) take derivative w.r.t $\beta_j$ and 2) set it to $0$. I will outline the steps to help go from 6.12 to 6.14 and leave the other one.

Step 1): Combining the summation and opening the quadratic expression of 6.12 gives $$ \sum_{j=1}^p y_j^2 - 2y_j \times \beta_j + (1 + \lambda) \beta_j^2.$$ Now we take partial derivative of this entire summation w.r.t $\beta_j$ and get $$ \frac{\partial}{\partial \beta_j} = -2 y_j + 2(1+\lambda) \beta_j. $$ Note the other $j'\neq j$ do not appear since they do no affect $\beta_j$.

Step 2): We set the partial derivative to $0$ to find $\hat{\beta}_j^R$. Normally, we need to check the second order derivative (make sure it is positive) so that we know have found a minimizer. But we don't have to here because the quadratic form in $\beta_j$ is convex in $\beta_j$.

Barring some additional careful checking of the conditions on $y_j$, the Lasso regression (from 6.13 to 6.15) is very similar.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.