# Solving Lasso with Linear Algebra

I want to find the parameters $$\beta$$ of a linear model using Lasso. When using OLS or Ridge Regression, the parameters of the linear model can be found using linear algebra.

I tried to do the same for Lasso as follows. I would like to know if my approach is correct. The quantity to minimize is:

$$(\mathrm{y} - \mathrm{X}\beta)^T \cdot (\mathrm{y} - \mathrm{X}\beta) + \lambda | \beta| =$$

$$\mathrm{y^T} \mathrm{y} + \beta^T \mathrm{X^T} \mathrm{X} \beta - 2 \beta^T \mathrm{X^T} \mathrm{y} + \lambda |\beta|$$

In order to find the $$\beta$$ that minimizes this quantity, the derivative and set it to 0:

$$\dfrac{\delta \left(\mathrm{y^T} \mathrm{y} + \beta^T \mathrm{X^T} \mathrm{X} \beta - 2 \beta^T \mathrm{X^T} \mathrm{y} + \lambda |\beta| \right)}{\delta \beta} = 0$$

Each component should result in:

$$\frac{\mathrm{y^T} \mathrm{y}}{\delta \beta} = 0$$

$$\frac{\beta^T \mathrm{X^T} \mathrm{X} \beta }{\delta \beta} = 2\mathrm{X^T} \mathrm{X} \beta$$

$$\frac{2 \beta^T \mathrm{X^T} \mathrm{y}}{\delta \beta} = 2 \mathrm{X^T} \mathrm{y}$$

$$\frac{\lambda|\beta|}{\delta \beta} = \lambda I \ \ \ \ \ \text{is this correct?}$$

Putting all these terms together:

$$- 2 \mathrm{X^T} \mathrm{y} + 2\mathrm{X^T} \mathrm{X} \beta + \lambda I = 0$$

Thus:

$$\beta = (2 \mathrm{X^T} \mathrm{X})^{-1} (2 \mathrm{X^T}\mathrm{y} - \lambda I)$$

Is my approach correct?

• Imagine the "V" shaped curve $\beta_1 \mapsto |\beta_1|$ which returns the absolute value of the one-dimensional argument $\beta_1 \in \mathbb{R}$. What's the "slope" when $\beta_1<0$, when $\beta_1=0$, or when $\beta_1>0$? May 8, 2021 at 17:16
• @user257566 It should not be differentiable when $\beta_1 = 0$ (discontinuity), while the derivative should be $-\lambda$ for $\beta < 0$ and $\lambda$ for $\beta > 0$. Is this correct? May 8, 2021 at 17:24
• What does your approach yield when everything is a scalar and linear algebra doesn't get in the way? Does it work?
– whuber
May 8, 2021 at 17:26
• @maurock yes, that's right. What does that say about the step where you wrote "Is this correct?"? May 8, 2021 at 21:16
• @user257566 It is not, and this means that probably the optimization is not as straightforward as it is for OLS and Ridge regression. I guess it is not possible to solve it in closed-form using linear algebra. May 9, 2021 at 9:35