2
$\begingroup$

I want to find the parameters $\beta$ of a linear model using Lasso. When using OLS or Ridge Regression, the parameters of the linear model can be found using linear algebra.

I tried to do the same for Lasso as follows. I would like to know if my approach is correct. The quantity to minimize is:

$$ (\mathrm{y} - \mathrm{X}\beta)^T \cdot (\mathrm{y} - \mathrm{X}\beta) + \lambda | \beta| = $$

$$ \mathrm{y^T} \mathrm{y} + \beta^T \mathrm{X^T} \mathrm{X} \beta - 2 \beta^T \mathrm{X^T} \mathrm{y} + \lambda |\beta| $$

In order to find the $\beta$ that minimizes this quantity, the derivative and set it to 0:

$$ \dfrac{\delta \left(\mathrm{y^T} \mathrm{y} + \beta^T \mathrm{X^T} \mathrm{X} \beta - 2 \beta^T \mathrm{X^T} \mathrm{y} + \lambda |\beta| \right)}{\delta \beta} = 0 $$

Each component should result in:

$$ \frac{\mathrm{y^T} \mathrm{y}}{\delta \beta} = 0 $$

$$ \frac{\beta^T \mathrm{X^T} \mathrm{X} \beta }{\delta \beta} = 2\mathrm{X^T} \mathrm{X} \beta $$

$$ \frac{2 \beta^T \mathrm{X^T} \mathrm{y}}{\delta \beta} = 2 \mathrm{X^T} \mathrm{y} $$

$$ \frac{\lambda|\beta|}{\delta \beta} = \lambda I \ \ \ \ \ \text{is this correct?} $$

Putting all these terms together:

$$ - 2 \mathrm{X^T} \mathrm{y} + 2\mathrm{X^T} \mathrm{X} \beta + \lambda I = 0 $$

Thus:

$$ \beta = (2 \mathrm{X^T} \mathrm{X})^{-1} (2 \mathrm{X^T}\mathrm{y} - \lambda I) $$

Is my approach correct?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Imagine the "V" shaped curve $\beta_1 \mapsto |\beta_1|$ which returns the absolute value of the one-dimensional argument $\beta_1 \in \mathbb{R}$. What's the "slope" when $\beta_1<0$, when $\beta_1=0$, or when $\beta_1>0$? $\endgroup$
    – user257566
    May 8, 2021 at 17:16
  • $\begingroup$ @user257566 It should not be differentiable when $\beta_1 = 0$ (discontinuity), while the derivative should be $-\lambda$ for $\beta < 0$ and $\lambda$ for $\beta > 0$. Is this correct? $\endgroup$
    – maurock
    May 8, 2021 at 17:24
  • $\begingroup$ What does your approach yield when everything is a scalar and linear algebra doesn't get in the way? Does it work? $\endgroup$
    – whuber ♦
    May 8, 2021 at 17:26
  • $\begingroup$ @maurock yes, that's right. What does that say about the step where you wrote "Is this correct?"? $\endgroup$
    – user257566
    May 8, 2021 at 21:16
  • 1
    $\begingroup$ @user257566 It is not, and this means that probably the optimization is not as straightforward as it is for OLS and Ridge regression. I guess it is not possible to solve it in closed-form using linear algebra. $\endgroup$
    – maurock
    May 9, 2021 at 9:35

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.