Lasso Regression I'm a beginner and hope someone could at least point me a direction on how to write out the gradient for Lasso Regression, thank you so much!
$J_{\beta_0,...\beta_m} = \frac{1}{2m}\sum_{i=1}^{m} (y_i - \beta_0 - \sum_{j=1}^{m}\beta_j X_j)^2 + \lambda\sum_{i=1}^{m}|\beta_i|$
@Firebug, is this the way to calculate the gradient for Lasso Regression?

 A: I'll give you hints towards the solution. We have your loss function $\mathcal L$:
$$\mathcal L(\boldsymbol\beta,\mathbf y,\mathbf X) = \frac{1}{2m}\sum_{i=1}^{m} (y_i - \beta_0 - \sum_{j=1}^{p}\beta_j X_{ji})^2 + \lambda\sum_{j=1}^{p}|\beta_j|$$
For context, gradient descent is a first-order optimization method for differentiable objectives.
Under gradient descent, we have:
$$\boldsymbol \beta_{n+1} := \boldsymbol \beta_{n} - \eta \nabla_\boldsymbol \beta \mathcal L$$
So we just need the gradient $\nabla_\boldsymbol \beta \mathcal L$ for every iteration.
That gradient is simply $\frac{\partial \mathcal L}{\partial \beta _k} \forall k \in [0,p]$.
For $\beta_k$ we can write generically:
$$\frac{\partial \mathcal L}{\partial \beta _k} = \frac{1}{2m}\sum_{i=1}^{m} \frac{\partial}{\partial \beta _k}(y_i - \beta_0 - \sum_{j=1}^{p}\beta_j X_{ji})^2 + \lambda \sum_{j=1}^{p}\frac{\partial}{\partial \beta _k}|\beta_j|$$
We can see that in the first term we have, using the chain rule:
$$\sum_{i=1}^{m} \frac{\partial}{\partial \beta _k}(y_i - \beta_0 - \sum_{j=1}^{p}\beta_j X_{ji})^2\\
=
\sum_{i=1}^{m} 2(y_i - \beta_0 - \sum_{j=1}^{p}\beta_j X_j)\frac{\partial}{\partial \beta _k}(y_i - \beta_0 - \sum_{j=1}^{p}\beta_j X_{ji})\\
=\begin{cases}
\begin{align}
&-\sum_{i=1}^{m} 2(y_i - \beta_0 - \sum_{j=1}^{p}\beta_j X_{ji}), &\text{if}\quad &k = 0\\
&-\sum_{i=1}^{m} 2(y_i - \beta_0 - \sum_{j=1}^{p}\beta_j X_{ji})X_{ki}, &\text{if}\quad &k \neq 0
\end{align}
\end{cases}
$$
The second term is easier to develop, since:
$$\sum_{j=1}^{p}\frac{\partial}{\partial \beta _k}|\beta_j|\\
=
\begin{cases}
0, &\text{if}\quad &k = 0\\
\frac{\partial |\beta_k|}{\partial \beta_k}, &\text{if}\quad &k \neq 0
\end{cases} $$
