I'm a beginner and hope someone could at least point me a direction on how to write out the gradient for Lasso Regression, thank you so much!
$J_{\beta_0,...\beta_m} = \frac{1}{2m}\sum_{i=1}^{m} (y_i - \beta_0 - \sum_{j=1}^{m}\beta_j X_j)^2 + \lambda\sum_{i=1}^{m}|\beta_i|$
@Firebug, is this the way to calculate the gradient for Lasso Regression?
I'll give you hints towards the solution. We have your loss function $\mathcal L$:
$$\mathcal L(\boldsymbol\beta,\mathbf y,\mathbf X) = \frac{1}{2m}\sum_{i=1}^{m} (y_i - \beta_0 - \sum_{j=1}^{p}\beta_j X_{ji})^2 + \lambda\sum_{j=1}^{p}|\beta_j|$$
For context, gradient descent is a first-order optimization method for differentiable objectives. Under gradient descent, we have:
$$\boldsymbol \beta_{n+1} := \boldsymbol \beta_{n} - \eta \nabla_\boldsymbol \beta \mathcal L$$
So we just need the gradient $\nabla_\boldsymbol \beta \mathcal L$ for every iteration.
That gradient is simply $\frac{\partial \mathcal L}{\partial \beta _k} \forall k \in [0,p]$.
For $\beta_k$ we can write generically:
$$\frac{\partial \mathcal L}{\partial \beta _k} = \frac{1}{2m}\sum_{i=1}^{m} \frac{\partial}{\partial \beta _k}(y_i - \beta_0 - \sum_{j=1}^{p}\beta_j X_{ji})^2 + \lambda \sum_{j=1}^{p}\frac{\partial}{\partial \beta _k}|\beta_j|$$
We can see that in the first term we have, using the chain rule:
$$\sum_{i=1}^{m} \frac{\partial}{\partial \beta _k}(y_i - \beta_0 - \sum_{j=1}^{p}\beta_j X_{ji})^2\\ = \sum_{i=1}^{m} 2(y_i - \beta_0 - \sum_{j=1}^{p}\beta_j X_j)\frac{\partial}{\partial \beta _k}(y_i - \beta_0 - \sum_{j=1}^{p}\beta_j X_{ji})\\ =\begin{cases} \begin{align} &-\sum_{i=1}^{m} 2(y_i - \beta_0 - \sum_{j=1}^{p}\beta_j X_{ji}), &\text{if}\quad &k = 0\\ &-\sum_{i=1}^{m} 2(y_i - \beta_0 - \sum_{j=1}^{p}\beta_j X_{ji})X_{ki}, &\text{if}\quad &k \neq 0 \end{align} \end{cases} $$
The second term is easier to develop, since:
$$\sum_{j=1}^{p}\frac{\partial}{\partial \beta _k}|\beta_j|\\ = \begin{cases} 0, &\text{if}\quad &k = 0\\ \frac{\partial |\beta_k|}{\partial \beta_k}, &\text{if}\quad &k \neq 0 \end{cases} $$
