# From a computational perspective, how does the lasso regression shrink coefficients to 0?

I understand the analytic proof that lasso regularisation tends to shrink coefficients to zero.

However, from a practical standpoint, most of those methods are combined with gradient optimisation (like SGD). For this reason, gradient of the loss w.r.t. each parameter is $$\lambda\texttt{sign}(w_i)$$, where $$\lambda$$ is the regularisation coefficient.

Combined with the learning rate, this parameter is $$\alpha\lambda\texttt{sign}(w_i)$$, where $$\alpha$$ is a nonzero learning rate.

So it can be seen, that unless the value of the parameter $$w_i$$ is already zero, the gradient w.r.t. the regularisation coefficient is either $$\alpha\lambda$$ or $$-\alpha\lambda$$.

With this said, how is it possible that many packages (like glmnet) produce coefficients that are strictly zero? Shouldn't the value of the coefficient "jump" around 0, with a magnitude not larger than $$\alpha\lambda$$? For example, let the value of the parameter $$w_p$$ be $$0.1\alpha\lambda$$ - then after a single optimisation step, the value of the parameter will change to $$-0.9\alpha\lambda$$, if only regularisation is concerned.

For this reason I have concluded that the value actually becoming zero is a result of an interaction of the regularisation and normal MSE optimisation, but I can't grasp the intuition of this interaciton. Could you please provide a simple intuition of how this interaction makes it possible for the coefficient to be 0 from a computational standpoint?

The problem you point out is one reason that starting from a high penalty and computing the whole regularisation path is about as fast as computing $$\hat\beta$$ for a single value of the penalty.