0
$\begingroup$

I understand the analytic proof that lasso regularisation tends to shrink coefficients to zero.

However, from a practical standpoint, most of those methods are combined with gradient optimisation (like SGD). For this reason, gradient of the loss w.r.t. each parameter is $\lambda\texttt{sign}(w_i)$, where $\lambda$ is the regularisation coefficient.

Combined with the learning rate, this parameter is $\alpha\lambda\texttt{sign}(w_i)$, where $\alpha$ is a nonzero learning rate.

So it can be seen, that unless the value of the parameter $w_i$ is already zero, the gradient w.r.t. the regularisation coefficient is either $\alpha\lambda$ or $-\alpha\lambda$.

With this said, how is it possible that many packages (like glmnet) produce coefficients that are strictly zero? Shouldn't the value of the coefficient "jump" around 0, with a magnitude not larger than $\alpha\lambda$? For example, let the value of the parameter $w_p$ be $0.1\alpha\lambda$ - then after a single optimisation step, the value of the parameter will change to $-0.9\alpha\lambda$, if only regularisation is concerned.

For this reason I have concluded that the value actually becoming zero is a result of an interaction of the regularisation and normal MSE optimisation, but I can't grasp the intuition of this interaciton. Could you please provide a simple intuition of how this interaction makes it possible for the coefficient to be 0 from a computational standpoint?

$\endgroup$

1 Answer 1

2
$\begingroup$

The coefficients start at zero. That is, the algorithm starts by applying a sufficiently large penalty that all the coefficient estimates are exactly zero. As the penalty is progressively decreased, coefficients start moving away from zero, one at a time.

The problem you point out is one reason that starting from a high penalty and computing the whole regularisation path is about as fast as computing $\hat\beta$ for a single value of the penalty.

(For linear regression lasso it's even more straightforward, since the path followed by the coefficients as the penalty decrease is piecewise linear and doesn't have to be approximated by gradient descent -- the LARS algorithm does it exactly)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.