I have a problem regarding the tuning parameter $\lambda$ in the LASSO or group LASSO.

Suppose I want to find a matrix $\mathbf{A} = [\mathbf{a}_1,...,\mathbf{a}_n]\in\cal{C}^{m\times n}$ that minimizes the distance to some known matrix $\mathbf{A}_0\in\cal{C}^{m\times n}$, with sparsity constraint on the columns: \begin{equation} \underset{\mathbf A}{\mathrm{min}}||\mathbf{A}-\mathbf{A}_0||_F^2 + \lambda\sum_{i=1}^{n}||\mathbf{a}_i||_2, \end{equation} then my question is:

Is there some approach that can find the smallest $\lambda$ that achieves a certain level of sparsity? For example, if I want 50% of the columns in $\mathbf{A}$ to be zeros, how should I determine $\lambda$ (error is of no concern at this point)?

One simple way would be to initialise $\lambda$ with some proper interval, and then check sparsity after solving the LASSO problem. We can do this repeatedly to narrow down the interval until certain sparsity is achieved. Is there any smarter or more rigorous way to solve this?

  • $\begingroup$ I had to do something similar a few years ago. My approach was to concatenate the columns of $A$ into a big vector and then do group lasso on $X\beta$, where $X$ was a blog diagonal matrix. I can attempt to provide a full answer if you like. $\endgroup$ – Demetri Pananos Jul 20 at 15:06

How the lasso solution is generated, is that the variables are included in to the model one by one, untill you will get solution for all possible lambdas. So you don't have to do the narrowing down the intervals, you can just fit the whole lasso and then find the lambda that goes with the desired sparsity level. Or even better, you might stop fitting the lasso, once the desired sparsity is reached. For example glmnet package in R can do it with the parameter dfmax.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.