# In variable selection, Lasso over-penalizes coefficients. How to prevent?

I was doing some simulation on the Lasso. Particularly, I set p=200 variables, where only the first 5 have non-zero coefficients. I generated a training sample of size n=100. Whatever I do to tune the hyper parameter lambda, it is hard to find a good lambda that do well in both variable selection (only the first 5 variables have nonzero coefficients) and prediction (low prediction error). The reason I observe is that we need to reach a certain value of lambda to leave only 5 nonzero coefficients, however, the estimated 5 coefficients become very small and almost have no effect due to the penalization by the large lambda.

Is there a way that we can manipulate the data to make Lasso work well in both variable selection and prediction?

P.S. I know doing an extra adaptive Lasso step may help a little bit, but is there any way that we can solve this by manipulating the data (transformations) only?

• Could you do elastic net and add in some ridge regression? – Dave May 24 at 21:47
• Hi @Dave, that's one option, but I was exploring the properties of the Lasso only. In theory, it works well, however the issue is there in practice. – kaixu May 24 at 21:58
• I would argue that precisely in practice, variable selection is a non-issue. You will never have a situation where you have five predictors with nonzero coefficients and 195 predictors with coefficients that are exactly zero. (Why would you have them in your pool of predictors in the first place if most of them were irrelevant?) So-called "tapering effect sizes" are far more likely. And then it comes down to the bias-variance tradeoff, which the lasso is very good at. – Stephan Kolassa May 25 at 7:49
• "is there any way that we can solve this" - yes, get more data – Hong Ooi May 25 at 12:11
• We use $\ell_1$ as a proxy for the actual norm we really wanted, the non-convex $\ell_0$. There's a trade-off attached – Firebug May 25 at 14:29