3
$\begingroup$

I was doing some simulation on the Lasso. Particularly, I set p=200 variables, where only the first 5 have non-zero coefficients. I generated a training sample of size n=100. Whatever I do to tune the hyper parameter lambda, it is hard to find a good lambda that do well in both variable selection (only the first 5 variables have nonzero coefficients) and prediction (low prediction error). The reason I observe is that we need to reach a certain value of lambda to leave only 5 nonzero coefficients, however, the estimated 5 coefficients become very small and almost have no effect due to the penalization by the large lambda.

Is there a way that we can manipulate the data to make Lasso work well in both variable selection and prediction?

P.S. I know doing an extra adaptive Lasso step may help a little bit, but is there any way that we can solve this by manipulating the data (transformations) only?

| cite | improve this question | |
$\endgroup$
  • 1
    $\begingroup$ Could you do elastic net and add in some ridge regression? $\endgroup$ – Dave May 24 at 21:47
  • $\begingroup$ Hi @Dave, that's one option, but I was exploring the properties of the Lasso only. In theory, it works well, however the issue is there in practice. $\endgroup$ – kaixu May 24 at 21:58
  • 3
    $\begingroup$ I would argue that precisely in practice, variable selection is a non-issue. You will never have a situation where you have five predictors with nonzero coefficients and 195 predictors with coefficients that are exactly zero. (Why would you have them in your pool of predictors in the first place if most of them were irrelevant?) So-called "tapering effect sizes" are far more likely. And then it comes down to the bias-variance tradeoff, which the lasso is very good at. $\endgroup$ – Stephan Kolassa May 25 at 7:49
  • 1
    $\begingroup$ "is there any way that we can solve this" - yes, get more data $\endgroup$ – Hong Ooi May 25 at 12:11
  • $\begingroup$ We use $\ell_1$ as a proxy for the actual norm we really wanted, the non-convex $\ell_0$. There's a trade-off attached $\endgroup$ – Firebug May 25 at 14:29
9
$\begingroup$

You might be asking too much of Lasso. Even in the first simulation, Example 1 in the original paper (available from here), Lasso only chose the "correct" model about 1/4 of the time. And that was with 3 out of 8 predictors having non-zero coefficients, a high signal-to-noise ratio, and more observations (20 per simulation) than predictors.

In Example 3 of that paper, with only 1 non-zero coefficient among 8 predictors and an even higher signal-to-noise ratio than in Example 1 above, Lasso only found 2 or 3 predictors excluded (depending on how the penalization value was chosen), instead of the 7 with true 0 values.

With what you are simulating--only 5 non-zero coefficients out of 200 predictors, and fewer observations than predictors--it would seem to be much harder to get simultaneous strict variable selection (down to the "true" 5 out of 200) and good prediction.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.