Timeline for What are disadvantages of using the lasso for variable selection for regression?
Current License: CC BY-SA 2.5
17 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 21, 2019 at 19:50 | comment | added | Tom Wenseleers | LASSO/LAR is by no means the best automatic selection method - there is plenty of other sparse learning techniques, e.g. iterative adaptive ridge (to approximate L0 penalized regression, cf the l0ara package) and L0Learn that massively outperform LASSO in terms of false positive rate AND get better prediction performance. | |
| Sep 27, 2016 at 15:07 | comment | added | Arne Jonas Warnke | It would be nice to give some references to substantiate such (strong) claims. | |
| Dec 15, 2012 at 21:48 | comment | added | Macro | OK, but the assumptions you're referring to are only relevant to statistical inference and the explicitly stated goal in my example is prediction, in which case all that matters is that you're not overfitting the training data-- nothing about orthogonal predictors or anything else. Anyhow, this was all to explain my disagreement with your initial sentence, at Elvis's request. I don't know whether this conversation has changed your rigid stance or whether you meant that statement hyperbolically, but I think I've given plenty of food for thought for future readers- Cheers. | |
| Dec 15, 2012 at 21:34 | comment | added | Peter Flom | OK, you're right but I don't think it's really relevant. Neither backwards NOR lasso (nor any variable selection method) solves all problems. There are things you have to do before you start modelling - and one of them is check for collinearity. I also wouldn't care which variable selection method worked for other data sets that violated the rules of the regression that both methods are meant to apply to. | |
| Dec 15, 2012 at 21:32 | comment | added | Macro | Fair enough, @PeterFlom, but I believe the question was about variable selection and LASSO and stepwise are competing methods for doing that, whereas ridge regression is not. All I'm concluding here is that if you're doing variable selection for prediction then backwards selection is a better idea than LASSO when there's collinearity, giving a counterexample to the very sweeping statement that brought me here. Elvis, I'm not quite sure either but when I have a chance to ponder it, I'll let you know if I come up with anything (you please do the same!! :)). | |
| Dec 15, 2012 at 21:25 | comment | added | Peter Flom | I think it's because backwards will delete some of the collinear terms early in the process. | |
| Dec 15, 2012 at 21:10 | comment | added | Elvis | @PeterFlom Or both: elastic net ? | |
| Dec 15, 2012 at 21:09 | comment | added | Elvis | @Macro Thanks, this is interesting. I do expect Lasso to perform well when most of the $\beta$ are 0, but I don’t get why collinearity is better handled by backward stepwise regression than by lasso. | |
| Dec 15, 2012 at 21:07 | comment | added | Peter Flom | You should certainly investigate collinearity before embarking on any regression. I'd say that if you have a large number of collinear variables you should not use LASSO or Stepwise; you should either solve the collinearity problem (delete variables, get more data, etc) or use a method designed for such problems (e.g. ridge regression) | |
| Dec 15, 2012 at 20:58 | comment | added | Macro | In these simulations I used $n=1000$ and $20\%$ was withheld to evaluate prediction for each model. Under these settings, the LASSO predictions, in terms of MSE, were about 4x less accurate (with MSE increasing from about 1.25 up to about 5 on most runs). If you increase the level of collinearity, the size of the coefficients, or the variance of the predictors, it gets even worse. To make for a slightly less pathological scenario, you can set, say, $50\%$ of the coefficients to zero in the data generation and you'll find the same result (although, the difference is not as dramatic, obviously). | |
| Dec 15, 2012 at 20:22 | comment | added | Macro | @Elvis, I'm no expert on the subject or an advocate for stepwise; I'm only taking issue with the unconditional nature of the statement. But, out of curiosity I did some simple simulations and found that when you have a large number of collinear predictors that all have roughly equal effects, backwards selection does better than LASSO, in terms of out-of-sample prediction. I used $$Y_i = \sum_{j=1}^{100} X_{ij} + \varepsilon_{i}$$ with $ \varepsilon \sim N(0,1)$. The predictors are standard normal with ${\rm cor} (X_{ij},X_{ik})=1/2$ for every pair $(j,k)$. | |
| Dec 15, 2012 at 10:30 | comment | added | probabilityislogic | -1 due to the blanket criticism of stepwise. Its not "just wrong" but has a place as a deterministic model search. You really do have a bee in your bonnet about automatic methods. | |
| Dec 14, 2012 at 20:13 | comment | added | Elvis | @Macro, can you point to situations where stepwise regression is the best method? (as I understand, you already explained that somewhere, just give a pointer) | |
| Dec 14, 2012 at 17:11 | comment | added | Macro | "There is NO reason to do stepwise selection. It's just wrong." - Almost never are incredibly sweeping statements like that, devoid of context, good statistical practice. If anything here is "just wrong", it's the bolded statement above. If your analysis is not emphasizing $p$-values or parameter estimates (e.g. predictive models) then stepwise variable selection may be a sensible thing to do and can ::gasp:: outperform LASSO in some cases. (Peter, I know we've had this convo before - this comment is more directed at a future reader who may only come across this post and not the other). | |
| Oct 30, 2012 at 18:21 | review | Suggested edits | |||
| Oct 30, 2012 at 18:37 | |||||
| Mar 7, 2011 at 20:53 | vote | accept | xuexue | ||
| Mar 7, 2011 at 0:58 | history | answered | Peter Flom | CC BY-SA 2.5 |