Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 14 down vote favorite
2

From what I know, using lasso for variable selection handles the problem of correlated inputs. Also, since it is equivalent to Least Angle Regression, it is not slow computationally. However, many people (for example people I know doing bio-statistics) still seem to favour stepwise or stagewise variable selection. Are there any practical disadvantages of using the lasso that makes it unfavourable?

share|improve this question
2  
I don't know where you heard that Lasso handles the problem of collinearity, that's absolutely not true. – Macro Dec 14 '12 at 16:57
Horseshoe prior is better than LASSO for model selection - at least in the sparse model case (where model selection is the most useful). You can find a discussion of these points in this Link. Two of the authors of this paper also got a similar article into the Valencia meetings, Bayesian Statistics 9 "Shrink Globally Act Locally: Sparse Bayesian regularisation and prediction". The Valencia article goes into much more detail on a penalty framework. – probabilityislogic Dec 16 '12 at 13:23

3 Answers

up vote 5 down vote accepted

There is NO reason to do stepwise selection. It's just wrong.

LASSO/LAR are the best automatic methods. But they are automatic methods. They let the analyst not think.

In many analyses, some variables should be in the model REGARDLESS of any measure of significance. Sometimes they are necessary control variables. Other times, finding a small effect can be substantively important.

share|improve this answer
8  
"There is NO reason to do stepwise selection. It's just wrong." - Almost never are incredibly sweeping statements like that, devoid of context, good statistical practice. If anything here is "just wrong", it's the bolded statement above. If your analysis is not emphasizing $p$-values or parameter estimates (e.g. predictive models) then stepwise variable selection may be a sensible thing to do and can ::gasp:: outperform LASSO in some cases. (Peter, I know we've had this convo before - this comment is more directed at a future reader who may only come across this post and not the other). – Macro Dec 14 '12 at 17:11
2  
-1 due to the blanket criticism of stepwise. Its not "just wrong" but has a place as a deterministic model search. You really do have a bee in your bonnet about automatic methods. – probabilityislogic Dec 15 '12 at 10:30
2  
@Elvis, I'm no expert on the subject or an advocate for stepwise; I'm only taking issue with the unconditional nature of the statement. But, out of curiosity I did some simple simulations and found that when you have a large number of collinear predictors that all have roughly equal effects, backwards selection does better than LASSO, in terms of out-of-sample prediction. I used $$Y_i = \sum_{j=1}^{100} X_{ij} + \varepsilon_{i}$$ with $ \varepsilon \sim N(0,1)$. The predictors are standard normal with ${\rm cor} (X_{ij},X_{ik})=1/2$ for every pair $(j,k)$. – Macro Dec 15 '12 at 20:22
1  
In these simulations I used $n=1000$ and $20\%$ was withheld to evaluate prediction for each model. Under these settings, the LASSO predictions, in terms of MSE, were about 4x less accurate (with MSE increasing from about 1.25 up to about 5 on most runs). If you increase the level of collinearity, the size of the coefficients, or the variance of the predictors, it gets even worse. To make for a slightly less pathological scenario, you can set, say, $50\%$ of the coefficients to zero in the data generation and you'll find the same result (although, the difference is not as dramatic, obviously). – Macro Dec 15 '12 at 20:58
1  
You should certainly investigate collinearity before embarking on any regression. I'd say that if you have a large number of collinear variables you should not use LASSO or Stepwise; you should either solve the collinearity problem (delete variables, get more data, etc) or use a method designed for such problems (e.g. ridge regression) – Peter Flom Dec 15 '12 at 21:07
show 7 more comments
up vote 4 down vote

One practical disadvantage of lasso and other regularization techniques is finding the optimal regularization coefficient, lambda. Using cross validation to find this value can be just as expensive as stepwise selection techniques.

share|improve this answer
What do you mean by "expensive"? – mark999 Mar 8 '12 at 10:41
This claim is not really true. If you adopt the "warm start" grid search as in the glmnet method, you can comput the entire grid very quickly. – probabilityislogic Dec 15 '12 at 10:50
@probabilityislogic True, I only read about warm starts after I made the above comment. What do you think of this paper, which indicates warm starts are slower and sometimes less effective than simple cross validation? users.cis.fiu.edu/~lzhen001/activities/KDD2011Program/docs/… – rm999 Dec 17 '12 at 21:59
up vote 1 down vote

One big one is the difficulty of doing hypothesis testing. You can't easily figure out which variables are statistically significant with Lasso. With stepwise regression, you can do hypothesis testing to some degree, if you're careful about your treatment of multiple testing.

share|improve this answer
3  
I'd say that's an advantage, not a disadvantage. It stops you from doing something you probably shouldn't be doing. – Peter Flom Mar 7 '11 at 0:59
@Peter: Why? I'm assuming that you would properly correct for multiple testing, etc. such that the P-values obtained would be valid. – dsimcha Mar 7 '11 at 1:51
3  
there really isn't a way to properly correct for multiple testing in stepwise. See, e.g. Harrell Regression modeling strategies. There's no way to know the right correction – Peter Flom Mar 8 '11 at 1:53

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.