From what I know, using lasso for variable selection handles the problem of correlated inputs. Also, since it is equivalent to Least Angle Regression, it is not slow computationally. However, many people (for example people I know doing bio-statistics) still seem to favour stepwise or stagewise variable selection. Are there any practical disadvantages of using the lasso that makes it unfavourable?
$\begingroup$
$\endgroup$
13
-
12$\begingroup$ I don't know where you heard that Lasso handles the problem of collinearity, that's absolutely not true. $\endgroup$– MacroCommented Dec 14, 2012 at 16:57
-
4$\begingroup$ Horseshoe prior is better than LASSO for model selection - at least in the sparse model case (where model selection is the most useful). You can find a discussion of these points in this Link. Two of the authors of this paper also got a similar article into the Valencia meetings, Bayesian Statistics 9 "Shrink Globally Act Locally: Sparse Bayesian regularisation and prediction". The Valencia article goes into much more detail on a penalty framework. $\endgroup$– probabilityislogicCommented Dec 16, 2012 at 13:23
-
9$\begingroup$ If you are only interested in prediction, then model selection doesn't help and usually hurts (as opposed to a quadratic penalty = L2 norm = ridge regression with no variable selection). LASSO pays a price in predictive discrimination for trying to do variable selection. $\endgroup$– Frank HarrellCommented Nov 28, 2013 at 14:37
-
4$\begingroup$ Tossing a coin to make an arbitrary decision often reveals that you do actually care about the outcome. Any method that offers to make decisions for you about selection of predictors often makes it plain that you do have ideas about which predictors belong more naturally in the model, ideas that you don't want ignored. LASSO can work like that. $\endgroup$– Nick CoxCommented Nov 28, 2013 at 15:11
-
5$\begingroup$ I second @Nick: "no theory available to guide model selection" is hardly ever realistic. Common sense is theory. $\endgroup$– Scortchi ♦Commented Nov 28, 2013 at 19:08
|
Show 8 more comments
11 Answers
$\begingroup$
$\endgroup$
14
There is NO reason to do stepwise selection. It's just wrong.
LASSO/LAR are the best automatic methods. But they are automatic methods. They let the analyst not think.
In many analyses, some variables should be in the model REGARDLESS of any measure of significance. Sometimes they are necessary control variables. Other times, finding a small effect can be substantively important.
answered Mar 7, 2011 at 0:58
-
71$\begingroup$ "There is NO reason to do stepwise selection. It's just wrong." - Almost never are incredibly sweeping statements like that, devoid of context, good statistical practice. If anything here is "just wrong", it's the bolded statement above. If your analysis is not emphasizing $p$-values or parameter estimates (e.g. predictive models) then stepwise variable selection may be a sensible thing to do and can ::gasp:: outperform LASSO in some cases. (Peter, I know we've had this convo before - this comment is more directed at a future reader who may only come across this post and not the other). $\endgroup$– MacroCommented Dec 14, 2012 at 17:11
-
5$\begingroup$ -1 due to the blanket criticism of stepwise. Its not "just wrong" but has a place as a deterministic model search. You really do have a bee in your bonnet about automatic methods. $\endgroup$ Commented Dec 15, 2012 at 10:30
-
8$\begingroup$ @Elvis, I'm no expert on the subject or an advocate for stepwise; I'm only taking issue with the unconditional nature of the statement. But, out of curiosity I did some simple simulations and found that when you have a large number of collinear predictors that all have roughly equal effects, backwards selection does better than LASSO, in terms of out-of-sample prediction. I used $$Y_i = \sum_{j=1}^{100} X_{ij} + \varepsilon_{i}$$ with $ \varepsilon \sim N(0,1)$. The predictors are standard normal with ${\rm cor} (X_{ij},X_{ik})=1/2$ for every pair $(j,k)$. $\endgroup$– MacroCommented Dec 15, 2012 at 20:22
-
12$\begingroup$ You should certainly investigate collinearity before embarking on any regression. I'd say that if you have a large number of collinear variables you should not use LASSO or Stepwise; you should either solve the collinearity problem (delete variables, get more data, etc) or use a method designed for such problems (e.g. ridge regression) $\endgroup$ Commented Dec 15, 2012 at 21:07
-
5$\begingroup$ OK, you're right but I don't think it's really relevant. Neither backwards NOR lasso (nor any variable selection method) solves all problems. There are things you have to do before you start modelling - and one of them is check for collinearity. I also wouldn't care which variable selection method worked for other data sets that violated the rules of the regression that both methods are meant to apply to. $\endgroup$ Commented Dec 15, 2012 at 21:34
$\begingroup$
$\endgroup$
LASSO encourages shrinking of coefficients to 0, i.e. dropping those variates from your model. On contrast, other regularization techniques like a ridge tend to keep all variates.
So I'd recommend to think about whether this dropping makes sense for your data. E.g. consider setting up a clinical diagnostic test either on gene microarray data or on vibrational spectroscopic data.
You'd expect some genes to carry relevant information, but lots of other genes are just noise wrt. your application. Dropping those variates is a perfectly sensible idea.
By contrast, vibrational spectroscopic data sets (while usually having similar dimensions compared to microarray data) tend to have the relevant information "smeared" over large parts of the spectrum (correlation). In this situation, asking the regularization to drop variates is not a particularly sensible approach. The more so, as other regularization techniques like PLS are more adapted to this type of data.
The Elements of Statistical Learning gives a good discussion of the LASSO, and contrasts it to other regularization techniques.
answered Nov 28, 2013 at 16:35
$\begingroup$
$\endgroup$
If you only care about prediction error and don't care about interpretability, casual-inference, model-simplicity, coefficients' tests, etc, why do you still want to use linear regression model?
You can use something like boosting on decision trees or support vector regression and get better prediction quality and still avoid overfitting in both mentioned cases. That is Lasso may not be the best choice to get best prediction quality.
If my understanding is correct, Lasso is intended for situations when you are still interested in the model itself, not only predictions. That is - see selected variables and their coefficients, interpret in some way etc. And for this - Lasso may not be the best choice in certain situations as discussed in other questions here.
$\begingroup$
$\endgroup$
3
This is already quite an old question but I feel that in the meantime most of the answers here are quite outdated (and the one that's checked as the correct answer is plain wrong imho).
First, in terms of getting good prediction performance it is not universally true that LASSO is always better than stepwise. The paper "Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso" by Hastie et al (2017) provides an extensive comparison of forward stepwise, LASSO and some LASSO variants like the relaxed LASSO as well as best subset, and they show that stepwise is sometimes better than LASSO. A variant of LASSO though --- relaxed LASSO - was the one that produced the highest model prediction accuracy under the widest range of circumstances. The conclusion about which is best depends a lot on what you consider best though, e.g. whether this would be highest prediction accuracy or selecting the fewest false positive variables.
There is a whole zoo of sparse learning methods though, most of which are better than LASSO. E.g. there is Meinhausen's relaxed LASSO, adaptive LASSO and SCAD and MCP penalized regression as implemented in the ncvreg and ordinis packages, which all have less bias than standard LASSO and so are preferrable. Furthermore, if you are interest in the absolute sparsest solution with the best prediction performance then L0 penalized regression (aka best subset, i.e. based on penalization of the nr of nonzero coefficients as opposed to the sum of the absolute value of the coefficients in LASSO) is better than LASSO, see e.g. the l0ara package and my own L0glm package (in development, some benchmarks here) which approximates L0 penalized GLMs using an iterative adaptive ridge procedure, and which unlike LASSO also works very well with highly collinear variables, and the L0Learn package, which can fit L0 penalized regression models using coordinate descent, potentially in combination with an L2 penalty to regularize collinearity. Recently, the abess package for best subset selection also came out.
So to come back to your original question: why not use LASSO for variable selection? :
because the coefficients will be highly biased, which is improved in relaxed LASSO, MCP and SCAD penalized regression, and resolved completely in L0 penalized regression (which has a full oracle property, ie it can pick out both the causal variables and retun unbiased coefficients, also for $p > n$ cases)
because it tends to produce way more false positives than L0 penalized regression (in my tests
L0glmperforms best on this front, ie iterative adaptive ridge, followed byL0Learn; for low dimensional problemsabessalso works well)because it cannot deal well with collinear variables (it would essentially just randomly select one of the collinear variables) - iterative adapative ridge /
l0ara/L0glmand the L0L2 penalties inL0Learnare much better at dealing with that. In the case ofL0glme.g. 2 perfectly collinear variables would end up having their effects split up equally across both.
Of course, in general, you'll still have to use cross validation to tune your regularization parameter(s) to get optimal prediction performance, or - even better - in the case of L0glm one can pick the level of regularisation so that the AIC, BIC, GIC, mBIC or eBIC of your model would be expected to be maximised (optimising AIC would then roughly correspond to optimising predictive performance, whereas optimising BIC, mBIC or eBIC would asymptomatically result in consistent variable selection). But that's not an issue. And you can even do high dimensional inference on your parameters and calculate 95% confidence intervals on your coefficients if you like via nonparametric bootstrapping (even taking into account uncertainty on the selection of the optimal regularization if you do your cross validation also on each bootstrapped dataset, though that becomes quite slow then).
Computationally LASSO is not slower to fit than stepwise approaches btw, certainly not if one uses highly optimized code that uses warm starts to optimize your LASSO regularization (you can compare yourself using the fs command for forward stepwise and lasso for LASSO in the bestsubset package). The fact that stepwise approaches are still popular probably has to do with the mistaken belief of many that one could then just keep your final model and report it's associated p values - which in fact is not a correct thing to do, as this doesn't take into account the uncertainty introduced by your model selection, resulting in way too optimistic p values.
And to address the comment of Peter Flom above (which I think completely misses the point): if there is any reason to include prior information then L0glm e.g. allows one to do that: the provided starting values with argument start can be seen as corresponding to the square root of the SD of the implied zero-centered Gaussian prior, and so correspond to your prior & belief in the coefficients being centered around a given prior mean (e.g. zero, but using prior.mean other more informative priors could be specified too) and there is also the argument no.pen to specify which variables should not be penalized at all (e.g. the intercept or particular variables that should always be in). The latter is in fact an option provided by all major regularized regression frameworks (e.g. glmnet, ncvreg, ordinis).
Hope this helps?
answered Aug 21, 2019 at 19:45
-
2$\begingroup$ But I do not understand. LASSO is being sold in the literature as being one of the methods to be used when you have multicollinearity. How come it cannot, in your words, 'deal well with collinear variables'? $\endgroup$– ARATCommented Nov 10, 2019 at 11:05
-
6$\begingroup$ When you have highly collinear variables LASSO will end up almost randomly selecting one of them (typically the first that enters the model if you fit via cyclical coordinate descent). Elastic net and also L0 penalized models fitted via iterative adaptive ridge (sometimes referred to as broken adaptive ridge) by contrast will select collinear variables in groups and divide the total effect over them (see sciencedirect.com/science/article/pii/S0047259X17305067 and refs therein). It's for this improved handling of collinear variables that methods like elastic net were developed. $\endgroup$ Commented Nov 11, 2019 at 12:32
-
1$\begingroup$ See stats.stackexchange.com/questions/264016/… $\endgroup$ Commented Jan 17, 2020 at 19:51
$\begingroup$
$\endgroup$
6
If two predictors are highly correlated LASSO can end up dropping one rather arbitrarily. That's not very good when you're wanting to make predictions for a population where those two predictors aren't highly correlated, & perhaps a reason for preferring ridge regression in those circumstances.
You might also think standardization of predictors (to say when coefficients are "big" or "small") rather arbitrary & be puzzled (like me) about sensible ways to standardize categorical predictors.
answered Nov 28, 2013 at 14:33
-
1$\begingroup$ Thanks for this answer. Do you know any papers that discuss the issues with correlated predictors / categorical predictors? $\endgroup$– Berk U.Commented Mar 6, 2015 at 3:59
-
2$\begingroup$ Its worth adding that there are other penalized regression methods that attempt to alleviate those issues (such as elastic net). $\endgroup$ Commented Jul 13, 2016 at 12:30
-
$\begingroup$ For doing variable selection with highly collinear variables, iterative adaptive ridge (which approximates L0 penalized regression and s implemented in the l0ara package) tends to perform best, or L0L2 penalties, as implemented in the L0Learn package also perform well... $\endgroup$ Commented Aug 21, 2019 at 19:47
-
$\begingroup$ For categorical predictors - these are first recoded into binary dummy variables, and these binary dummy variables can then be standardizes like any other continuous variable... $\endgroup$ Commented Jan 19, 2020 at 8:18
-
$\begingroup$ @TomWenseleers: They can be: the issue then is that the solution depends on arbitrary aspects of the encoding - & the same goes for other multiple-degree-of-freedom predictors, e.g. spline, polynomial, or Fourier terms, or a numerical measure plus 'not applicable' indicator variable. With L2-norm penalization alone it suffices to use an orthonormal basis for the set of vectors of the design matrix making up the predictor to remove that dependence. The group LASSO also fixes the problem - I'll update my answer to discuss it. $\endgroup$– Scortchi ♦Commented Mar 3, 2020 at 20:30
$\begingroup$
$\endgroup$
Lasso is only useful if you're restricting yourself to consider models which are linear in the parameters to be estimated. Stated another way, the lasso does not evaluate whether you have chosen the correct form of the relationship between the independent and dependent variable(s).
It is very plausible that there may be nonlinear, interactive or polynomial effects in an arbitrary data set. However, these alternative model specifications will only be evaluated if the user conducts that analysis; the lasso is not a substitute for doing so.
For a simple example of how this can go wrong, consider a data set in which disjoint intervals of the independent variable will predict alternating high and low values of the dependent variable. This will be challenging to sort out using conventional linear models, since there is not a linear effect in the manifest variables present for analysis (but some transformation of the manifest variables may be helpful). Left in its manifest form, the lasso will incorrectly conclude that this feature is extraneous and zero out its coefficient because there is no linear relationship. On the other hand, because there are axis-aligned splits in the data, a tree-based model like a random forest will probably do pretty well.
answered Jul 26, 2016 at 14:55
$\begingroup$
$\endgroup$
4
One practical disadvantage of lasso and other regularization techniques is finding the optimal regularization coefficient, lambda. Using cross validation to find this value can be just as expensive as stepwise selection techniques.
-
1
-
5$\begingroup$ This claim is not really true. If you adopt the "warm start" grid search as in the glmnet method, you can comput the entire grid very quickly. $\endgroup$ Commented Dec 15, 2012 at 10:50
-
1$\begingroup$ @probabilityislogic True, I only read about warm starts after I made the above comment. What do you think of this paper, which indicates warm starts are slower and sometimes less effective than simple cross validation? users.cis.fiu.edu/~lzhen001/activities/KDD2011Program/docs/… $\endgroup$– rm999Commented Dec 17, 2012 at 21:59
-
5$\begingroup$ Bayesian lasso does not require a fixed $\lambda$ :) $\endgroup$ Commented Aug 25, 2014 at 9:11
$\begingroup$
$\endgroup$
2
I am not a LASSO expert but I am an expert in time series. If you have time series data or spatial data then I would studiously avoid a solution that was predicated on independent observations. Furthermore if there are unknown deterministic effects that have played havoc with your data (level shifts / time trends etc) then LASSO would be even less a good hammer. In closing when you have time series data you often need to segment the data when faced with parameters or error variance that change over time.
-
1$\begingroup$ LASSO may deliver good forecasting performance when applied on regression-based time series models such ar autoregressions (AR), vector autoregressions (VAR) and vector error correction models (VECM). For example, seach for lasso vector autoregression and you will find many examples in the academic literature. In my own experience, using LASSO for stationary VAR models provides superior forecasting performance as compared to all subset selection or ridge regularization, while ridge regularization beats LASSO for integrated VAR models (due to multicollinearity, as per Scortchi's answer). $\endgroup$ Commented May 20, 2016 at 16:28
-
$\begingroup$ So the failure of LASSO is not inherent in the data being time series. $\endgroup$ Commented May 20, 2016 at 16:29
$\begingroup$
$\endgroup$
5
One big one is the difficulty of doing hypothesis testing. You can't easily figure out which variables are statistically significant with Lasso. With stepwise regression, you can do hypothesis testing to some degree, if you're careful about your treatment of multiple testing.
-
8$\begingroup$ I'd say that's an advantage, not a disadvantage. It stops you from doing something you probably shouldn't be doing. $\endgroup$ Commented Mar 7, 2011 at 0:59
-
$\begingroup$ @Peter: Why? I'm assuming that you would properly correct for multiple testing, etc. such that the P-values obtained would be valid. $\endgroup$– dsimchaCommented Mar 7, 2011 at 1:51
-
10$\begingroup$ there really isn't a way to properly correct for multiple testing in stepwise. See, e.g. Harrell Regression modeling strategies. There's no way to know the right correction $\endgroup$ Commented Mar 8, 2011 at 1:53
-
4$\begingroup$ It is true that the difficulty of doing hypothesis testing is a potential disadvantage of LASSO. It is not true that this is a disadvantage vis-a-vie stepwise regression. $\endgroup$ Commented Jul 26, 2016 at 20:06
-
3$\begingroup$ Well there is the selective inference framework (implemented in the selectiveInference package) to do (post selection) inference for the LASSO... Or for any variable selection method one could use nonparametric bootstrapping to do inference and get confidence intervals on your parameter estimates... $\endgroup$ Commented Aug 21, 2019 at 19:20
$\begingroup$
$\endgroup$
2
There is a simple reason why not using LASSO for variable selection. It just does not work as well as advertised. This is due to its fitting algorithm that includes a penalty factor that penalizes the model against higher regression coefficients. It seems like a good idea, as people think it always reduces model overfitting, and improves predictions (on new data). In reality it very often does the opposite ... increase model under-fitting and weakens prediction accuracy. You can see many examples of that by searching the Internet for Images and searching specifically for "LASSO MSE graph." Whenever such graphs show the lowest MSE at the beginning of the X-axis, it shows a LASSO that has failed (increase model under-fitting).
The above unintended consequences are due to the penalty algorithm. Because of it LASSO has no way of distinguishing between a strong causal variable with predictive information and an associated high regression coefficient and a weak variable with no explanatory or predictive information value that has a low regression coefficient. Often, LASSO will prefer the weak variable over the strong causal variable. Also, it may at times even cause to shift the directional signs of variables (shifting from one direction that makes sense to an opposite direction that does not). You can see many examples of that by searching the Internet for Images and searching specifically for "LASSO coefficient path".
-
$\begingroup$ If one digests the evidence that supports my argument as stated in the last sentence of my answer, they may consider removing the down-vote. $\endgroup$– SympaCommented Jan 21, 2020 at 3:59
-
$\begingroup$ Feature selection using only data is primarily a mirage. The data simply do not possess the information needed to make correct choices. This is more pronounced when there are collinearities but is true even in ideal orthogonal predictor cases. See this presentation which shows how the lasso for example has almost no chance of fiding the "right" features. $\endgroup$ Commented Jul 13, 2023 at 11:35
$\begingroup$
$\endgroup$
I have always found variable reduction techniques hurting the predictability, especially for multi classification. Stepwise elimination methods are also not very effective with highly correlated predictors, they are time consuming too. It is a tough area to deal with and it should be done differently on case to case basis. In my experience the dimensionality reduction techniques like LDA or PLS worked well, however they demand huge memory allocation if the number of predictors are too large in number. Even running LASSO on large size will demand huge memory allocation. Hence we should continuously look for more creative statistical based approaches for reducing large size of number of predictors.
