From what I know, using lasso for variable selection handles the problem of correlated inputs. Also, since it is equivalent to Least Angle Regression, it is not slow computationally. However, many people (for example people I know doing bio-statistics) still seem to favour stepwise or stagewise variable selection. Are there any practical disadvantages of using the lasso that makes it unfavourable?
There is NO reason to do stepwise selection. It's just wrong.
LASSO/LAR are the best automatic methods. But they are automatic methods. They let the analyst not think.
In many analyses, some variables should be in the model REGARDLESS of any measure of significance. Sometimes they are necessary control variables. Other times, finding a small effect can be substantively important.
LASSO encourages shrinking of coefficients to 0, i.e. dropping those variates from your model. On contrast, other regularization techniques like a ridge tend to keep all variates.
So I'd recommend to think about whether this dropping makes sense for your data. E.g. consider setting up a clinical diagnostic test either on gene microarray data or on vibrational spectroscopic data.
You'd expect some genes to carry relevant information, but lots of other genes are just noise wrt. your application. Dropping those variates is a perfectly sensible idea.
By contrast, vibrational spectroscopic data sets (while usually having similar dimensions compared to microarray data) tend to have the relevant information "smeared" over large parts of the spectrum (correlation). In this situation, asking the regularization to drop variates is not a particularly sensible approach. The more so, as other regularization techniques like PLS are more adapted to this type of data.
The Elements of Statistical Learning gives a good discussion of the LASSO, and contrasts it to other regularization techniques.
If you only care about prediction error and don't care about interpretability, casual-inference, model-simplicity, coefficients' tests, etc, why do you still want to use linear regression model?
You can use something like boosting on decision trees or support vector regression and get better prediction quality and still avoid overfitting in both mentioned cases. That is Lasso may not be the best choice to get best prediction quality.
If my understanding is correct, Lasso is intended for situations when you are still interested in the model itself, not only predictions. That is - see selected variables and their coefficients, interpret in some way etc. And for this - Lasso may not be the best choice in certain situations as discussed in other questions here.
If two predictors are highly correlated LASSO can end up dropping one rather arbitrarily. That's not very good when you're wanting to make predictions for a population where those two predictors aren't highly correlated, & perhaps a reason for preferring ridge regression in those circumstances.
You might also think standardization of predictors (to say when coefficients are "big" or "small") rather arbitrary & be puzzled (like me) about sensible ways to standardize categorical predictors.
This is already quite an old question but I feel that in the meantime most of the answers here are quite outdated (and the one that's checked as the correct answer is plain wrong imho).
First, in terms of getting good prediction performance it is not universally true that LASSO is always better than stepwise. The paper "Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso" by Hastie et al (2017) provides an extensive comparison of forward stepwise, LASSO and some LASSO variants like the relaxed LASSO as well as best subset, and they show that stepwise is sometimes better than LASSO. A variant of LASSO though --- relaxed LASSO - was the one that produced the highest model prediction accuracy under the widest range of circumstances. The conclusion about which is best depends a lot on what you consider best though, e.g. whether this would be highest prediction accuracy or selecting the fewest false positive variables.
There is a whole zoo of sparse learning methods though, most of which are better than LASSO. E.g. there is Meinhausen's relaxed LASSO, adaptive LASSO and SCAD and MCP penalized regression as implemented in the
ncvreg package, which all have less bias than standard LASSO and so are preferrable. Furthermore, if you are interest in the absolute sparsest solution with the best prediction performance then L0 penalized regression (aka best subset, i.e. based on penalization of the nr of nonzero coefficients as opposed to the sum of the absolute value of the coefficients in LASSO) is better than LASSO, see e.g. the
l0ara package which approximates L0 penalized GLMs using an iterative adaptive ridge procedure, and which unlike LASSO also works very well with highly collinear variables, and the
L0Learn package, which can fit L0 penalized regression models using coordinate descent, potentially in combination with an L2 penalty to regularize collinearity.
So to come back to your original question: why not use LASSO for variable selection? :
because the coefficients will be highly biased, which is improved in relaxed LASSO, MCP and SCAD penalized regression, and resolved completely in L0 penalized regression (which has a full oracle property, ie it can pick out both the causal variables and retun unbiased coefficients, also for $p > n$ cases)
because it tends to produce way more false positives than L0 penalized regression (in my tests
l0araperforms best then, ie iterative adaptive ridge, followed by
because it cannot deal well with collinear variables (it would essentially just randomly select one of the collinear variables) - iterative adapative ridge /
l0araand the L0L2 penalties in
L0Learnare much better at dealing with that.
Of course, in general, you'll still have to use cross validation to tune your regularization parameter(s) to get optimal prediction performance, but that's not an issue. And you can even do high dimensional inference on your parameters and calculate 95% confidence intervals on your coefficients if you like via nonparametric bootstrapping (even taking into account uncertainty on the selection of the optimal regularization if you do your cross validation also on each bootstrapped dataset, though that becomes quite slow then).
Computationally LASSO is not slower to fit than stepwise approaches btw, certainly not if one uses highly optimized code that uses warm starts to optimize your LASSO regularization (you can compare yourself using the
fs command for forward stepwise and
lasso for LASSO in the
bestsubset package). The fact that stepwise approaches are still popular probably has to do with the mistaken belief of many that one could then just keep your final model and report it's associated p values - which in fact is not a correct thing to do, as this doesn't take into account the uncertainty introduced by your model selection, resulting in way too optimistic p values.
Hope this helps?
Lasso is only useful if you're restricting yourself to consider models which are linear in the parameters to be estimated. Stated another way, the lasso does not evaluate whether you have chosen the correct form of the relationship between the independent and dependent variable(s).
It is very plausible that there may be nonlinear, interactive or polynomial effects in an arbitrary data set. However, these alternative model specifications will only be evaluated if the user conducts that analysis; the lasso is not a substitute for doing so.
For a simple example of how this can go wrong, consider a data set in which disjoint intervals of the independent variable will predict alternating high and low values of the dependent variable. This will be challenging to sort out using conventional linear models, since there is not a linear effect in the manifest variables present for analysis (but some transformation of the manifest variables may be helpful). Left in its manifest form, the lasso will incorrectly conclude that this feature is extraneous and zero out its coefficient because there is no linear relationship. On the other hand, because there are axis-aligned splits in the data, a tree-based model like a random forest will probably do pretty well.
I am not a LASSO expert but I am an expert in time series. If you have time series data or spatial data then I would studiously avoid a solution that was predicated on independent observations. Furthermore if there are unknown deterministic effects that have played havoc with your data (level shifts / time trends etc) then LASSO would be even less a good hammer. In closing when you have time series data you often need to segment the data when faced with parameters or error variance that change over time.
One big one is the difficulty of doing hypothesis testing. You can't easily figure out which variables are statistically significant with Lasso. With stepwise regression, you can do hypothesis testing to some degree, if you're careful about your treatment of multiple testing.
I have always found variable reduction techniques hurting the predictability, especially for multi classification. Stepwise elimination methods are also not very effective with highly correlated predictors, they are time consuming too. It is a tough area to deal with and it should be done differently on case to case basis. In my experience the dimensionality reduction techniques like LDA or PLS worked well, however they demand huge memory allocation if the number of predictors are too large in number. Even running LASSO on large size will demand huge memory allocation. Hence we should continuously look for more creative statistical based approaches for reducing large size of number of predictors.
There is a simple reason why not using LASSO for variable selection. It just does not work as well as advertised. This is due to its fitting algorithm that includes a penalty factor that penalizes the model against higher regression coefficients. It seems like a good idea, as people think it always reduces model overfitting, and improves predictions (on new data). In reality it very often does the opposite ... increase model under-fitting and weakens prediction accuracy. You can see many examples of that by searching the Internet for Images and searching specifically for "LASSO MSE graph." Whenever such graphs show the lowest MSE at the beginning of the X-axis, it shows a LASSO that has failed (increase model under-fitting).
The above unintended consequences are due to the penalty algorithm. Because of it LASSO has no way of distinguishing between a strong causal variable with predictive information and an associated high regression coefficient and a weak variable with no explanatory or predictive information value that has a low regression coefficient. Often, LASSO will prefer the weak variable over the strong causal variable. Also, it may at times even cause to shift the directional signs of variables (shifting from one direction that makes sense to an opposite direction that does not). You can see many examples of that by searching the Internet for Images and searching specifically for "LASSO coefficient path".