60
$\begingroup$

From what I know, using lasso for variable selection handles the problem of correlated inputs. Also, since it is equivalent to Least Angle Regression, it is not slow computationally. However, many people (for example people I know doing bio-statistics) still seem to favour stepwise or stagewise variable selection. Are there any practical disadvantages of using the lasso that makes it unfavourable?

$\endgroup$
  • 9
    $\begingroup$ I don't know where you heard that Lasso handles the problem of collinearity, that's absolutely not true. $\endgroup$ – Macro Dec 14 '12 at 16:57
  • 3
    $\begingroup$ Horseshoe prior is better than LASSO for model selection - at least in the sparse model case (where model selection is the most useful). You can find a discussion of these points in this Link. Two of the authors of this paper also got a similar article into the Valencia meetings, Bayesian Statistics 9 "Shrink Globally Act Locally: Sparse Bayesian regularisation and prediction". The Valencia article goes into much more detail on a penalty framework. $\endgroup$ – probabilityislogic Dec 16 '12 at 13:23
  • 9
    $\begingroup$ If you are only interested in prediction, then model selection doesn't help and usually hurts (as opposed to a quadratic penalty = L2 norm = ridge regression with no variable selection). LASSO pays a price in predictive discrimination for trying to do variable selection. $\endgroup$ – Frank Harrell Nov 28 '13 at 14:37
  • 3
    $\begingroup$ Tossing a coin to make an arbitrary decision often reveals that you do actually care about the outcome. Any method that offers to make decisions for you about selection of predictors often makes it plain that you do have ideas about which predictors belong more naturally in the model, ideas that you don't want ignored. LASSO can work like that. $\endgroup$ – Nick Cox Nov 28 '13 at 15:11
  • 5
    $\begingroup$ I second @Nick: "no theory available to guide model selection" is hardly ever realistic. Common sense is theory. $\endgroup$ – Scortchi - Reinstate Monica Nov 28 '13 at 19:08
29
$\begingroup$

There is NO reason to do stepwise selection. It's just wrong.

LASSO/LAR are the best automatic methods. But they are automatic methods. They let the analyst not think.

In many analyses, some variables should be in the model REGARDLESS of any measure of significance. Sometimes they are necessary control variables. Other times, finding a small effect can be substantively important.

$\endgroup$
  • 43
    $\begingroup$ "There is NO reason to do stepwise selection. It's just wrong." - Almost never are incredibly sweeping statements like that, devoid of context, good statistical practice. If anything here is "just wrong", it's the bolded statement above. If your analysis is not emphasizing $p$-values or parameter estimates (e.g. predictive models) then stepwise variable selection may be a sensible thing to do and can ::gasp:: outperform LASSO in some cases. (Peter, I know we've had this convo before - this comment is more directed at a future reader who may only come across this post and not the other). $\endgroup$ – Macro Dec 14 '12 at 17:11
  • 4
    $\begingroup$ -1 due to the blanket criticism of stepwise. Its not "just wrong" but has a place as a deterministic model search. You really do have a bee in your bonnet about automatic methods. $\endgroup$ – probabilityislogic Dec 15 '12 at 10:30
  • 8
    $\begingroup$ @Elvis, I'm no expert on the subject or an advocate for stepwise; I'm only taking issue with the unconditional nature of the statement. But, out of curiosity I did some simple simulations and found that when you have a large number of collinear predictors that all have roughly equal effects, backwards selection does better than LASSO, in terms of out-of-sample prediction. I used $$Y_i = \sum_{j=1}^{100} X_{ij} + \varepsilon_{i}$$ with $ \varepsilon \sim N(0,1)$. The predictors are standard normal with ${\rm cor} (X_{ij},X_{ik})=1/2$ for every pair $(j,k)$. $\endgroup$ – Macro Dec 15 '12 at 20:22
  • 10
    $\begingroup$ You should certainly investigate collinearity before embarking on any regression. I'd say that if you have a large number of collinear variables you should not use LASSO or Stepwise; you should either solve the collinearity problem (delete variables, get more data, etc) or use a method designed for such problems (e.g. ridge regression) $\endgroup$ – Peter Flom - Reinstate Monica Dec 15 '12 at 21:07
  • 5
    $\begingroup$ OK, you're right but I don't think it's really relevant. Neither backwards NOR lasso (nor any variable selection method) solves all problems. There are things you have to do before you start modelling - and one of them is check for collinearity. I also wouldn't care which variable selection method worked for other data sets that violated the rules of the regression that both methods are meant to apply to. $\endgroup$ – Peter Flom - Reinstate Monica Dec 15 '12 at 21:34
22
$\begingroup$

If you only care about prediction error and don't care about interpretability, casual-inference, model-simplicity, coefficients' tests, etc, why do you still want to use linear regression model?

You can use something like boosting on decision trees or support vector regression and get better prediction quality and still avoid overfitting in both mentioned cases. That is Lasso may not be the best choice to get best prediction quality.

If my understanding is correct, Lasso is intended for situations when you are still interested in the model itself, not only predictions. That is - see selected variables and their coefficients, interpret in some way etc. And for this - Lasso may not be the best choice in certain situations as discussed in other questions here.

$\endgroup$
20
$\begingroup$

LASSO encourages shrinking of coefficients to 0, i.e. dropping those variates from your model. On contrast, other regularization techniques like a ridge tend to keep all variates.

So I'd recommend to think about whether this dropping makes sense for your data. E.g. consider setting up a clinical diagnostic test either on gene microarray data or on vibrational spectroscopic data.

  • You'd expect some genes to carry relevant information, but lots of other genes are just noise wrt. your application. Dropping those variates is a perfectly sensible idea.

  • By contrast, vibrational spectroscopic data sets (while usually having similar dimensions compared to microarray data) tend to have the relevant information "smeared" over large parts of the spectrum (correlation). In this situation, asking the regularization to drop variates is not a particularly sensible approach. The more so, as other regularization techniques like PLS are more adapted to this type of data.

The Elements of Statistical Learning gives a good discussion of the LASSO, and contrasts it to other regularization techniques.

$\endgroup$
14
$\begingroup$

If two predictors are highly correlated LASSO can end up dropping one rather arbitrarily. That's not very good when you're wanting to make predictions for a population where those two predictors aren't highly correlated, & perhaps a reason for preferring ridge regression in those circumstances.

You might also think standardization of predictors (to say when coefficients are "big" or "small") rather arbitrary & be puzzled (like me) about sensible ways to standardize categorical predictors.

$\endgroup$
  • 1
    $\begingroup$ Thanks for this answer. Do you know any papers that discuss the issues with correlated predictors / categorical predictors? $\endgroup$ – Berk U. Mar 6 '15 at 3:59
  • 2
    $\begingroup$ Its worth adding that there are other penalized regression methods that attempt to alleviate those issues (such as elastic net). $\endgroup$ – bdeonovic Jul 13 '16 at 12:30
  • $\begingroup$ For doing variable selection with highly collinear variables, iterative adaptive ridge (which approximates L0 penalized regression and s implemented in the l0ara package) tends to perform best, or L0L2 penalties, as implemented in the L0Learn package also perform well... $\endgroup$ – Tom Wenseleers Aug 21 at 19:47
9
$\begingroup$

Lasso is only useful if you're restricting yourself to consider models which are linear in the parameters to be estimated. Stated another way, the lasso does not evaluate whether you have chosen the correct form of the relationship between the independent and dependent variable(s).

It is very plausible that there may be nonlinear, interactive or polynomial effects in an arbitrary data set. However, these alternative model specifications will only be evaluated if the user conducts that analysis; the lasso is not a substitute for doing so.

For a simple example of how this can go wrong, consider a data set in which disjoint intervals of the independent variable will predict alternating high and low values of the dependent variable. This will be challenging to sort out using conventional linear models, since there is not a linear effect in the manifest variables present for analysis (but some transformation of the manifest variables may be helpful). Left in its manifest form, the lasso will incorrectly conclude that this feature is extraneous and zero out its coefficient because there is no linear relationship. On the other hand, because there are axis-aligned splits in the data, a tree-based model like a random forest will probably do pretty well.

enter image description here

$\endgroup$
5
$\begingroup$

One practical disadvantage of lasso and other regularization techniques is finding the optimal regularization coefficient, lambda. Using cross validation to find this value can be just as expensive as stepwise selection techniques.

$\endgroup$
  • $\begingroup$ What do you mean by "expensive"? $\endgroup$ – mark999 Mar 8 '12 at 10:41
  • 4
    $\begingroup$ This claim is not really true. If you adopt the "warm start" grid search as in the glmnet method, you can comput the entire grid very quickly. $\endgroup$ – probabilityislogic Dec 15 '12 at 10:50
  • 1
    $\begingroup$ @probabilityislogic True, I only read about warm starts after I made the above comment. What do you think of this paper, which indicates warm starts are slower and sometimes less effective than simple cross validation? users.cis.fiu.edu/~lzhen001/activities/KDD2011Program/docs/… $\endgroup$ – rm999 Dec 17 '12 at 21:59
  • 5
    $\begingroup$ Bayesian lasso does not require a fixed $\lambda$ :) $\endgroup$ – Stéphane Laurent Aug 25 '14 at 9:11
5
$\begingroup$

I am not a LASSO expert but I am an expert in time series. If you have time series data or spatial data then I would studiously avoid a solution that was predicated on independent observations. Furthermore if there are unknown deterministic effects that have played havoc with your data (level shifts / time trends etc) then LASSO would be even less a good hammer. In closing when you have time series data you often need to segment the data when faced with parameters or error variance that change over time.

$\endgroup$
  • 1
    $\begingroup$ LASSO may deliver good forecasting performance when applied on regression-based time series models such ar autoregressions (AR), vector autoregressions (VAR) and vector error correction models (VECM). For example, seach for lasso vector autoregression and you will find many examples in the academic literature. In my own experience, using LASSO for stationary VAR models provides superior forecasting performance as compared to all subset selection or ridge regularization, while ridge regularization beats LASSO for integrated VAR models (due to multicollinearity, as per Scortchi's answer). $\endgroup$ – Richard Hardy May 20 '16 at 16:28
  • $\begingroup$ So the failure of LASSO is not inherent in the data being time series. $\endgroup$ – Richard Hardy May 20 '16 at 16:29
4
$\begingroup$

This is already quite an old question but I feel that in the meantime most of the answers here are quite outdated (and the one that's checked as the correct answer is plain wrong imho).

First, in terms of getting good prediction performance it is not universally true that LASSO is always better than stepwise. The paper "Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso" by Hastie et al (2017) provides an extensive comparison of forward stepwise, LASSO and some LASSO variants like the relaxed LASSO as well as best subset, and they show that stepwise is sometimes better than LASSO. A variant of LASSO though - relaxed LASSO - was the one that produced the highest model prediction accuracy under the widest range of circumstances. The conclusion about which is best depends a lot on what you consider best though, e.g. whether this would be highest prediction accuracy or selecting the fewest false positive variables.

There is a whole zoo of sparse learning methods though, most of which are better than LASSO. E.g. there is Meinhausen's relaxed LASSO, adaptive LASSO and SCAD and MCP penalized regression as implemented in the ncvreg package, which all have less bias than standard LASSO and so are preferrable. Furthermore, if you are interest in the absolute sparsest solution with the best prediction performance then L0 penalized regression (aka best subset, i.e. based on penalization of the nr of nonzero coefficients as opposed to the sum of the absolute value of the coefficients in LASSO) is better than LASSO, see e.g. the l0ara package which approximates L0 penalized GLMs using an iterative adaptive ridge procedure, and which unlike LASSO also works very well with highly collinear variables, and the L0Learn package, which can fit L0 penalized regression models using coordinate descent, potentially in combination with an L2 penalty to regularize collinearity.

So to come back to your original question: why not use LASSO for variable selection? :

(1) because the coefficients will be highly biased, which is improved in relaxed LASSO, MCP and SCAD penalized regression, and resolved completely in L0 penalized regression (which has a full oracle property, ie it can pick out both the causal variables and retun unbiased coefficients, also for p > n cases)

(2) because it tends to produce way more false positives than L0 penalized regression (in my tests l0ara performs best then, ie iterative adaptive ridge, followed by L0Learn)

(3) because it cannot deal well with collinear variables (it would essentially just randomly select one of the collinear variables) - iterative adapative ridge / l0ara and the L0L2 penalties in L0Learn are much better at dealing with that.

Of course, in general, you'll still have to use cross validation to tune your regularization parameter(s) to get optimal prediction performance, but that's not an issue. And you can even do high dimensional inference on your parameters and calculate 95% confidence intervals on your coefficients if you like via nonparametric bootstrapping (even taking into account uncertainty on the selection of the optimal regularization if you do your cross validation also on each bootstrapped dataset, though that becomes quite slow then).

Computationally LASSO is not slower to fit than stepwise approaches btw, certainly not if one uses highly optimized code that uses warm starts to optimize your LASSO regularization (you can compare yourself using the fs command for forward stepwise and lasso for LASSO in the bestsubset package). The fact that stepwise approaches are still popular probably has to do with the mistaken belief of many that one could then just keep your final model and report it's associated p values - which in fact is not a correct thing to do, as this doesn't take into account the uncertainty introduced by your model selection, resulting in way too optimistic p values.

Hope this helps?

$\endgroup$
  • $\begingroup$ But I do not understand. LASSO is being sold in the literature as being one of the methods to be used when you have multicollinearity. How come it cannot, in your words, 'deal well with collinear variables'? $\endgroup$ – ARAT Nov 10 at 11:05
  • $\begingroup$ When you have highly collinear variables LASSO will end up almost randomly selecting one of them (typically the first that enters the model if you fit via cyclical coordinate descent). Elastic net and also L0 penalized models fitted via iterative adaptive ridge (sometimes referred to as broken adaptive ridge) by contrast will select collinear variables in groups and divide the total effect over them (see sciencedirect.com/science/article/pii/S0047259X17305067 and refs therein). It's for this improved handling of collinear variables that methods like elastic net were developed. $\endgroup$ – Tom Wenseleers Nov 11 at 12:32
0
$\begingroup$

One big one is the difficulty of doing hypothesis testing. You can't easily figure out which variables are statistically significant with Lasso. With stepwise regression, you can do hypothesis testing to some degree, if you're careful about your treatment of multiple testing.

$\endgroup$
  • 8
    $\begingroup$ I'd say that's an advantage, not a disadvantage. It stops you from doing something you probably shouldn't be doing. $\endgroup$ – Peter Flom - Reinstate Monica Mar 7 '11 at 0:59
  • $\begingroup$ @Peter: Why? I'm assuming that you would properly correct for multiple testing, etc. such that the P-values obtained would be valid. $\endgroup$ – dsimcha Mar 7 '11 at 1:51
  • 10
    $\begingroup$ there really isn't a way to properly correct for multiple testing in stepwise. See, e.g. Harrell Regression modeling strategies. There's no way to know the right correction $\endgroup$ – Peter Flom - Reinstate Monica Mar 8 '11 at 1:53
  • 4
    $\begingroup$ It is true that the difficulty of doing hypothesis testing is a potential disadvantage of LASSO. It is not true that this is a disadvantage vis-a-vie stepwise regression. $\endgroup$ – gung - Reinstate Monica Jul 26 '16 at 20:06
  • 2
    $\begingroup$ Well there is the selective inference framework (implemented in the selectiveInference package) to do (post selection) inference for the LASSO... Or for any variable selection method one could use nonparametric bootstrapping to do inference and get confidence intervals on your parameter estimates... $\endgroup$ – Tom Wenseleers Aug 21 at 19:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.