# LASSO vs forward selection

I have two questions:

1. I use cross validation to select a LASSO model, does the step in which a particular variable enter, indicate its relative importance?
Let's age enter in step 1 and sex enter in step 2, country enter in step 3. Can I make the claim that age is more important than sex , which in turn is more important than country. (Sorry about my bad language, I wonder what would be the precise statement I can make here.)

2. How does the procedure in question 1 compare to forward selection, and what is its theoretical advantage of weakness?

The point of the LASSO is that all variables are in the model until they are subject to sufficient shrinkage that their coefficient would change sign (i.e. go through zero effect). Variables are only eliminated from the model, they do not enter, at progressively larger values of shrinkage. Algorithms exist that compute the entire path from least-squares solution to full shrinkage in an efficient manner. The plots often produced for these paths are counter-intuitive as the least squares solution is on the right and the path progresses to the left as more shrinkage is applied.

The terms in the model are all standardised and hence the (absolute) sizes of the coefficients at any stage in the LASSO path are an indication of the "importance" of each variable in terms of their effect on the response.

The LASSO is very different to forward selection; all variables are in the model and then shrinkage is applied to the coefficients which places a restriction on the cumulative size of the absolute values of the set of coefficients. As shrinkage is increased, the maximum size on the set coefficients is reduced. Variables contributing little (or nothing) to the fit to the response (i.e. of lesser importance) are shrunk more than important variables. However, the LASSO doesn't handle correlated variables so well, unlike ridge regression, and hence a procedure called the elastic net has been developed which combines elements of the two (LASSO and ridge penalties).

One of the advantages of the LASSO is that variable selection is part of the model fitting process. IIRC however, the LASSO estimates of the coefficients are subject to greater bias, but lower variance, than the least squares solutions. This trade-off may be a good thing in terms of prediction if the reduction in variance contribution to model error is larger than the additional bias contribution. But as the LASSO estimates are somewhat biased, that complicates their interpretation as one would in ordinary least squares.

• (+1) The Lasso Page also contains useful references.
– chl
Feb 28, 2013 at 11:24
• Sorry would you please elaborate more on the question "does the step in which a particular variable enter, indicate its relative importance?". What is the precise statement I can make here, since lasso doesn't handle correlated variable well, perhaps it is not really ranking all variables in a ranking of importance, or perhaps would you please define precisely what 'importance' is Mar 1, 2013 at 12:55
• Your question doesn't make sense; variables do not enter in the LASSO, their coefficients are shrunk and possibly eliminated from the model. Correlated variable often show up as having large coefficient with opposite sign (e.g. a large +ve coef balanced by a large -ve coef). That may mean that some other more useful variables than one of the correlated pair get shrunk out of the model sooner (as they have smaller coefficients than the spuriously large one). The LASSO is not really about importance, but it includes model selection. Mar 1, 2013 at 14:44