LASSO vs forward selection I have two questions:


*

*I use cross validation to select a LASSO model, does the step in which a particular variable enter, indicate its relative importance?
Let's age enter in step 1 and sex enter in step 2, country enter in step 3. Can I make the claim that age is more important than sex , which in turn is more important than country. (Sorry about my bad language, I wonder what would be the precise statement I can make here.)

*How does the procedure in question 1 compare to forward selection, and what is its theoretical advantage of weakness?
 A: The point of the LASSO is that all variables are in the model until they are subject to sufficient shrinkage that their coefficient would change sign (i.e. go through zero effect). Variables are only eliminated from the model, they do not enter, at progressively larger values of shrinkage. Algorithms exist that compute the entire path from least-squares solution to full shrinkage in an efficient manner. The plots often produced for these paths are counter-intuitive as the least squares solution is on the right and the path progresses to the left as more shrinkage is applied.
The terms in the model are all standardised and hence the (absolute) sizes of the coefficients at any stage in the LASSO path are an indication of the "importance" of each variable in terms of their effect on the response.
The LASSO is very different to forward selection; all variables are in the model and then shrinkage is applied to the coefficients which places a restriction on the cumulative size of the absolute values of the set of coefficients. As shrinkage is increased, the maximum size on the set coefficients is reduced. Variables contributing little (or nothing) to the fit to the response (i.e. of lesser importance) are shrunk more than important variables. However, the LASSO doesn't handle correlated variables so well, unlike ridge regression, and hence a procedure called the elastic net has been developed which combines elements of the two (LASSO and ridge penalties).
One of the advantages of the LASSO is that variable selection is part of the model fitting process. IIRC however, the LASSO estimates of the coefficients are subject to greater bias, but lower variance, than the least squares solutions. This trade-off may be a good thing in terms of prediction if the reduction in variance contribution to model error is larger than the additional bias contribution. But as the LASSO estimates are somewhat biased, that complicates their interpretation as one would in ordinary least squares.
