Using LASSO only for feature selection

Question

In my machine learning class, we have learned about how LASSO regression is very good at performing feature selection, since it makes use of $l_1$ regularization.

My question: do people normally use the LASSO model just for doing feature selection (and then proceed to dump those features into a different machine learning model), or do they typically use LASSO to perform both the feature selection and the actual regression?

For example, suppose that you want to do ridge regression, but you believe that many of your features are not very good. Would it be wise to run LASSO, take only the features that are not near-zeroed out by the algorithm, and then use only those in dumping your data into a ridge regression model? This way, you get the benefit of $l_1$ regularization for performing feature selection, but also the benefit of $l_2$ regularization for reducing overfitting. (I know that this basically amounts to Elastic Net Regression, but it seems like you don't need to have both the $l_1$ and $l_2$ terms in the final regression objective function.)

Aside from regression, is this a wise strategy when performing classification tasks (using SVMs, neural networks, random forests, etc.)?

Answer 1

Almost any approach that does some form of model selection and then does further analyses as if no model selection had previously happened typically has poor properties. Unless there are compelling theoretical arguments backed up by evidence from e.g. extensive simulation studies for realistic sample sizes and feature versus sample size ratios to show that this is an exception, it is likely that such an approach will have unsatisfactory properties. I am not aware of any such positive evidence for this approach, but perhaps someone else is. Given that there are reasonable alternatives that achieve all desired goals (e.g. the elastic net), it this approach is hard to justify using such a suspect ad-hoc approach instead.

Answer 2

Besides all the answers above: It is possible to calculate an exact chi2 permutation test for 2x2 and rxc tables. Instead of comparing our observed value of the chi-square statistic to an asymptotic chi-square distribution we need to compare it to the exact permutation distribution. We need to permute our data in all possible ways keeping the row and column margins constant. For each permuted dataed set we caluclated the chi2 statistics . We then compare our observed chi2 with the (sorted) chi2 statistics The ranking of the real test statistic among the permuted chi2 test statistics gives a p-value.

Answer 3

The idea of performing a second regression with a subselection of regressors defined by a first LASSO regression is called Relaxed LASSO. See for instance this question: Advantages of doing "double lasso" or performing lasso twice?

The idea can certainly have benefits. LASSO will not only select regressors/features, and it will also shrink the related estimates of the model parameters. The shrinkage is not neccesarily bad and it can improve model performance, but too much shrinking/bias is not good. With a two stage method one can seperately regulate the amount of shrinking and the amount of feature selection.

The popular elastic net regularization is not as versatile in this task. The additional $l_2$ penalty term can only be used to increase the shrinking. But it doesn't help to decrease the shrinking for a given amount of feature selection.

A two stage method has benefits when for a given ideal amount of feature selection there is too much shrinking. This potentially happens in cases where there are a lot of features where severe selection is neccesary, while the amount of observations is sufficient to have little overfitting with the selected features.