I understand the advantages of using the Lasso (e.g. scalability, regularization). That being said, I am also aware that the Lasso is an approximate method for feature selection, and that it does not necessarily return the optimal subset of features (i.e., the one that would be identified through $\ell_0$-minimization, or a brute-force search).

In light of this, I'm wondering:

  1. What are the practical disadvantages of using the Lasso for feature selection in binary classification problems?

  2. Is there a realistic example where the Lasso returns a subset of features that is completely different from the true optimal set of features?

Note: To be clear, I know that there was a related discussion on Lasso vs. stepwise regression. The reason why I've posted a new question instead of posting in the old forum is because:

  • the old question was about regression problems
  • the old question compares Lasso to stepwise regression (also an approximate method). In comparison, I suppose this is trying to compare Lasso ($\ell_1$-penalty regularization) to brute force ($\ell_0$-penalty regularization), which would be optimal.

Lasso doesn't just do feature selection. It's trying to minimise the sum of squared errors subject penalised by the magnitude of the regression coefficients. This will often lead to a lower mean square error compared to an OLS procedure.

The nature of the $l_1$ penalty pushes many regression coefficients to zero; inducing sparsity and thus constituting form of feature selection.

However, if we are picking the best subset of all available predictors by minimum sum of squared errors - the best choice would be all of them.

If we set an arbitrary $k=10$, how do we know we should not choose $k=11$? Lasso doesn't have this problem as $k$ is chosen on a principled basis according to the optimisation problem.

  • $\begingroup$ Not to mention that the brute force approach would require more data and more care to avoid overfitting. $\endgroup$
    – Wayne
    Mar 6 '15 at 14:41
  • $\begingroup$ @conjectures I agree with your comment entirely, which I think is saying that there is a benefit in Lasso due to a regularization effect. However, my question was really about practical disadvantages with regards to using Lasso for feature selection (e.g. it supposedly doesn't deal well with correlated or categorial inputs). In practice, there are some problems where you have a hard cap to use at most $k \leq 10$ input variables due to operational constraints (e.g. if we had a fixed budget, and each feature could only be obtained by means of a test). $\endgroup$
    – Berk U.
    Mar 6 '15 at 17:14
  • $\begingroup$ @Wayne Is that because of the regularization effect of the Lasso? $\endgroup$
    – Berk U.
    Mar 6 '15 at 17:16
  • $\begingroup$ @BerkU.: I'm not sure. I was thinking more of the number of tests you'd end up doing -- especially in the case that conjectures talks about where $k$ varies -- which could be so large in comparison to the amount of data you have that you end up overfitting even if you try CV or other techniques that might avoid overfitting in more normal circumstances. $\endgroup$
    – Wayne
    Mar 6 '15 at 18:36
  • $\begingroup$ Also, the least angle regression procedure for Lasso is very smart in terms of efficiency: statweb.stanford.edu/~tibs/lasso/simple.html $\endgroup$ Mar 9 '15 at 14:38

Not the answer you're looking for? Browse other questions tagged or ask your own question.