This question already has an answer here:
I understand the advantages of using the Lasso (e.g. scalability, regularization). That being said, I am also aware that the Lasso is an approximate method for feature selection, and that it does not necessarily return the optimal subset of features (i.e., the one that would be identified through $\ell_0$-minimization, or a brute-force search).
In light of this, I'm wondering:
What are the practical disadvantages of using the Lasso for feature selection in binary classification problems?
Is there a realistic example where the Lasso returns a subset of features that is completely different from the true optimal set of features?
Note: To be clear, I know that there was a related discussion on Lasso vs. stepwise regression. The reason why I've posted a new question instead of posting in the old forum is because:
- the old question was about regression problems
- the old question compares Lasso to stepwise regression (also an approximate method). In comparison, I suppose this is trying to compare Lasso ($\ell_1$-penalty regularization) to brute force ($\ell_0$-penalty regularization), which would be optimal.