A similar question was asked here Why can't ridge regression provide better interpretability than LASSO?, and the answer suggested that a main difference between lasso and ridge is that a zero threshold from lasso is a more principled threshold than an arbitrarily picked one for the ridge regression coefficients. Given the nature of the L2 regularization norm, a principled threshold in Ridge regression could be at 1 (since that is when the squared penalty becomes smaller than the absolute penalty/the value that is squared), but that would arguably be worse than some really small number, so I'm not sure which qualities of "principled" are attractive here? Is there something magical with using Lasso with a zero threshold here (to me it seems similar that the model doesn't use a feature at all, or uses it very little)?
On this topic, I read a few comments saying that Lasso will be more suitable when there are only a few features that are truly important to the model, and the the L2 norm will be more suitable when there are many features that are truly important to the model. Since we can't know what is truly important to the model, the suggestions I saw was to use Elastic Net regularization to solve this via a data-driven approach to variable selection that strikes a suitable balance of L1 and L2 regularization penalties. From reading up about Elastic Net, I understand that it combines some properties of the L1 and L2 curve above, but I don't quite understand how it is a "data-driven" approach that could somehow guarantee that only the variables that are truly important are selected. Is there an intuitive explanation for this and is elastic net generally the preferred alternative to get the benefits from Ridge together with variable selection?
