48
$\begingroup$

Note: I know that L1 has feature selection property. I am trying to understand which one to choose when feature selection is completely irrelevant.

  1. How to decide which regularization (L1 or L2) to use?
  2. What are the pros & cons of each of L1 / L2 regularization?
  3. Is it recommended to 1st do feature selection using L1 & then apply L2 on these selected variables?
$\endgroup$
3
  • 3
    $\begingroup$ Note that "L1 feature selection" should be rather called regularisation of the feature space; there are many way better methods to do feature selection understood as getting information what is relevant to the modelled problem. $\endgroup$
    – user88
    Nov 28, 2015 at 18:07
  • 2
    $\begingroup$ @mbq: I am curious which "many way better methods" did you mean here? $\endgroup$
    – amoeba
    Nov 29, 2015 at 1:47
  • 2
    $\begingroup$ Like those enumerated here. $\endgroup$
    – user88
    Nov 29, 2015 at 17:58

2 Answers 2

42
$\begingroup$

How to decide which regularization (L1 or L2) to use?

What is your goal? Both can improve model generalization by penalizing coefficients, since features with opposite relationship to the outcome can "offset" each other (a large positive value is counterbalanced by a large negative value). This can arise when there are collinear features. Small changes in the data can result in dramatically different parameter estimates (high variance estimates). Penalization can restrain both coefficients to be smaller. (Hastie et al, Elements of Statistical Learning, 2nd edition, p. 63)

What are the pros & cons of each of L1 / L2 regularization?

L1 regularization can address the multicollinearity problem by constraining the coefficient norm and pinning some coefficient values to 0. Computationally, Lasso regression (regression with an L1 penalty) is a quadratic program which requires some special tools to solve. When you have more features than observations $N$, lasso will keep at most $N$ non-zero coefficients. Depending on context, that might not be what you want.

L1 regularization is sometimes used as a feature selection method. Suppose you have some kind of hard cap on the number of features you can use (because data collection for all features is expensive, or you have tight engineering constraints on how many values you can store, etc.). You can try to tune the L1 penalty to hit your desired number of non-zero features.

L2 regularization can address the multicollinearity problem by constraining the coefficient norm and keeping all the variables. It's unlikely to estimate a coefficient to be exactly 0. This isn't necessarily a drawback, unless a sparse coefficient vector is important for some reason.

In the regression setting, it's the "classic" solution to the problem of estimating a regression with more features than observations. L2 regularization can estimate a coefficient for each feature even if there are more features than observations (indeed, this was the original motivation for "ridge regression").

As an alternative, elastic net allows L1 and L2 regularization as special cases. A typical use-case in for a data scientist in industry is that you just want to pick the best model, but don't necessarily care if it's penalized using L1, L2 or both. Elastic net is nice in situations like these.

Is it recommended to 1st do feature selection using L1 & then apply L2 on these selected variables?

I'm not familiar with a publication proposing an L1-then-L2 pipeline, but this is probably just ignorance on my part. There doesn't seem to be anything wrong with it. I'd conduct a literature review.

A few examples of similar "phased" pipelines exist. One is the "relaxed lasso", which applies lasso regression twice, once to down-select from a large group to a small group of features, and second to estimate coefficients for use in a model. This uses cross-validation at each step to choose the magnitude of the penalty. The reasoning is that in the first step, you cross-validate and will likely choose a large penalty to screen out irrelevant predictors; in the second step, you cross-validate and will likely pick a smaller penalty (and hence larger coefficients). This is mentioned briefly in Elements of Statistical Learning with a citation to Nicolai Meinshausen ("Relaxed Lasso." Computational Statistics & Data Analysis Volume 52, Issue 1, 15 September 2007, pp 374-393).

User @amoeba also suggests an L1-then-OLS pipeline; this might be nice because it only has 1 hyperparameter for the magnitude of the L1 penalty, so less fiddling would be required.

One problem that can arise with any "phased" analysis pipeline (that is, a pipeline which does some steps, and then some other steps separately) is that there's no "visibility" between those different phases (algorithms applied at each step). This means that one process inherits any data snooping that happened at the previous steps. This effect is not negligible; poorly-conceived modeling can result in garbage models.

One way to hedge against data-snooping side-effects is to cross-validate all of your choices. However, the increased computational costs can be prohibitive, depending on the scale of the data and the complexity of each step.

$\endgroup$
9
  • $\begingroup$ Sorry I didn't follow the reply to my 3rd point. Can you explain? $\endgroup$ Nov 28, 2015 at 17:23
  • 1
    $\begingroup$ It's all about properly accounting for optimism. For the same reason that we measure performance on out-of-sample data, it's necessary to do all filtering/preprocessing steps in a way that doesn't permit information leakage between steps. If you do feature selection on your whole data set and then run some analysis, you'll find signal in the noise. $\endgroup$
    – Sycorax
    Nov 28, 2015 at 17:25
  • $\begingroup$ Ok. Then what is the recommended approach to feature selection before running a ML model? $\endgroup$ Nov 28, 2015 at 17:31
  • 3
    $\begingroup$ My recommendation is "don't." See here for an example of how this can go awry: stats.stackexchange.com/questions/164048/… But this is sufficiently distinct from your initial question that you should simply ask a new question. (This is to your advantage, as you'll be able to accrue additional rep on the new question.) $\endgroup$
    – Sycorax
    Nov 28, 2015 at 17:54
  • 3
    $\begingroup$ (+1) I haven't seen L1-followed-by-L2 discussed in the literature, but it does make sense to me. There are L1-followed-by-OLS (aka "LARS-OLS hybrid") and L1-followed-by-L1 (relaxed lasso), so one could as well consider L1-followed-by-L2. As long as both hyperparameters are cross-validated, it should be a viable regularization strategy. $\endgroup$
    – amoeba
    Apr 1, 2019 at 22:42
30
$\begingroup$

Generally speaking if you want optimum prediction use L2. If you want parsimony at some sacrifice of predictive discrimination use L1. But note that the parsimony can be illusory, e.g., repeating the lasso process using the bootstrap will often reveal significant instability in the list of features "selected" especially when predictors are correlated with each other.

$\endgroup$
7
  • 1
    $\begingroup$ "Optimum prediction" - You mean L2 in general gives better accuracy on unseen data? $\endgroup$ Nov 28, 2015 at 17:21
  • 3
    $\begingroup$ Yes, especially with regard to predictive discrimination. $\endgroup$ Nov 28, 2015 at 17:54
  • 1
    $\begingroup$ +1 to this comment (and the answer). I have come across this phenomenon of $L_2$ being usually better in terms of in predictive discrimination (ie. classification tasks) than $L_1$ and it always kind of annoys me.. I bootstrap intensively only for my elastic-net to propose a near or fully ridge solution. :) $\endgroup$
    – usεr11852
    Jul 14, 2017 at 22:56
  • 2
    $\begingroup$ Predictive discrimination is much more general a concept than classification. But to your point, $L_2$ is usually better than $L_1$ because it doesn't spend any information trying to be parsimoneous. It allows lots of little effects to add up. $\endgroup$ Jul 15, 2017 at 11:39
  • 1
    $\begingroup$ Predictive discrimination is the degree to which predictive signals can separate those with good outcomes from those with worse outcomes. The most popular measures of discrimination are $R^2$ and $c$-index (concordance probability; equal to AUROC when Y is binary). Rank correlations between X and Y are measures of predictive discrimination. This is more general than classification as it takes into account tendencies/gray zones as in probability models. See fharrell.com/post/addvalue $\endgroup$ Aug 24 at 3:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.