Note: I know that L1 has feature selection property. I am trying to understand which one to choose when feature selection is completely irrelevant.

  1. How to decide which regularization (L1 or L2) to use?
  2. What are the pros & cons of each of L1 / L2 regularization?
  3. Is it recommended to 1st do feature selection using L1 & then apply L2 on these selected variables?
  • 2
    $\begingroup$ Note that "L1 feature selection" should be rather called regularisation of the feature space; there are many way better methods to do feature selection understood as getting information what is relevant to the modelled problem. $\endgroup$ – user88 Nov 28 '15 at 18:07
  • $\begingroup$ @mbq: I am curious which "many way better methods" did you mean here? $\endgroup$ – amoeba says Reinstate Monica Nov 29 '15 at 1:47
  • 1
    $\begingroup$ Like those enumerated here. $\endgroup$ – user88 Nov 29 '15 at 17:58

How to decide which regularization (L1 or L2) to use?

What is your goal? Both can improve model generalization by penalizing coefficients, since features with opposite relationship to the outcome can "offset" each other (a large positive value is counterbalanced by a large negative value). This can arise when there are collinear features. Small changes in the data can result in dramatically different parameter estimates (high variance estimates). Penalization can restrain both coefficients to be smaller. (Hastie et al, Elements of Statistical Learning, 2nd edition, p. 63)

What are the pros & cons of each of L1 / L2 regularization?

L1 regularization can address the multicollinearity problem by constraining the coefficient norm and pinning some coefficient values to 0. Computationally, Lasso regression (regression with an L1 penalty) is a quadratic program which requires some special tools to solve. When you have more features than observations $N$, lasso will keep at most $N$ non-zero coefficients. Depending on context, that might not be what you want.

L1 regularization is sometimes used as a feature selection method. Suppose you have some kind of hard cap on the number of features you can use (because data collection for all features is expensive, or you have tight engineering constraints on how many values you can store, etc.). You can try to tune the L1 penalty to hit your desired number of non-zero features.

L2 regularization can address the multicollinearity problem by constraining the coefficient norm and keeping all the variables. It's unlikely to estimate a coefficient to be exactly 0. This isn't necessarily a drawback, unless a sparse coefficient vector is important for some reason.

In the regression setting, it's the "classic" solution to the problem of estimating a regression with more features than observations. L2 regularization can estimate a coefficient for each feature even if there are more features than observations (indeed, this was the original motivation for "ridge regression").

As an alternative, elastic net allows L1 and L2 regularization as special cases. A typical use-case in for a data scientist in industry is that you just want to pick the best model, but don't necessarily care if it's penalized using L1, L2 or both. Elastic net is nice in situations like these.

Is it recommended to 1st do feature selection using L1 & then apply L2 on these selected variables?

I'm not familiar with a publication proposing an L1-then-L2 pipeline, but this is probably just ignorance on my part. There doesn't seem to be anything wrong with it. I'd conduct a literature review.

A few examples of similar "phased" pipelines exist. One is the "relaxed lasso", which applies lasso regression twice, once to down-select from a large group to a small group of features, and second to estimate coefficients for use in a model. This uses cross-validation at each step to choose the magnitude of the penalty. The reasoning is that in the first step, you cross-validate and will likely choose a large penalty to screen out irrelevant predictors; in the second step, you cross-validate and will likely pick a smaller penalty (and hence larger coefficients). This is mentioned briefly in Elements of Statistical Learning with a citation to Nicolai Meinshausen ("Relaxed Lasso." Computational Statistics & Data Analysis Volume 52, Issue 1, 15 September 2007, pp 374-393).

User @amoeba also suggests an L1-then-OLS pipeline; this might be nice because it only has 1 hyperparameter for the magnitude of the L1 penalty, so less fiddling would be required.

One problem that can arise with any "phased" analysis pipeline that does some steps and then some other steps separately is that there's no "visibility" between those different algorithms, so one process inherits any data snooping that happened at the previous steps. This effect is not negligible; poorly-conceived modeling can result in garbage models.

One way to hedge against data-snooping side-effects is to cross-validate all of your choices. However, the increased computational costs can prove prohibitive.

  • $\begingroup$ Sorry I didn't follow the reply to my 3rd point. Can you explain? $\endgroup$ – GeorgeOfTheRF Nov 28 '15 at 17:23
  • 1
    $\begingroup$ It's all about properly accounting for optimism. For the same reason that we measure performance on out-of-sample data, it's necessary to do all filtering/preprocessing steps in a way that doesn't permit information leakage between steps. If you do feature selection on your whole data set and then run some analysis, you'll find signal in the noise. $\endgroup$ – Sycorax says Reinstate Monica Nov 28 '15 at 17:25
  • $\begingroup$ Ok. Then what is the recommended approach to feature selection before running a ML model? $\endgroup$ – GeorgeOfTheRF Nov 28 '15 at 17:31
  • 3
    $\begingroup$ My recommendation is "don't." See here for an example of how this can go awry: stats.stackexchange.com/questions/164048/… But this is sufficiently distinct from your initial question that you should simply ask a new question. (This is to your advantage, as you'll be able to accrue additional rep on the new question.) $\endgroup$ – Sycorax says Reinstate Monica Nov 28 '15 at 17:54
  • 3
    $\begingroup$ (+1) I haven't seen L1-followed-by-L2 discussed in the literature, but it does make sense to me. There are L1-followed-by-OLS (aka "LARS-OLS hybrid") and L1-followed-by-L1 (relaxed lasso), so one could as well consider L1-followed-by-L2. As long as both hyperparameters are cross-validated, it should be a viable regularization strategy. $\endgroup$ – amoeba says Reinstate Monica Apr 1 '19 at 22:42

Generally speaking if you want optimum prediction use L2. If you want parsimony at some sacrifice of predictive discrimination use L1. But note that the parsimony can be illusory, e.g., repeating the lasso process using the bootstrap will often reveal significant instability in the list of features "selected" especially when predictors are correlated with each other.

  • $\begingroup$ "Optimum prediction" - You mean L2 in general gives better accuracy on unseen data? $\endgroup$ – GeorgeOfTheRF Nov 28 '15 at 17:21
  • 3
    $\begingroup$ Yes, especially with regard to predictive discrimination. $\endgroup$ – Frank Harrell Nov 28 '15 at 17:54
  • 1
    $\begingroup$ +1 to this comment (and the answer). I have come across this phenomenon of $L_2$ being usually better in terms of in predictive discrimination (ie. classification tasks) than $L_1$ and it always kind of annoys me.. I bootstrap intensively only for my elastic-net to propose a near or fully ridge solution. :) $\endgroup$ – usεr11852 Jul 14 '17 at 22:56
  • 2
    $\begingroup$ Predictive discrimination is much more general a concept than classification. But to your point, $L_2$ is usually better than $L_1$ because it doesn't spend any information trying to be parsimoneous. It allows lots of little effects to add up. $\endgroup$ – Frank Harrell Jul 15 '17 at 11:39
  • $\begingroup$ Cool, thank you for the clarification. It is makes good sense. (Yes, you are right; I associate PD with measures relating to the probability of correct classification and/or ordering of subjects in terms of risk so I was quick to say "classification tasks"; my bad, I should be more careful.) $\endgroup$ – usεr11852 Jul 15 '17 at 20:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.