6
$\begingroup$

There is no shortage of resources on the advantages of using an elastic net penalty over L1 and L2 penalties (LASSO and ridge respectively).

The following questions pertain to situations where regularization is desirable, i.e., looking for answers contrasting elastic net with other regularization strategies.

When doing regression analysis, when should one not apply an elastic net penalty? Or more generally, what should one be wary of when choosing elastic net penalty over other regularization methods?

$\endgroup$
5
  • 1
    $\begingroup$ Good question. I don't there will be any example where you should choose a lasso or ridge regression over elastic net since can try penalties which correspond to ride and lasso when you tune your model. It would be interesting if someone can give a theory based argument though. $\endgroup$
    – Eli
    Jan 29, 2022 at 22:20
  • $\begingroup$ @Eli: Agreed. In that sense I almost never use LASSO on it's own because it quickly becomes unstable. Putting even a relatively small ridge regularisation make the paths much more stable. $\endgroup$
    – usεr11852
    Jan 29, 2022 at 22:24
  • 1
    $\begingroup$ As elastic net, unlike pure ridge, involves some predictor selection, this thread on that topic is worth a read. $\endgroup$
    – EdM
    Jan 29, 2022 at 22:37
  • $\begingroup$ I replied to your question. You may already know some (most??) of the information. I think it's worthwhile to write it out explicitly for others, though. $\endgroup$
    – Eli
    Feb 4, 2022 at 15:26
  • $\begingroup$ Closely related threat: stats.stackexchange.com/questions/184029/… $\endgroup$
    – Sycorax
    Feb 4, 2022 at 15:33

1 Answer 1

6
$\begingroup$

I am not aware of any practical situation where Ridge or Lasso are preferable to Elastic Net. The large-sample (asymptotic) theory for Ridge and Lasso seem to be better developed, so people may use them when they develop theory or if they want theoretical guarantees on the performance of their method.

As OP is already aware, Ridge and Lasso are a special case of Elastic Net. Elastic Net minimizes the function

$$ \hat{\beta} \equiv \underset{\beta}{\operatorname{argmin}}\left(\|y-X \beta\|^{2}+\lambda_{2}\|\beta\|^{2}+\lambda_{1}\|\beta\|_{1}\right).$$

When $\lambda_1 = 0$ and $\lambda_2 > 0$, Elastic Net becomes Ridge regression, and when $\lambda_2 = 0$ and $\lambda_1 >0$, it becomes Ridge regression. Generally, we select the best values of $\lambda_1$ and $\lambda_2$ using cross validation. If Ridge or Lasso were to outperform Elastic Net in a particular case, cross validation would choose a $\lambda_1$ or $\lambda_2$ that reduces the model to Ridge or Lasso.

In the special case when the design matrix is orthonormal, Zhou and Hastie give the closed form solution of each coefficient $\beta_i$ estimated by Elastic Net, Lasso, and Ridge:

$$\text{Elastic Net:} \enspace \hat{\beta}_{i} =\frac{\left(\mid \hat{\beta}_{i}(\text { OLS }) \mid-\lambda_{1} / 2\right)_{+}}{1+\lambda_{2}} \operatorname{sgn}\left\{\hat{\beta}_{i}(\text { OLS })\right\}$$

$$\text{Lasso:} \enspace \hat{\beta}_{i} \text { (lasso) }=\left(\mid \hat{\beta}_{i}(\text { OLS }) \mid-\lambda_{1} / 2\right)_{+} \operatorname{sgn}\left\{\hat{\beta}_{i}(\text { OLS })\right\}$$

$$\text{Ridge:} \enspace \hat{\boldsymbol{\beta}} \text { (ridge) }=\hat{\boldsymbol{\beta}} \text { (OLS) } /\left(1+\lambda_{2}\right)$$

Zhou and Hastie match up terms and plot the solution paths of each estimator to explain the behavior of Elastic Net. They state "elastic net can be viewed as a two-stage procedure: a ridge-type direct shrinkage followed by a lasso-type thresholding." This further supports the idea of Elastic Net always outperforming Ridge/Lasso. If either direct shrinkage or thresholding isn't necessary, you can simply omit this step with Elastic Net

The classic example of Ridge outperforming Lasso is when you have many correlated predictors. but this does not appear to negatively impact Elastic Net. In the same paper, Zhou and Hastie examine the problem of correlated predictors analytically and through simulations. They find that the predictive performance of Elastic Net isn't negatively impacted in the same way Lasso is by correlated predictors.

The only benefit of Ridge and Lasso over Elastic Net I'm aware of is that they can be fit faster than Elastic Net. Ridge and Lasso only have a single tuning parameter while Elastic Net has two tuning parameters. However, Elastic Net can be fit quickly with existing software so this may not be a meaningful increase in computation time.

There may be a pathological situation where Elastic Net performs worse than Ridge and Lasso. The only way this could happen is if something in the data causes a poor selection of $\lambda_1$ and $\lambda_2$.

EDIT: User @richard-hardy points out in a comment that Lasso/Ridge vs Elastic Net can be interpreted as a bias-variance tradeoff. The additional parameter in Elastic Net increases the variance of the model relative to Lasso and Ridge. That is, Elastic Net is more likely to overfit than Ridge or Lasso. Richard and I both suspect this isn't an issue in practice, and increased variance doesn't dominate the reduced bias of Elastic Net.

$\endgroup$
9
  • 2
    $\begingroup$ (+1) This is a good answer. A spirited debate about the relative merits of elastic net, lasso and ridge regression can be found in the answers and comments on this thread. stats.stackexchange.com/questions/184029/… $\endgroup$
    – Sycorax
    Feb 4, 2022 at 15:32
  • $\begingroup$ That's a great resource. $\endgroup$
    – Eli
    Feb 4, 2022 at 15:34
  • 2
    $\begingroup$ Being picky: there is no free lunch, and a bias-variance trade-off applies. Elastic net has greater variance than ridge or lasso due to an additional parameter (the relative weight of lasso vs. ridge) that has to be tuned. However, I suspect in most practical cases it would pay to use elastic net, as the increase in variance is not that great while the reduction in bias could be considerable. $\endgroup$ Feb 4, 2022 at 16:02
  • $\begingroup$ @Richard, that’s a good point. If you can point to a source that shows Elastic net has higher variance or provides the bias/variance decomposition of each estimator in special cases, I’ll amend my answer. $\endgroup$
    – Eli
    Feb 4, 2022 at 16:06
  • 1
    $\begingroup$ @user, I share your intuition. $\endgroup$ Feb 5, 2022 at 6:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.