Questions tagged [elastic-net]

A regularization method for regression models that combines the penalties of lasso and of ridge regression.

Filter by
Sorted by
Tagged with
45 votes
4 answers
40k views

Ridge, lasso and elastic net

How do ridge, LASSO and elasticnet regularization methods compare? What are their respective advantages and disadvantages? Any good technical paper, or lecture notes would be appreciated as well.
user3269's user avatar
  • 5,162
44 votes
2 answers
43k views

What is elastic net regularization, and how does it solve the drawbacks of Ridge ($L^2$) and Lasso ($L^1$)?

Is elastic net regularization always preferred to Lasso & Ridge since it seems to solve the drawbacks of these methods? What is the intuition and what is the math behind elastic net?
GeorgeOfTheRF's user avatar
41 votes
2 answers
24k views

Why is lambda "within one standard error from the minimum" is a recommended value for lambda in an elastic net regression?

I understand what role lambda plays in an elastic-net regression. And I can understand why one would select lambda.min, the value of lambda that minimizes cross validated error. My question is Where ...
jhersh's user avatar
  • 413
35 votes
1 answer
3k views

Why does glmnet use "naive" elastic net from the Zou & Hastie original paper?

The original elastic net paper Zou & Hastie (2005) Regularization and variable selection via the elastic net introduced elastic net loss function for linear regression (here I assume all ...
amoeba's user avatar
  • 105k
33 votes
2 answers
49k views

Choosing optimal alpha in elastic net logistic regression

I'm performing an elastic-net logistic regression on a health care dataset using the glmnet package in R by selecting lambda values over a grid of $\alpha$ from 0 ...
RobertF's user avatar
  • 6,104
31 votes
1 answer
23k views

Feature selection & model with glmnet on Methylation data (p>>N)

I would like to use GLM and Elastic Net to select those relevant features + build a linear regression model (i.e., both prediction and understanding, so it would be better to be left with relatively ...
PGreen's user avatar
  • 565
31 votes
2 answers
6k views

Fitting an ARIMAX model with regularization or penalization (e.g. with the lasso, elastic net, or ridge regression)

I use the auto.arima() function in the forecast package to fit ARMAX models with a variety of covariates. However, I often have a large number of variables to select from and usually end up with a ...
Zach's user avatar
  • 23.8k
27 votes
1 answer
5k views

Bridge penalty vs. Elastic Net regularization

Some penalty functions and approximations are well studied, such as the LASSO ($L_1$) and the Ridge ($L_2$) and how these compare in regression. I've been reading about the Bridge penalty, which is ...
Firebug's user avatar
  • 19.1k
26 votes
2 answers
17k views

Why Lasso or ElasticNet perform better than Ridge when the features are correlated

I have a set of 150 features, and many of them are highly correlated with each other. My goal is to predict the value of a discrete variable, whose range is 1-8. My sample size is 550, and I am using ...
renakre's user avatar
  • 857
25 votes
3 answers
5k views

Using regularization when doing statistical inference

I know about the benefits of regularization when building predictive models (bias vs. variance, preventing overfitting). But, I'm wondering if it is a good idea to also do regularization (lasso, ...
user162381's user avatar
22 votes
3 answers
24k views

Lasso vs. adaptive Lasso

LASSO and adaptive LASSO are two different things, right? (To me, the penalties look different, but I'm just checking whether I miss something.) When you generally speak about elastic net, is the ...
Mr Validation's user avatar
22 votes
1 answer
28k views

Is R-squared value appropriate for comparing models?

I'm trying to identify the best model to predict the prices of automobiles, using the prices and features available on automobile classified advertisement sites. For this I used a couple of models ...
Manik's user avatar
  • 321
22 votes
3 answers
4k views

Model stability when dealing with large $p$, small $n$ problem

Intro: I have a dataset with a classical "large p, small n problem". The number available samples n=150 while the number of possible predictors p=400. The outcome is a continuous variable. I want ...
dimi's user avatar
  • 321
21 votes
1 answer
3k views

Is there a Bayesian interpretation of linear regression with simultaneous L1 and L2 regularization (aka elastic net)?

It's well known that linear regression with an $l^2$ penalty is equivalent to finding the MAP estimate given a Gaussian prior on the coefficients. Similarly, using an $l^1$ penalty is equivalent to ...
Michael Curry's user avatar
20 votes
2 answers
4k views

Elastic/ridge/lasso analysis, what then?

I'm getting really interested in the elastic net procedure for predictor shrinkage/selection. It seems very powerful. But from the scientific point of view I don't know well what to do once I got the ...
Bakaburg's user avatar
  • 2,917
19 votes
2 answers
12k views

Why can't ridge regression provide better interpretability than LASSO?

I already have an idea about pros and cons of ridge regression and the LASSO. For the LASSO, L1 penalty term will yield a sparse coefficient vector, which can be viewed as a feature selection method. ...
Brad Li's user avatar
  • 330
18 votes
1 answer
1k views

Is there a clear set of conditions under which lasso, ridge, or elastic net solution paths are monotone?

The question What to conclude from this lasso plot (glmnet) demonstrates solution paths for the lasso estimator that are not monotonic. That is, some of the cofficients grow in absolute value before ...
shadowtalker's user avatar
  • 12.6k
15 votes
3 answers
3k views

Showing the Equivalence Between the $ {L}_{2} $ Norm Regularized Regression and $ {L}_{2} $ Norm Constrained Regression Using KKT

According to the references Book 1, Book 2 and paper. It has been mentioned that there is an equivalence between the regularized regression (Ridge, LASSO and Elastic Net) and their constraint ...
jeza's user avatar
  • 2,089
15 votes
1 answer
9k views

Difference between ElasticNet in scikit-learn Python and Glmnet in R

Has anybody tried to verify whether fitting an Elastic Net model with ElasticNet in scikit-learn in Python and glmnet in R on ...
Dion's user avatar
  • 954
14 votes
3 answers
7k views

Confusion related to elastic net

I was reading this article related to elastic net. They say that they use elastic net because if we just use Lasso it tends to select only one predictor among the predictors that are highly correlated....
user31820's user avatar
  • 1,501
14 votes
2 answers
7k views

Coordinate descent for the lasso or elastic net

Are there any good papers or books dealing with the use of coordinate descent for L1 (lasso) and/or elastic net regularization for linear regression problems?
NPE's user avatar
  • 5,591
14 votes
1 answer
7k views

How is `tol` used in scikit-learn's `Lasso` and `ElasticNet`?

As a followup to this question, how does scikit-learn implementation of Lasso (and coordinate_descent algorithm) uses the tol parameter in practice? More precisely, ...
Phylliade's user avatar
  • 253
13 votes
3 answers
1k views

Why l2 norm squared but l1 norm not squared?

In the Lasso, and ElasticNet, we use, as penalty, the l1 norm without squaring. But in the ElasticNet and Ridge, we use the l2 norm squared. Why is that, is there a particular reason (computational, ...
William de Vazelhes's user avatar
11 votes
1 answer
1k views

Why is Elastic Net called Elastic Net?

What is the etymology of "Elastic Net" in Elastic Net Regularization? Does it have anything to do with the name of "lasso"? Related: Why is ridge regression called "ridge&...
brazofuerte's user avatar
11 votes
2 answers
8k views

Any disadvantages of elastic net over lasso?

What are the disadvantages of using elastic net in comparison to lasso. I know that the elastic net is able to select groups of variables when they are highly correlated. It doesn't have the ...
Ville's user avatar
  • 857
11 votes
2 answers
6k views

How is the intercept computed in GLMnet?

I've been implementing the GLMNET version of elastic net for linear regression with another software than R. I compared my results with the R function glmnet in lasso mode on diabetes data. The ...
yelh's user avatar
  • 331
11 votes
1 answer
1k views

Replicating results for glmnet linear regression using a generic optimizer

As the title states, I'm trying to replicate the results from glmnet linear using the LBFGS optimizer from library lbfgs. This optimizer allows us to add an L1 ...
user3294195's user avatar
10 votes
2 answers
906 views

What are some of the most important "early papers" on Regularization methods?

In several answers I have seen CrossValidated users suggest OP find early papers on Lasso, Ridge, and Elastic Net. For posterity, what are the seminal works on Lasso, Ridge, and Elastic Net?
10 votes
1 answer
3k views

How to interpret coefficients of a multinomial elastic net (glmnet) regression

I'm trying to model a membership in one of three well-being clusters (flourisher, normative, languisher) based on a set of predictors, using elastic net for both variable selection & modelling. I ...
Adam B.'s user avatar
  • 649
10 votes
1 answer
1k views

Penalized methods for categorical data: combining levels in a factor

Penalized models can be used to estimate models where the number of parameters is equal to or even greater than the sample size. This situation can arise in log-linear models of large sparse tables of ...
andrewH's user avatar
  • 3,117
9 votes
3 answers
9k views

Cross validation with two parameters: elastic net case

I want to know the cross validation procedure to find the two parameters of elastic net presented by Zou and Hastie on the prostate dataset as example. I can't improve the error rate lasso with k-fold ...
grant's user avatar
  • 1,541
9 votes
1 answer
4k views

Stepwise regression vs. elastic net

I understand that Stepwise regression analysis has lots of limitations, including the assumption that the predictors are not highly correlated with each other. In fact, this limitation was the most ...
Niousha's user avatar
  • 361
9 votes
2 answers
2k views

Why does glmnet use coordinate descent for Ridge regression?

If I understand it correctly, glmnet uses cyclical coordinate descent not only for lasso and elastics nets, but also for Ridge regression. Why does it use this algorithm, which sometimes gives ...
Benykō-Zamurai's user avatar
9 votes
1 answer
4k views

Group elastic net

The lasso and the elastic net are not able to handle variables with more than two categories and therefore a split of categorical variables into dummies is necessary for the application of these ...
Joachim Schork's user avatar
9 votes
1 answer
2k views

Range of lambda in elastic net regression

$\def\l{|\!|}$ Given the elastic net regression $$\min_b \frac{1}{2}\l y - Xb \l^2 + \alpha\lambda \l b\l_2^2 + (1 - \alpha) \lambda \l b\l_1$$ how can an appropriate range of $\lambda$ be chosen ...
Chris Taylor's user avatar
  • 3,682
9 votes
1 answer
142 views

What is the correct way to write the elastic net?

I am confused about the correct way to write the elastic net. After reading some research papers there seems to be three forms 1) $\exp\{-\lambda_1|\beta_k|-\lambda_2\beta_k^2\}$ 2) $\exp\{-\frac{(\...
gbd's user avatar
  • 273
9 votes
0 answers
2k views

Selecting regularization penalty: cross validation or information criteria?

I will use an elastic net to estimate a regression model which will later be used for forecasting. I have a grid of $\alpha$ values within [0,1] representing the proportion of $L_1$ versus $L_2$ ...
Richard Hardy's user avatar
8 votes
1 answer
2k views

Grid fineness and overfitting when tuning $\lambda$ in LASSO, ridge, elastic net

I wonder about the optimal grid fineness and what the relation between grid fineness and overfitting is in regularization methods such as LASSO, ridge regression or elastic net. Suppose I want ...
Richard Hardy's user avatar
8 votes
1 answer
6k views

Is feature selection with dummy coding of categorical variables problematic? [duplicate]

In the context of feature selection it is common to recode categorical variables with more than 2 categories into dummies. Selection methods such as elastic nets or lasso regression select the best ...
Joachim Schork's user avatar
8 votes
1 answer
2k views

Equivalence between Elastic Net formulations

According to Hastie's paper, the elastic net has two equivalent formulations: $$\hat{\beta} = \underset{\beta}{\operatorname{argmin}} \left\{ \sum_{i=1}^N\left(y_i-\sum_{j=1}^p x_{ij} \beta_j\right)^...
skd's user avatar
  • 344
8 votes
1 answer
2k views

Calculating $R^2$ for Elastic Net

I am trying to do variable selection using elastic net (Matlab Lasso function with alpha of 0.5). I have 75 predictors in total (some are correlated with each other,...
Niousha's user avatar
  • 361
8 votes
1 answer
2k views

Elastic net arbitrary alpha selection

I'm trying to solve a prediction problem given the following constraints: I need an interpretable model to be used for experimental validation I need a model that performs feature selection to reduce ...
A Magen's user avatar
  • 91
8 votes
2 answers
1k views

Post processing random forests using regularised regression: what about bias?

I have been playing around with post processing the results of the random forest for regression machine learning algorithm in order to try and do better than the default mean of all trees prediction. ...
Bogdanovist's user avatar
  • 6,619
7 votes
1 answer
15k views

Tune alpha and lambda parameters of elastic nets in an optimal way

I am trying to tune alpha and lambda parameters for an elastic net based on the glmnet package. I found some sources, which propose different options for that ...
Joachim Schork's user avatar
7 votes
1 answer
4k views

Gradient descent and elastic-net logistic regression

I'm currently in the process of trying to understand the paper Regularization Paths for Generalized Linear Models via Coordinate Descent by Friedman et al. with regard to the regularization of ...
sebp's user avatar
  • 2,097
7 votes
1 answer
2k views

Biased prediction (overestimation) for xgboost

I run xgboost and elastic-net on the same dataset for a classification problem, say we have ...
Salty Gold Fish's user avatar
7 votes
2 answers
3k views

How to find the smallest $\lambda$ such that all Lasso / Elastic Net coefficients are zero?

In the documentation to R's glmnet package it states that when fitting an elastic net, the glmnet function will use a series of $\lambda$ values starting at the smallest $\lambda$ for which all ...
badmax's user avatar
  • 2,211
7 votes
1 answer
2k views

How to select the final model with elastic net feature selection, cross validation and SVM?

I have a dataset of some 100 samples, each with >10,000 features, some of which highly correlated. Here's what I am doing currently. Split the data set into three folds. For each fold, 2.1 Run ...
user9563's user avatar
7 votes
1 answer
580 views

Cross-validation for elastic net regression: squared error vs. correlation on the test set

Consider elastic net regression with glmnet-like parametrization of the loss function$$\mathcal L = \frac{1}{2n}\big\lVert y - \beta_0-X\beta\big\rVert^2 + \lambda\...
amoeba's user avatar
  • 105k
7 votes
1 answer
2k views

R glmnet and elasticnet gives different results, why?

My question is simple: when glmnet use alpha between 0 and 1 (i.e. elastic net), is it returning naive elastic net or adjusted one? Especially, for every method inbuilt (coef, cv.glmnet, predict) is ...
Yuan Tao's user avatar

1
2 3 4 5
7