35
votes
Accepted
Why Lasso or ElasticNet perform better than Ridge when the features are correlated
Suppose you have two highly correlated predictor variables $x,z$, and suppose both are centered and scaled (to mean zero, variance one). Then the ridge penalty on the parameter vector is $\beta_1^2 + ...
- 82.8k
35
votes
Accepted
Why does glmnet use "naive" elastic net from the Zou & Hastie original paper?
I emailed this question to Zou and to Hastie and got the following reply from Hastie (I hope he wouldn't mind me quoting it here):
I think in Zou et al we were worried about the additional bias, but ...
- 107k
34
votes
Ridge, lasso and elastic net
To summarize, here are some salient differences between Lasso, Ridge and Elastic-net:
Lasso does a sparse selection, while Ridge does not.
When you have highly-correlated variables, Ridge regression ...
- 441
21
votes
Accepted
Why is Elastic Net called Elastic Net?
Zou and Hastie in their paper proposing the method give the following explanation:
In this paper we propose a new regularization technique which we call the elastic net. Similar to the lasso, the ...
- 1,037
16
votes
Accepted
Choosing optimal alpha in elastic net logistic regression
Clarifying what is meant by $\alpha$ and Elastic Net parameters
Different terminology and parameters are used by different packages, but the meaning is generally the same:
The R package Glmnet uses ...
- 9,986
16
votes
Accepted
Any disadvantages of elastic net over lasso?
One disadvantage is the computational cost. You need to cross-validate the relative weight of L1 vs. L2 penalty, $\alpha$, and that increases the computational cost by the number of values in the $\...
- 69.5k
14
votes
Using regularization when doing statistical inference
There is a major difference between performing estimating using ridge type penalties and lasso-type penalties. Ridge type estimators tend to shrink all regression coefficients towards zero and are ...
- 798
12
votes
Tune alpha and lambda parameters of elastic nets in an optimal way
Cross-validation is a noisy process and you shouldn't expect the results from two runs to be similar, even if everything is working fine. You can try repeating your experiment several times and see ...
- 8,349
12
votes
Accepted
Replicating results for glmnet linear regression using a generic optimizer
tl;dr version:
The objective implicitly contains a scaling factor $\hat{s} = sd(y)$, where $sd(y)$ is the sample standard deviation.
Longer version
If you read the fine print of the glmnet ...
- 3,004
12
votes
Accepted
What are some of the most important "early papers" on Regularization methods?
Since you're simply looking for references, here is the list:
Tikhonov, Andrey Nikolayevich (1943). "Об устойчивости обратных задач" [On the stability of inverse problems]. Doklady Akademii Nauk SSSR....
12
votes
Showing the Equivalence Between the $ {L}_{2} $ Norm Regularized Regression and $ {L}_{2} $ Norm Constrained Regression Using KKT
The more technical answer is because the constrained optimization problem can be written in terms of Lagrange multipliers. In particular, the Lagrangian associated with the constrained optimization ...
- 2,515
11
votes
Lasso vs. adaptive Lasso
Adaptive LASSO is used for consistent variable selection. The problems we encounter when using the LASSO for variable selection are:
The shrinkage parameter must be larger for selection than ...
- 720
11
votes
Why is lambda "within one standard error from the minimum" is a recommended value for lambda in an elastic net regression?
Breiman et al.'s book (cited in the other answer's quote from Krstajic) is the oldest reference I've found for the 1SE rule.
This is Breiman, Friedman, Stone, and Olshen's Classification and ...
- 4,603
11
votes
Using regularization when doing statistical inference
The term "regularization" covers a very wide variety of methods. For the purpose of this answer, I am going to narrow in to mean "penalized optimization", i.e. adding an $L_1$ or $L_2$ penalty to your ...
- 21.6k
10
votes
Elastic/ridge/lasso analysis, what then?
What you're doing with elastic, ridge, or lasso, using cross-validation to choose regularization parameters, is fitting some linear form to optimize prediction. Why these particular regularization ...
- 23k
10
votes
How to interpret coefficients of a multinomial elastic net (glmnet) regression
I emailed kind Dr. Hastie who is the maintainer of the glmnet package and got the following answer:
In the traditional case, the base category is arbitrary.
In ...
- 689
9
votes
Accepted
Elastic/ridge/lasso analysis, what then?
These methods--the lasso and elastic net--were born out of the problems of both feature selection and prediction. It's through these two lenses that I think an explanation can be found.
Matthew Gunn ...
- 2,972
9
votes
Accepted
Is there a Bayesian interpretation of linear regression with simultaneous L1 and L2 regularization (aka elastic net)?
Ben's comment is likely sufficient, but I provide some more references one of which is from before the paper Ben referenced.
A Bayesian elastic net representation was proposed by Kyung et. al. in ...
- 16k
8
votes
Accepted
Group elastic net
Let $\mathcal{G}$ be the grouping that you're interested in; that is, let $\mathcal{G}$ be a partition of $\{1, \dots, p\}$, where we consider there to be $p$ features. With response $y \in \mathbb{R}^...
- 2,972
8
votes
Glmnet: How to select Lambda and Alpha
It appears that the default in glmnet is to select lambda from a range of values from min.lambda to max.lambda, then the optimal ...
- 221
8
votes
Accepted
How is `tol` used in scikit-learn's `Lasso` and `ElasticNet`?
I am going to explain the case of Lasso, you can apply the same logic to ElasticNet.
How is the duality gap defined in the case of Lasso (/ElasticNet)?
The duality gap is the difference between a ...
- 96
8
votes
Accepted
How can a glmnet model with no coefficients have perfect performance?
It looks like in your data there is no relationship between the covariates and the outcome. I imagine that the model is discovering that and shrinks the coefficients to 0. If you fit on all the data ...
- 39.6k
7
votes
Why does glmnet use coordinate descent for Ridge regression?
I think this is due to speed. Cyclical coordinate descent does not find the exact solution in finite time, but it is faster, not only for a grid of $\lambda$'s but also for a single $\lambda$.
...
- 156
7
votes
Accepted
Can scikit-learn's ElasticNetCV be used for classification problems?
You can use the elasticnet penalty in sklearn's Logistic Regression classifier:
...
- 101
7
votes
Accepted
Math behind applying elastic net penalties to logistic regression
The elastic net terms are added to the maximum likelihood cost function.i.e. the final cost function is:
$\sum_{i = 0}^{N}\bigg[- (y\log(p) + (1-y)\log(1-p))\bigg] + \lambda_1 \sum_{i=0}^{k}|w_i| + \...
- 1,128
7
votes
Why l2 norm squared but l1 norm not squared?
But in the ElasticNet and Ridge, we use the l2 norm squared. Why is that, is there a particular reason (computational, optimization dynamics, statistical?)
A possible reason for the l2 norm being ...
- 86.6k
6
votes
How does glmnet handle larger datasets?
glmnet is very strong in this respect. Of course it depends on the situation but to give you an impression of its performance, I made an example with 3 mio lines ...
- 12.1k
6
votes
Accepted
Cross-validation for elastic net regression: squared error vs. correlation on the test set
I think I figured out what was happening here.
Note that the value of correlation does not depend on the length of $\hat\beta$. So if the test correlation keeps increasing while the test R-squared ...
- 107k
6
votes
Accepted
Biased prediction (overestimation) for xgboost
What is described is not very surprising.
Boosting methods usually do not give well calibrated probabilistic predictions (e.g. see
Caruana et al. (2004) "Ensemble Selection from Libraries of Models",...
- 46.1k
6
votes
Using elastic-net only for feature selection
This is not a good approach, as it is essentially leaking information from the test data.
The resulting coefficients will be overestimated, because no regularization is applied anymore, and yet the ...
- 14.3k
Only top scored, non community-wiki answers of a minimum length are eligible