45

1. Which method is preferred? Yes, elastic net is always preferred over lasso & ridge regression because it solves the limitations of both methods, while also including each as special cases. So if the ridge or lasso solution is, indeed, the best, then any good model selection routine will identify that as part of the modeling process. Comments to my ...


43

Part 1 In the elastic net two types of constraints on the parameters are employed Lasso constraints (i.e. on the size of the absolute values of $\beta_j$) Ridge constraints (i.e. on the size of the squared values of $\beta_j$) $\alpha$ controls the relative weighting of the two types. The Lasso constraints allow for the selection/removal of variables in ...


42

In The Elements of Statistical Learning book, Hastie et al. provide a very insightful and thorough comparison of these shrinkage techniques. The book is available online (pdf). The comparison is done in section 3.4.3, page 69. The main difference between Lasso and Ridge is the penalty term they use. Ridge uses $L_2$ penalty term which limits the size of the ...


26

I emailed this question to Zou and to Hastie and got the following reply from Hastie (I hope he wouldn't mind me quoting it here): I think in Zou et al we were worried about the additional bias, but of course rescaling increases the variance. So it just shifts one along the bias-variance tradeoff curve. We will soon be including a version of relaxed ...


25

To summarize, here are some salient differences between Lasso, Ridge and Elastic-net: Lasso does a sparse selection, while Ridge does not. When you have highly-correlated variables, Ridge regression shrinks the two coefficients towards one another. Lasso is somewhat indifferent and generally picks one over the other. Depending on the context, one does not ...


23

Friedman, Hastie, and Tibshirani (2010), citing The Elements of Statistical Learning, write, We often use the “one-standard-error” rule when selecting the best model; this acknowledges the fact that the risk curves are estimated with error, so errs on the side of parsimony. The reason for using one standard error, as opposed to any other amount, seems to ...


23

Suppose you have two highly correlated predictor variables $x,z$, and suppose both are centered and scaled (to mean zero, variance one). Then the ridge penalty on the parameter vector is $\beta_1^2 + \beta_2^2$ while the lasso penalty term is $ \mid \beta_1 \mid + \mid \beta_2 \mid$. Now, since the model is supposed highly colinear, so that $x$ and $z$ ...


20

How bridge regression and elastic net differ is a fascinating question, given their similar-looking penalties. Here's one possible approach. Suppose we solve the bridge regression problem. We can then ask how the elastic net solution would differ. Looking at the gradients of the two loss functions can tell us something about this. Bridge regression Say $X$ ...


18

I think the crucial part to consider in answering your question is I'm trying to identify the best model to predict the prices of automobiles because this statement implies something about why you want to use the model. Model choice and evaluation should be based on what you want to achieve with your fitted values. First, lets recap what $R^2$ does: It ...


16

If you order 1 million ridge-shrunk, scaled, but non-zero features, you will have to make some kind of decision: you will look at the n best predictors, but what is n? The LASSO solves this problem in a principled, objective way, because for every step on the path (and often, you'd settle on one point via e.g. cross validation), there are only m coefficients ...


15

Brief answers to your questions: Lasso and adaptive lasso are different. (Check Zou (2006) to see how adaptive lasso differs from standard lasso.) Lasso is a special case of elastic net. (See Zou & Hastie (2005).) Adaptive lasso is not a special case of elastic net. Elastic net is not a special case of lasso or adaptive lasso. Function glmnet in "...


14

Zou and Hastie in their paper proposing the method give the following explanation: In this paper we propose a new regularization technique which we call the elastic net. Similar to the lasso, the elastic net simultaneously does automatic variable selection and continuous shrinkage, and it can select groups of correlated variables. It is like a stretchable ...


12

LASSO solutions are solutions that minimize $$Q(\beta|X,y) = \dfrac{1}{2n}||y-X\beta||^2 + \lambda\sum_{j}|\beta_j|$$ the adaptive lasso simply adds weights to this to try to counteract the known issue of LASSO estimates being biased. $$Q_a(\beta|X,y,w) = \dfrac{1}{2n}||y-X\beta||^2 + \lambda\sum_{j}w_j|\beta_j|$$ Often you will see $w_j = 1/\tilde{\...


11

Suppose two predictors have a strong effect on the response but are highly correlated in the sample from which you build your model. If you drop one from the model it won't predict well for samples from similar populations in which the predictors aren't highly correlated. If you want to improve the precision of your coefficient estimates in the presence of ...


11

"Sparse Algorithms are not Stable: A No-free-lunch Theorem" I guess the title says a lot, as you pointed out. [...] a sparse algorithm can have non-unique optimal solutions, and is therefore ill-posed Check out randomized lasso, and the talk by Peter Buhlmann. Update: I found this paper easier to follow than the paper by Meinshausen and Buhlmann ...


11

You're confused; $\alpha$ and $\lambda$ are totally different. $\alpha$ sets the degree of mixing between ridge regression and lasso: when $\alpha = 0$, the elastic net does the former, and when $\alpha = 1$, it does the latter. Values of $\alpha$ between those extremes will give a result that is a blend of the two. Meanwhile, $\lambda$ is the shrinkage ...


11

I generally agree with @Sycorax answer, but I would like to add some qualification. Saying that "elastic net is always preferred over lasso & ridge regression" may be a little too strong. In small or medium samples elastic net may not select pure LASSO or pure ridge solution even if the former or the latter is actually the relevant one. Given strong ...


11

tl;dr version: The objective implicitly contains a scaling factor $\hat{s} = sd(y)$, where $sd(y)$ is the sample standard deviation. Longer version If you read the fine print of the glmnet documentation, you will see: Note that the objective function for ‘"gaussian"’ is 1/2 RSS/nobs + lambda*penalty, and for ...


11

Since you're simply looking for references, here is the list: Tikhonov, Andrey Nikolayevich (1943). "Об устойчивости обратных задач" [On the stability of inverse problems]. Doklady Akademii Nauk SSSR. 39 (5): 195–198. Tikhonov, A. N. (1963). "О решении некорректно поставленных задач и методе регуляризации". Doklady Akademii Nauk SSSR. 151: 501–504.. ...


10

Your question has an implicit assumption that $R^2$ is a good measure of the quality of the fit and is appropriate for comparing between models. I think that your background information provides evidence that $R^2$ is not a good tool for what you are trying to do. After all, you can increase $R^2$ by adding nonsense variables to your model. Did you take ...


10

I'd like to try answer your main question, here are two options: Use the one-standard-error (1SE) rule When cross-validating for selection purposes, it can help to use the 1SE rule. The standard error of the CV estimate is calculated for each fold. Instead of selecting the model corresponding to the minimum CV error, use the most parsimonious model where ...


10

The more technical answer is because the constrained optimization problem can be written in terms of Lagrange multipliers. In particular, the Lagrangian associated with the constrained optimization problem is given by $$\mathcal L(\beta) = \underset{\beta}{\mathrm{argmin}}\,\left\{\sum_{i=1}^N \left(y_i - \sum_{j=1}^p x_{ij} \beta_j\right)^2\right\} + \mu \...


9

What you're doing with elastic, ridge, or lasso, using cross-validation to choose regularization parameters, is fitting some linear form to optimize prediction. Why these particular regularization parameters? Because they work best for prediction on new data. Shrinking coefficient estimates towards zero, introducing bias, (as is done in either Ridge or Lasso)...


9

Cross-validation is a noisy process and you shouldn't expect the results from two runs to be similar, even if everything is working fine. You can try repeating your experiment several times and see what happens. That said, here's a narrow answer to this specific question: Question: How could I tune alpha and lambda for an elastic net in R? My ...


8

These methods--the lasso and elastic net--were born out of the problems of both feature selection and prediction. It's through these two lenses that I think an explanation can be found. Matthew Gunn nicely explains in his reply that these two goals are distinct and often taken up by different people. However, fortunately for us, the methods we're ...


8

The term "regularization" covers a very wide variety of methods. For the purpose of this answer, I am going to narrow in to mean "penalized optimization", i.e. adding an $L_1$ or $L_2$ penalty to your optimization problem. If that's the case, then the answer is a definitive "Yes! Well kinda". The reason for this is that adding an $L_1$ or $L_2$ penalty ...


8

There is a major difference between performing estimating using ridge type penalties and lasso-type penalties. Ridge type estimators tend to shrink all regression coefficients towards zero and are biased, but have an easy to derive asymptotic distribution because they do not shrink any variable to exactly zero. The bias in the ridge estimates may be ...


8

One disadvantage is the computational cost. You need to cross-validate the relative weight of L1 vs. L2 penalty, $\alpha$, and that increases the computational cost by the number of values in the $\alpha$ grid. Another disadvantage (but at the same time an advantage) is the flexibility of the estimator. With greater flexibility comes increased probability ...


7

Clarifying what is meant by $\alpha$ and Elastic Net parameters Different terminology and parameters are used by different packages, but the meaning is generally the same: The R package Glmnet uses the following definition $\min_{\beta_0,\beta} \frac{1}{N} \sum_{i=1}^{N} w_i l(y_i,\beta_0+\beta^T x_i) + \lambda\left[(1-\alpha)||\beta||_2^2/2 + \alpha ||\...


7

This is not a solution but some reflections on the possibilities and difficulties that I know of. Whenever it is possible to specify a time-series model as $$Y_{t+1} = \mathbf{x}_t \beta + \epsilon_{t+1}$$ with $\mathbf{x}_t$ computable from covariates and time-lagged observations, it is also possible to compute the least-squares elastic net penalized ...


Only top voted, non community-wiki answers of a minimum length are eligible