When should I use lasso vs ridge?

Question

Say I want to estimate a large number of parameters, and I want to penalize some of them because I believe they should have little effect compared to the others. How do I decide what penalization scheme to use? When is ridge regression more appropriate? When should I use lasso?

Answer 1

Keep in mind that ridge regression can't zero out coefficients; thus, you either end up including all the coefficients in the model, or none of them. In contrast, the LASSO does both parameter shrinkage and variable selection automatically. If some of your covariates are highly correlated, you may want to look at the Elastic Net [3] instead of the LASSO.

I'd personally recommend using the Non-Negative Garotte (NNG) [1] as it's consistent in terms of estimation and variable selection [2]. Unlike LASSO and ridge regression, NNG requires an initial estimate that is then shrunk towards the origin. In the original paper, Breiman recommends the least-squares solution for the initial estimate (you may however want to start the search from a ridge regression solution and use something like GCV to select the penalty parameter).

In terms of available software, I've implemented the original NNG in MATLAB (based on Breiman's original FORTRAN code). You can download it from:

http://www.emakalic.org/blog/wp-content/uploads/2010/04/nngarotte.zip

BTW, if you prefer a Bayesian solution, check out [4,5].

References:

[1] Breiman, L. Better Subset Regression Using the Nonnegative Garrote Technometrics, 1995, 37, 373-384

[2] Yuan, M. & Lin, Y. On the non-negative garrotte estimator Journal of the Royal Statistical Society (Series B), 2007, 69, 143-161

[3] Zou, H. & Hastie, T. Regularization and variable selection via the elastic net Journal of the Royal Statistical Society (Series B), 2005, 67, 301-320

[4] Park, T. & Casella, G. The Bayesian Lasso Journal of the American Statistical Association, 2008, 103, 681-686

[5] Kyung, M.; Gill, J.; Ghosh, M. & Casella, G. Penalized Regression, Standard Errors, and Bayesian Lassos Bayesian Analysis, 2010, 5, 369-412

Answer 2

Ridge or lasso are forms of regularized linear regressions. The regularization can also be interpreted as prior in a maximum a posteriori estimation method. Under this interpretation, the ridge and the lasso make different assumptions on the class of linear transformation they infer to relate input and output data. In the ridge, the coefficients of the linear transformation are normal distributed and in the lasso they are Laplace distributed. In the lasso, this makes it easier for the coefficients to be zero and therefore easier to eliminate some of your input variable as not contributing to the output.

There are also some practical considerations. The ridge is a bit easier to implement and faster to compute, which may matter depending on the type of data you have.

If you have both implemented, use subsets of your data to find the ridge and the lasso and compare how well they work on the left out data. The errors should give you an idea of which to use.

Answer 3

Generally, when you have many small/medium sized effects you should go with ridge. If you have only a few variables with a medium/large effect, go with lasso. Hastie, Tibshirani, Friedman

Answer 4

While the answers here do provide some interesting/useful insights about the two techniques, I don't believe the difference between techniques was thoroughly discussed, nor when to use one over the other. There are hints at answers to this question around here, but I believe they are somewhat limited in how they answer OP's question.

Defining Ridge and Lasso

One of the core parts of both methods is reinterpreting the residual sum of squares (RSS), normally defined in ordinary least squares (OLS) as:

$$ \text{RSS} = e^2_1 + e^2_2 + e^2_3 + ... + e^2_k $$

where the $e^2_k$ is a squared residual (error) and the RSS as a sum approximates the variability in the response that can't be explained by a regression model. The formulation of ridge and lasso incorporates RSS for their calculation in very similar ways. The formula for estimating the coefficients in ridge regression is essentially taking the residual sum of squares (by subtracting $y_i$ from the fitted values) and adding a $\lambda$ shrinkage penalty, called the $\ell_2$ penalty, to the coefficients:

$$ \begin{equation} \sum_{i=1}^n \left(y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij} \right)^2 + \lambda \sum_{j=1}^p \beta^2_j \end{equation} $$

$$ \text{...which is more simply represented as...} $$

$$ \begin{equation} \text{RSS} + \lambda \sum_{j=1}^p \beta^2_j \end{equation} $$

Where the left part of the equation (RSS) is the "leftovers" of the variance and the right part of the equation (the penalized betas) are the penalized coefficients that explain the variance in the response. Contrarily, the equation for lasso is slightly different in that instead of the squared $\beta^2_j$, it takes an absolute $|\beta_j|$ to get the $\ell_1$ penalty:

$$ \begin{equation} \sum_{i=1}^n \left(y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij} \right)^2 + \lambda \sum_{j=1}^p |\beta_j| \end{equation} $$

$$ \text{...which again can be symbolized as...} $$

$$ \begin{equation} \text{RSS} + \lambda \sum_{j=1}^p |\beta_j| \end{equation} $$

The way these are defined leads to "corners" in the constraint of the lasso as shown in the left part of the image below (see p.222 of this book), where hitting this corner leads to a coefficient shrunk to zero (if $\lambda$ is set to a large value):

Differences in Methods

In practice, this means a few things:

• If one is deeply concerned about retaining all of the predictors, ridge may be superior. This is because it simply doesn't zero out the predictors in the model whereas lasso does. In scenarios where variable selection is ideal, lasso would serve this purpose better.
• Ridge is commonly used to address multicollinearity for this very reason. Because it can shrink the coefficients while still retaining them in the final model, it can be employed in contexts when there are extremely correlated predictors.
• Ridge may be worse than lasso if there are many redundant predictors. For example, if we have a model with several predictors that all have negligible influence on the estimated dependent variable, ridge will retain all of the predictors and will have a consequently poor predictive power. Lasso will drop a lot of these redundant variables when they're not doing anything anyway, which can help with variable selection.
• Lasso is sometimes better to achieve more interpretable models. For similar reasons, if we have 50 predictors in a model, a lasso that drops redundant predictors will provide a more parsimonious model that can be easier to interpret.
• In some cases, because of the similarity in formulation, the lasso and ridge will provide similar estimates of $\beta$ weights. Thus depending on the circumstance, each may have minimal practical difference in usage.

I will add some caveats about these methods...automatic model selection tools, particularly awful methods like stepwise regression and the like, are not always great instruments. They should be used carefully, and for those who do research, their use should be minimized with respect to theoretical plausible and actionable models. Throwing 100 predictors at a model often comes at the cost of interpretability, and while both can in some part elucidate theoretical testing of ideas, they can often do the opposite if one is not careful.