When to use regularization methods for regression?

Question

In what circumstances should one consider using regularization methods (ridge, lasso or least angles regression) instead of OLS?

In case this helps steer the discussion, my main interest is improving predictive accuracy.

Answer 1

Short answer: Whenever you are facing one of these situations:

• large number of variables or low ratio of no. observations to no. variables (including the $n\ll p$ case),
• high collinearity,
• seeking for a sparse solution (i.e., embed feature selection when estimating model parameters), or
• accounting for variables grouping in high-dimensional data set.

Ridge regression generally yields better predictions than OLS solution, through a better compromise between bias and variance. Its main drawback is that all predictors are kept in the model, so it is not very interesting if you seek a parsimonious model or want to apply some kind of feature selection.

To achieve sparsity, the lasso is more appropriate but it will not necessarily yield good results in presence of high collinearity (it has been observed that if predictors are highly correlated, the prediction performance of the lasso is dominated by ridge regression). The second problem with L1 penalty is that the lasso solution is not uniquely determined when the number of variables is greater than the number of subjects (this is not the case of ridge regression). The last drawback of lasso is that it tends to select only one variable among a group of predictors with high pairwise correlations. In this case, there are alternative solutions like the group (i.e., achieve shrinkage on block of covariates, that is some blocks of regression coefficients are exactly zero) or fused lasso. The Graphical Lasso also offers promising features for GGMs (see the R glasso package).

But, definitely, the elasticnet criteria, which is a combination of L1 and L2 penalties achieve both shrinkage and automatic variable selection, and it allows to keep $m>p$ variables in the case where $n\ll p$. Following Zou and Hastie (2005), it is defined as the argument that minimizes (over $\beta$)

$$ L(\lambda_1,\lambda_2,\mathbf{\beta}) = \|Y-X\beta\|^2 + \lambda_2\|\beta\|^2 + \lambda_1\|\beta\|_1 $$

where $\|\beta\|^2=\sum_{j=1}^p\beta_j^2$ and $\|\beta\|^1=\sum_{j=1}^p|\beta_j |$.

The lasso can be computed with an algorithm based on coordinate descent as described in the recent paper by Friedman and coll., Regularization Paths for Generalized Linear Models via Coordinate Descent (JSS, 2010) or the LARS algorithm. In R, the penalized, lars or biglars, and glmnet packages are useful packages; in Python, there's the scikit.learn toolkit, with extensive documentation on the algorithms used to apply all three kind of regularization schemes.

As for general references, the Lasso page contains most of what is needed to get started with lasso regression and technical details about L1-penalty, and this related question features essential references, When should I use lasso vs ridge?

Answer 2

A theoretical justification for the use of ridge regression is that its solution is the posterior mean given a normal prior on the coefficients. That is, if you care about squared error and you believe in a normal prior, the ridge estimates are optimal.

Similarly, the lasso estimate is the posterior mode under a double-exponential prior on your coefficients. This is optimal under a zero-one loss function.

In practice, these techniques typically improve predictive accuracy in situations where you have many correlated variables and not a lot of data. While the OLS estimator is best linear unbiased, it has high variance in these situations. If you look at the bias-variance trade off, prediction accuracy improves because the small increase in bias is more than offset by the large reduction in variance.