What problem do shrinkage methods solve?

Question

The holiday season has given me the opportunity to curl up next to the fire with The Elements of Statistical Learning. Coming from a (frequentist) econometrics perspective, I'm having trouble grasping the uses of shrinkage methods like ridge regression, lasso, and least angle regression (LAR). Typically, I'm interested in the parameter estimates themselves and in achieving unbiasedness or at least consistency. Shrinkage methods don't do that.

It seems to me that these methods are used when the statistician is worried that the regression function becomes too responsive to the predictors, that it considers the predictors to be more important (measured by the magnitude of the coefficients) than they actually are. In other words, overfitting.

But, OLS typically provides unbiased and consistent estimates.(footnote) I've always viewed the problem of overfitting not of giving estimates that are too big, but rather confidence intervals that are too small because the selection process isn't taken into account (ESL mentions this latter point).

Unbiased/consistent coefficient estimates lead to unbiased/consistent predictions of the outcome. Shrinkage methods push predictions closer to the mean outcome than OLS would, seemingly leaving information on the table.

To reiterate, I don't see what problem the shrinkage methods are trying to solve. Am I missing something?

Footnote: We need the full column rank condition for identification of the coefficients. The exogeneity/zero conditional mean assumption for the errors and the linear conditional expectation assumption determine the interpretation that we can give to the coefficients, but we get an unbiased or consistent estimate of something even if these assumptions aren't true.

Answer 1

I suspect you want a deeper answer, and I'll have to let someone else provide that, but I can give you some thoughts on ridge regression from a loose, conceptual perspective.

OLS regression yields parameter estimates that are unbiased (i.e., if such samples are gathered and parameters are estimated indefinitely, the sampling distribution of parameter estimates will be centered on the true value). Moreover, the sampling distribution will have the lowest variance of all possible unbiased estimates (this means that, on average, an OLS parameter estimate will be closer to the true value than an estimate from some other unbiased estimation procedure will be). This is old news (and I apologize, I know you know this well), however, the fact that the variance is lower does not mean that it is terribly low. Under some circumstances, the variance of the sampling distribution can be so large as to make the OLS estimator essentially worthless. (One situation where this could occur is when there is a high degree of multicollinearity.)

What is one to do in such a situation? Well, a different estimator could be found that has lower variance (although, obviously, it must be biased, given what was stipulated above). That is, we are trading off unbiasedness for lower variance. For example, we get parameter estimates that are likely to be substantially closer to the true value, albeit probably a little below the true value. Whether this tradeoff is worthwhile is a judgment the analyst must make when confronted with this situation. At any rate, ridge regression is just such a technique. The following (completely fabricated) figure is intended to illustrate these ideas.

This provides a short, simple, conceptual introduction to ridge regression. I know less about lasso and LAR, but I believe the same ideas could be applied. More information about the lasso and least angle regression can be found here, the "simple explanation..." link is especially helpful. This provides much more information about shrinkage methods.

I hope this is of some value.

Answer 2

The error of an estimator is a combination of (squared) bias and variance components. However in practice we want to fit a model to a particular finite sample of data and we want to minimise the total error of the estimator evaluated on the particular sample of data we actually have, rather than a zero error on average over some population of samples (that we don't have). Thus we want to reduce both the bias and variance, to minimise the error, which often means sacrificing unbiasedness to make a greater reduction in the variance component. This is especially true when dealing with small datasets, where the variance is likely to be high.

I think the difference in focus depends on whether one is interested in the properties of a procedure, or getting the best results on a particular sample. Frequentists typically find the former easier to deal with within that framework; Bayesians are often more focussed on the latter.

Answer 3

I guess that there are a few answers that may be applicable:

• Ridge regression can provide identification when the matrix of predictors is not full column rank.
• Lasso and LAR can be used when the number of predictors is greater than the number of observations (another variant of the non-singular issue).
• Lasso and LAR are automatic variable selection algorithms.

I'm not sure that the first point regarding ridge regression is really a feature; I think that I'd rather change my model to deal with non-identification. Even without a modeling change, OLS provides unique (and unbiased/consistent) predictions of the outcome in this case.

I could see how the second point could be helpful, but forward selection can also work in the case of the number of parameters exceeding the number of observations while yielding unbiased/consistent estimates.

On the last point, forward/backward selection, as examples, are easily automated.

So I still don't see the real advantages.

Answer 4

Here's a basic applied example from Biostatistics

Let's assume that I am studying possible relationships between the presence of ovarian cancer and a set of genes.

My dependent variable is a binary (coded as a zero or a 1) My independent variables codes data from a proteomic database.

As is common in many genetics studies, my data is much wider than it is tall. I have 216 different observations but 4000 or so possible predictors.

Linear regression is right out (the system is horrible over determined).

feature selection techniques really aren't feasible. With 4,000+ different independent variables all possible subset techniques are completely out of the question and even sequential feature selection is dubious.

The best option is probably to use logistic regression with an elastic net.

I want to do feature selection (identify which independent variables are important) so ridge regression really isn't appropriate.

It's entirely possible that there are more than 216 independent variables that have significant influence, so I probably shouldn't use a lasso (Lasso can't identify more predictors than you have observations)...

Enter the elastic net...

Answer 5

Another problem which linear regression shrinkage methods can address is obtaining a low variance (possibly unbiased) estimate of an average treatment effect (ATE) in high-dimensional case-control studies on observational data.

Specifically, in cases where 1) there are a large number of variables (making it difficult to select variables for exact matching), 2) propensity score matching fails to eliminate imbalance in the treatment and control samples, and 3) multicollinearity is present, there are several techniques, such as the adaptive lasso (Zou, 2006) that obtain asymptotically unbiased estimates. There have been several papers that discuss using lasso regression for causal inference and generating confidence intervals on coefficient estimates (see the following post: Inference after using Lasso for variable selection).