# What problem do shrinkage methods solve?

The holiday season has given me the opportunity to curl up next to the fire with The Elements of Statistical Learning. Coming from a (frequentist) econometrics perspective, I'm having trouble grasping the uses of shrinkage methods like ridge regression, lasso, and least angle regression (LAR). Typically, I'm interested in the parameter estimates themselves and in achieving unbiasedness or at least consistency. Shrinkage methods don't do that.

It seems to me that these methods are used when the statistician is worried that the regression function becomes too responsive to the predictors, that it considers the predictors to be more important (measured by the magnitude of the coefficients) than they actually are. In other words, overfitting.

But, OLS typically provides unbiased and consistent estimates.(footnote) I've always viewed the problem of overfitting not of giving estimates that are too big, but rather confidence intervals that are too small because the selection process isn't taken into account (ESL mentions this latter point).

Unbiased/consistent coefficient estimates lead to unbiased/consistent predictions of the outcome. Shrinkage methods push predictions closer to the mean outcome than OLS would, seemingly leaving information on the table.

To reiterate, I don't see what problem the shrinkage methods are trying to solve. Am I missing something?

Footnote: We need the full column rank condition for identification of the coefficients. The exogeneity/zero conditional mean assumption for the errors and the linear conditional expectation assumption determine the interpretation that we can give to the coefficients, but we get an unbiased or consistent estimate of something even if these assumptions aren't true.

-
There are several related questions here. This is one: stats.stackexchange.com/questions/10478/… –  cardinal Dec 27 '11 at 23:52
Note that there are simple and fairly weak conditions on the choice of shrinkage parameter to attain parameter consistency. This is detailed in the famous Knight & Fu (2000) paper and cover cases far beyond ridge regression and the lasso. Model selection consistency has also become a popular topic over the last few years. –  cardinal Dec 27 '11 at 23:55
@cardinal, thanks for the pointers to model consistency results for lasso; I'll have a look. Of course, these results can also be found for OLS. The results imply that both procedures get to the same place. So I still don't understand why we'd use lasso over OLS. –  Charlie Dec 28 '11 at 0:01
Model consistency is a different concept than asymptotic consistency of the parameter estimates. Are you aware of (familiar with) this difference? –  cardinal Dec 28 '11 at 0:09
@cardinal, By model consistency, I suppose that you mean that the correct predictors are included. We can get this by using the AIC criterion in the selection process using OLS. I guess that you're implying that, in the limit, lasso selects the right model with "wrong" coefficients? –  Charlie Dec 28 '11 at 0:45

I suspect you want a deeper answer, and I'll have to let someone else provide that, but I can give you some thoughts on ridge regression from a loose, conceptual perspective.

OLS regression yields parameter estimates that are unbiased (i.e., if such samples are gathered and parameters are estimated indefinitely, the sampling distribution of parameter estimates will be centered on the true value). Moreover, the sampling distribution will have the lowest variance of all possible unbiased estimates (this means that, on average, an OLS parameter estimate will be closer to the true value than an estimate from some other unbiased estimation procedure will be). This is old news (and I apologize, I know you know this well), however, the fact that the variance is lower does not mean that it is terribly low. Under some circumstances, the variance of the sampling distribution can be so large as to make the OLS estimator essentially worthless. (One situation where this could occur is when there is a high degree of multicollinearity.)

What is one to do in such a situation? Well, a different estimator could be found that has lower variance (although, obviously, it must be biased, given what was stipulated above). That is, we are trading off unbiasedness for lower variance. For example, we get parameter estimates that are likely to be substantially closer to the true value, albeit probably a little below the true value. Whether this tradeoff is worthwhile is a judgment the analyst must make when confronted with this situation. At any rate, ridge regression is just such a technique. The following (completely fabricated) figure is intended to illustrate these ideas.

This provides a short, simple, conceptual introduction to ridge regression. I know less about lasso and LAR, but I believe the same ideas could be applied. More information about the lasso and least angle regression can be found here, the "simple explanation..." link is especially helpful. This provides much more information about shrinkage methods.

I hope this is of some value.

-
This gives some nice conceptual hints. In the second paragraph there is a lot of focus on unbiasedness, but an important caveat is missing. Unless (a) the linear model is "correct" (and, when is it?) and (b) all relevant predictors are included in the model, the coefficient estimates will still be biased, in general. –  cardinal Dec 28 '11 at 13:10
My limited understanding of the bias/variance tradeoff is that someone looking for an explanation (as perhaps the original poster) would prefer unbiasedness, even if the variance were larger, but someone making a forecast might well prefer something with small variance, even if bias is introduced. –  Wayne Dec 28 '11 at 15:08
@Wayne: Indeed, this is (one of) the crux(es) of the matter. Much of the viewpoint in ESL is coming from a prediction perspective and so this colors a large part of their analysis. Performing inference on a single coefficient, particularly in an observational setting, is a very slippery matter. It would take some serious convincing to claim that the coefficient estimates were truly "unbiased". –  cardinal Dec 28 '11 at 15:23
Given some time, I may try to expand on my already overly voluminous comments a bit later. –  cardinal Dec 28 '11 at 15:25

The error of an estimator is a combination of (squared) bias and variance components. However in practice we want to fit a model to a particular finite sample of data and we want to minimise the total error of the estimator evaluated on the particular sample of data we actually have, rather than a zero error on average over some population of samples (that we don't have). Thus we want to reduce both the bias and variance, to minimise the error, which often means sacrificing unbiasedness to make a greater reduction in the variance component. This is especially true when dealing with small datasets, where the variance is likely to be high.

I think the difference in focus depends on whether one is interested in the properties of a procedure, or getting the best results on a particular sample. Frequentists typically find the former easier to deal with within that framework; Bayesians are often more focussed on the latter.

-

I guess that there are a few answers that may be applicable:

• Ridge regression can provide identification when the matrix of predictors is not full column rank.
• Lasso and LAR can be used when the number of predictors is greater than the number of observations (another variant of the non-singular issue).
• Lasso and LAR are automatic variable selection algorithms.

I'm not sure that the first point regarding ridge regression is really a feature; I think that I'd rather change my model to deal with non-identification. Even without a modeling change, OLS provides unique (and unbiased/consistent) predictions of the outcome in this case.

I could see how the second point could be helpful, but forward selection can also work in the case of the number of parameters exceeding the number of observations while yielding unbiased/consistent estimates.

On the last point, forward/backward selection, as examples, are easily automated.

So I still don't see the real advantages.

-
Some remarks: (1) The OLS estimates are not unique when the matrix of predictors is not full rank. (2) Consistency is an asymptotic concept and so requires a sequence of estimators. This means you need to define the type of sequence that you are considering, and the type of growth you are interested in does matter. (3) There are multiple types of consistency and understanding the differences among them can be illustrative. The Zhao & Yu (2006) paper has a nice discussion. (4) Unbiasedness is overrated. –  cardinal Dec 28 '11 at 0:01
(5) The original motivation of ridge regression in Hoerl & Kennard (1970) was to handle ill-conditioned design matrices, which is a "soft" form of rank deficiency. –  cardinal Dec 28 '11 at 0:07
@cardinal, re. (1): Sorry, I meant predictions of the outcome, rather than estimates of the coefficients. –  Charlie Dec 28 '11 at 0:08
Ah, ok. That squares better with your footnote in the question. –  cardinal Dec 28 '11 at 0:19

Here's a basic applied example from Biostatistics

Let's assume that I am studying possible relationships between the presence of ovarian cancer and a set of genes.

My dependent variable is a binary (coded as a zero or a 1) My independent variables codes data from a proteomic database.

As is common in many genetics studies, my data is much wider than it is tall. I have 216 different observations but 4000 or so possible predictors.

Linear regression is right out (the system is horrible over determined).

feature selection techniques really aren't feasible. With 4,000+ different independent variables all possible subset techniques are completely out of the question and even sequential feature selection is dubious.

The best option is probably to use logistic regression with an elastic net.

I want to do feature selection (identify which independent variables are important) so ridge regression really isn't appropriate.

It's entirely possible that there are more than 216 independent variables that have significant influence, so I probably shouldn't use a lasso (Lasso can't identify more predictors than you have observations)...

Enter the elastic net...

-