Why does shrinkage work?

In order to solve problems of model selection, a number of methods (LASSO, ridge regression, etc.) will shrink the coefficients of predictor variables towards zero. I am looking for an intuitive explanation of why this improves predictive ability. If the true effect of the variable was actually very large, why doesn't shrinking the parameter result in a worse prediction?

Roughly speaking, there are three different sources of prediction error:

1. the bias of your model
2. the variance of your model
3. unexplainable variance

We can't do anything about point 3 (except for attempting to estimate the unexplained variance and incorporating it in our predictive densities and prediction intervals). This leaves us with 1 and 2.

If you actually have the "right" model, then, say, OLS parameter estimates will be unbiased and have minimal variance among all unbiased (linear) estimators (they are BLUE). Predictions from an OLS model will be best linear unbiased predictions (BLUPs). That sounds good.

However, it turns out that although we have unbiased predictions and minimal variance among all unbiased predictions, the variance can still be pretty large. More importantly, we can sometimes introduce "a little" bias and simultaneously save "a lot" of variance - and by getting the tradeoff just right, we can get a lower prediction error with a biased (lower variance) model than with an unbiased (higher variance) one. This is called the "bias-variance tradeoff", and this question and its answers is enlightening: When is a biased estimator preferable to unbiased one?

And regularization like the lasso, ridge regression, the elastic net and so forth do exactly that. They pull the model towards zero. (Bayesian approaches are similar - they pull the model towards the priors.) Thus, regularized models will be biased compared to non-regularized models, but also have lower variance. If you choose your regularization right, the result is a prediction with a lower error.

If you search for "bias-variance tradeoff regularization" or similar, you get some food for thought. This presentation, for instance, is useful.

EDIT: amoeba quite rightly points out that I am handwaving as to why exactly regularization yields lower variance of models and predictions. Consider a lasso model with a large regularization parameter $\lambda$. If $\lambda\to\infty$, your lasso parameter estimates will all be shrunk to zero. A fixed parameter value of zero has zero variance. (This is not entirely correct, since the threshold value of $\lambda$ beyond which your parameters will be shrunk to zero depends on your data and your model. But given the model and the data, you can find a $\lambda$ such that the model is the zero model. Always keep your quantifiers straight.) However, the zero model will of course also have a giant bias. It doesn't care about the actual observations, after all.

And the same applies to not-all-that-extreme values of your regularization parameter(s): small values will yield the unregularized parameter estimates, which will be less biased (unbiased if you have the "correct" model), but have higher variance. They will "jump around", following your actual observations. Higher values of your regularization $\lambda$ will "constrain" your parameter estimates more and more. This is why the methods have names like "lasso" or "elastic net": they constrain the freedom of your parameters to float around and follow the data.

(I am writing up a little paper on this, which will hopefully be rather accessible. I'll add a link once it's available.)

• It seems that the crucial piece of the puzzle is: why do shrinkage methods decrease the variance? (That they introduce some bias is more or less obvious.) You simply state that they do; can you provide some intuition for that? – amoeba Nov 2 '15 at 21:33
• @Stephan Kolassa So adding the penalization term accounting for the size of the coefficients adds a little bit of bias but reduces variability because it penalizes large coefficients, which will generally have more variability than smaller coefficients. Is that correct? Then, ultimately we are not so concerned about getting the 'correct' value for any particular coefficient, we are just interested in the overall predictive ability of the model? – aspiringstatistician Nov 2 '15 at 21:37
• @aspiringstatistician: Your second sentence is right on the mark. (Recall George Box about "wrong but useful" models.) I wouldn't worry all that much about whether large parameter estimates are shrunk more than small ones. First, this will depend on standardization. Second, if your large parameter values are well estimated (i.e., with low error), then they won't necessarily be shrunk a lot. Regularization "prefers" to shrink those parameters that are badly defined, i.e., which have a high variance. – Stephan Kolassa Nov 2 '15 at 21:46
• +1. Good luck with the paper! @aspiringstatistician: Very good observation about shrinkage not being concerned with getting the correct model; this is exactly right (and is worth contemplating): correctly specified model can have worse predictive ability than the regularized and "less true" one (see Appendix on page 307 of this paper for an example). – amoeba Nov 2 '15 at 21:56
• +1. Just wanted to add, that while the question was about the intuition behind regularized models, it feels a little incomplete not to mention the Bayesian derivation of these models. For example, when comparing ridge regression to simple MLE, in most applications it is seems natural to me to think of the effect being drawn from a normal distribution, as opposed to a uniform (improper) distribution. So seeing these techniques both as special cases of MAP estimation makes it clear why one would choose ridge regression. – jlimahaverford Nov 2 '15 at 22:35

Just to add something to @Kolassa's fine answer, the whole question of shrinkage estimates is bound up with Stein's paradox. For multivariate processes with $p \geq 3$, the vector of sample averages is not admissible. In other words, for some parameter value, there is a different estimator with lower expected risk. Stein proposed a shrinkage estimator as an example. So we're dealing with the curse of dimensionality, since shrinkage does not help you when you have only 1 or 2 independent variables.

Read this answer for more. Apparently, Stein's paradox is related to the well-known theorem that a Browian motion process in 3 or more dimensions is non-recurrent (wanders all over the place without returning to the origin), whereas the 1 and 2 dimensional Brownians are recurrent.

Stein's paradox holds regardless of what you shrink towards, although in practice, it does better if you shrink towards the true parameter values. This is what Bayesians do. They think they know where the true parameter is and they shrink towards it. Then they claim that Stein validates their existence.

It's called a paradox precisely because it does challenge our intuition. However, if you think of Brownian motion, the only way to get a 3D Brownian motion to return to the origin would be to impose a damping penalty on the steps. A shrinkage estimator also imposes a sort of damper on the estimates (reduces variance), which is why it works.

• Do you have a reference for the connection between Stein's paradox and Brownian processes? – kjetil b halvorsen Sep 11 '16 at 16:27
• Follow my link under "Read this answer for more". There is a link in that response to a paper that makes the connection. – Placidia Sep 12 '16 at 0:04
• bayes estimators are admissible by the complete class theorem: it has nothing to do with the JS estimator directly. However, the result that JS dominates sample mean did make people more interested in studying bayes estimators. (I'm objecting to the claim that bayesians "claim that Stein validates their existence.") – user795305 Sep 16 '17 at 15:55