In order to solve problems of model selection, a number of methods (LASSO, ridge regression, etc.) will shrink the coefficients of predictor variables towards zero. I am looking for an intuitive explanation of why this improves predictive ability. If the true effect of the variable was actually very large, why doesn't shrinking the parameter result in a worse prediction?
Roughly speaking, there are three different sources of prediction error:
- the bias of your model
- the variance of your model
- unexplainable variance
We can't do anything about point 3 (except for attempting to estimate the unexplained variance and incorporating it in our predictive densities and prediction intervals). This leaves us with 1 and 2.
If you actually have the "right" model, then, say, OLS parameter estimates will be unbiased and have minimal variance among all unbiased (linear) estimators (they are BLUE). Predictions from an OLS model will be best linear unbiased predictions (BLUPs). That sounds good.
However, it turns out that although we have unbiased predictions and minimal variance among all unbiased predictions, the variance can still be pretty large. More importantly, we can sometimes introduce "a little" bias and simultaneously save "a lot" of variance - and by getting the tradeoff just right, we can get a lower prediction error with a biased (lower variance) model than with an unbiased (higher variance) one. This is called the "bias-variance tradeoff", and this question and its answers is enlightening: When is a biased estimator preferable to unbiased one?
And regularization like the lasso, ridge regression, the elastic net and so forth do exactly that. They pull the model towards zero. (Bayesian approaches are similar - they pull the model towards the priors.) Thus, regularized models will be biased compared to non-regularized models, but also have lower variance. If you choose your regularization right, the result is a prediction with a lower error.
If you search for "bias-variance tradeoff regularization" or similar, you get some food for thought. This presentation, for instance, is useful.
EDIT: amoeba quite rightly points out that I am handwaving as to why exactly regularization yields lower variance of models and predictions. Consider a lasso model with a large regularization parameter $\lambda$. If $\lambda\to\infty$, your lasso parameter estimates will all be shrunk to zero. A fixed parameter value of zero has zero variance. (This is not entirely correct, since the threshold value of $\lambda$ beyond which your parameters will be shrunk to zero depends on your data and your model. But given the model and the data, you can find a $\lambda$ such that the model is the zero model. Always keep your quantifiers straight.) However, the zero model will of course also have a giant bias. It doesn't care about the actual observations, after all.
And the same applies to not-all-that-extreme values of your regularization parameter(s): small values will yield the unregularized parameter estimates, which will be less biased (unbiased if you have the "correct" model), but have higher variance. They will "jump around", following your actual observations. Higher values of your regularization $\lambda$ will "constrain" your parameter estimates more and more. This is why the methods have names like "lasso" or "elastic net": they constrain the freedom of your parameters to float around and follow the data.
(I am writing up a little paper on this, which will hopefully be rather accessible. I'll add a link once it's available.)
Just to add something to @Kolassa's fine answer, the whole question of shrinkage estimates is bound up with Stein's paradox. For multivariate processes with $p \geq 3$, the vector of sample averages is not admissible. In other words, for some parameter value, there is a different estimator with lower expected risk. Stein proposed a shrinkage estimator as an example. So we're dealing with the curse of dimensionality, since shrinkage does not help you when you have only 1 or 2 independent variables.
Read this answer for more. Apparently, Stein's paradox is related to the well-known theorem that a Browian motion process in 3 or more dimensions is non-recurrent (wanders all over the place without returning to the origin), whereas the 1 and 2 dimensional Brownians are recurrent.
Stein's paradox holds regardless of what you shrink towards, although in practice, it does better if you shrink towards the true parameter values. This is what Bayesians do. They think they know where the true parameter is and they shrink towards it. Then they claim that Stein validates their existence.
It's called a paradox precisely because it does challenge our intuition. However, if you think of Brownian motion, the only way to get a 3D Brownian motion to return to the origin would be to impose a damping penalty on the steps. A shrinkage estimator also imposes a sort of damper on the estimates (reduces variance), which is why it works.
@Kolassa has a great mathematical answer. For a more intuitive visual answer here is a picture. I'm doing simple linear regression here with a slope and y-intercept. A population of 17 points are loosely correlated. At random I picked two points and created a regression. In general, 2 points is not enough observations and my regression lines are going to vary wildly in shape and quality. However, the r^2 error will be perfect, the line hits both of my test points. The solid lines (R1 through R5) represent these regressions. The dashed lines (G1 through G5) represent the regression with a shrinkage effect applied.
- Shrinkage is going to shrink the slope towards zero. This isn't an arbitrary value. We are stating that this parameter is less likely to have an effect. In my 2d linear regression we are stating the values are less likely to be correlated. It is a way of softening our result and fighting against overfit. It makes sense when we only have a few points out of the total sample that we're more likely to see false correlation.
- Shrinkage doesn't always yield a better result. Going from R3 to G3 we ended up with a poorer estimate of the final regression. It is simply more likely to yield a better regression.
- Shrinkage isn't just a matter of rotating the final regression line towards zero. When you change the slope you need to change the y-intercept as well. In this case we're taking a line that goes exactly through both points and ended up with a line that goes somewhere through the middle.
- You can see that the variance of the dashed lines is considerably lower than the variance of the solid lines as we would expect.
- Imagine there was no noise. Shrinkage would be terrible. Any two points we pick would give us the perfect line. If we applied shrinkage we would only end up with a worse result.
- If you want further explanation then Josh Starmer with StatQuest has a great video here