In order to solve problems of model selection, a number of methods (LASSO, ridge regression, etc.) will shrink the coefficients of predictor variables towards zero. I am looking for an intuitive explanation of why this improves predictive ability. If the true effect of the variable was actually very large, why doesn't shrinking the parameter result in a worse prediction?
Roughly speaking, there are three different sources of prediction error:
- the bias of your model
- the variance of your model
- unexplainable variance
We can't do anything about point 3 (except for attempting to estimate the unexplained variance and incorporating it in our predictive densities and prediction intervals). This leaves us with 1 and 2.
If you actually have the "right" model, then, say, OLS parameter estimates will be unbiased and have minimal variance among all unbiased (linear) estimators (they are BLUE). Predictions from an OLS model will be best linear unbiased predictions (BLUPs). That sounds good.
However, it turns out that although we have unbiased predictions and minimal variance among all unbiased predictions, the variance can still be pretty large. More importantly, we can sometimes introduce "a little" bias and simultaneously save "a lot" of variance - and by getting the tradeoff just right, we can get a lower prediction error with a biased (lower variance) model than with an unbiased (higher variance) one. This is called the "bias-variance tradeoff", and this question and its answers is enlightening: When is a biased estimator preferable to unbiased one?
And regularization like the lasso, ridge regression, the elastic net and so forth do exactly that. They pull the model towards zero. (Bayesian approaches are similar - they pull the model towards the priors.) Thus, regularized models will be biased compared to non-regularized models, but also have lower variance. If you choose your regularization right, the result is a prediction with a lower error.
If you search for "bias-variance tradeoff regularization" or similar, you get some food for thought. This presentation, for instance, is useful.
EDIT: amoeba quite rightly points out that I am handwaving as to why exactly regularization yields lower variance of models and predictions. Consider a lasso model with a large regularization parameter $\lambda$. If $\lambda\to\infty$, your lasso parameter estimates will all be shrunk to zero. A fixed parameter value of zero has zero variance. (This is not entirely correct, since the threshold value of $\lambda$ beyond which your parameters will be shrunk to zero depends on your data and your model. But given the model and the data, you can find a $\lambda$ such that the model is the zero model. Always keep your quantifiers straight.) However, the zero model will of course also have a giant bias. It doesn't care about the actual observations, after all.
And the same applies to not-all-that-extreme values of your regularization parameter(s): small values will yield the unregularized parameter estimates, which will be less biased (unbiased if you have the "correct" model), but have higher variance. They will "jump around", following your actual observations. Higher values of your regularization $\lambda$ will "constrain" your parameter estimates more and more. This is why the methods have names like "lasso" or "elastic net": they constrain the freedom of your parameters to float around and follow the data.
(I am writing up a little paper on this, which will hopefully be rather accessible. I'll add a link once it's available.)
Just to add something to @Kolassa's fine answer, the whole question of shrinkage estimates is bound up with Stein's paradox. For multivariate processes with $p \geq 3$, the vector of sample averages is not admissible. In other words, for some parameter value, there is a different estimator with lower expected risk. Stein proposed a shrinkage estimator as an example. So we're dealing with the curse of dimensionality, since shrinkage does not help you when you have only 1 or 2 independent variables.
Read this answer for more. Apparently, Stein's paradox is related to the well-known theorem that a Browian motion process in 3 or more dimensions is non-recurrent (wanders all over the place without returning to the origin), whereas the 1 and 2 dimensional Brownians are recurrent.
Stein's paradox holds regardless of what you shrink towards, although in practice, it does better if you shrink towards the true parameter values. This is what Bayesians do. They think they know where the true parameter is and they shrink towards it. Then they claim that Stein validates their existence.
It's called a paradox precisely because it does challenge our intuition. However, if you think of Brownian motion, the only way to get a 3D Brownian motion to return to the origin would be to impose a damping penalty on the steps. A shrinkage estimator also imposes a sort of damper on the estimates (reduces variance), which is why it works.