In order to solve problems of model selection, a number of methods (LASSO, ridge regression, etc.) will shrink the coefficients of predictor variables towards zero. I am looking for an intuitive explanation of why this improves predictive ability. If the true effect of the variable was actually very large, why doesn't shrinking the parameter result in a worse prediction?
$\begingroup$
$\endgroup$
0
asked Nov 2, 2015 at 20:29
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
11
Roughly speaking, there are three different sources of prediction error:
- the bias of your model
- the variance of your model
- unexplainable variance
We can't do anything about point 3 (except for attempting to estimate the unexplained variance and incorporating it in our predictive densities and prediction intervals). This leaves us with 1 and 2.
If you actually have the "right" model, then, say, OLS parameter estimates will be unbiased and have minimal variance among all unbiased (linear) estimators (they are BLUE). Predictions from an OLS model will be best linear unbiased predictions (BLUPs). That sounds good.
However, it turns out that although we have unbiased predictions and minimal variance among all unbiased predictions, the variance can still be pretty large. More importantly, we can sometimes introduce "a little" bias and simultaneously save "a lot" of variance - and by getting the tradeoff just right, we can get a lower prediction error with a biased (lower variance) model than with an unbiased (higher variance) one. This is called the "bias-variance tradeoff", and this question and its answers is enlightening: When is a biased estimator preferable to unbiased one?
And regularization like the lasso, ridge regression, the elastic net and so forth do exactly that. They pull the model towards zero. (Bayesian approaches are similar - they pull the model towards the priors.) Thus, regularized models will be biased compared to non-regularized models, but also have lower variance. If you choose your regularization right, the result is a prediction with a lower error.
If you search for "bias-variance tradeoff regularization" or similar, you get some food for thought. This presentation, for instance, is useful.
EDIT: amoeba quite rightly points out that I am handwaving as to why exactly regularization yields lower variance of models and predictions. Consider a lasso model with a large regularization parameter $\lambda$. If $\lambda\to\infty$, your lasso parameter estimates will all be shrunk to zero. A fixed parameter value of zero has zero variance. (This is not entirely correct, since the threshold value of $\lambda$ beyond which your parameters will be shrunk to zero depends on your data and your model. But given the model and the data, you can find a $\lambda$ such that the model is the zero model. Always keep your quantifiers straight.) However, the zero model will of course also have a giant bias. It doesn't care about the actual observations, after all.
And the same applies to not-all-that-extreme values of your regularization parameter(s): small values will yield the unregularized parameter estimates, which will be less biased (unbiased if you have the "correct" model), but have higher variance. They will "jump around", following your actual observations. Higher values of your regularization $\lambda$ will "constrain" your parameter estimates more and more. This is why the methods have names like "lasso" or "elastic net": they constrain the freedom of your parameters to float around and follow the data.
(I am writing up a little paper on this, which will hopefully be rather accessible. I'll add a link once it's available.)
answered Nov 2, 2015 at 20:48
-
4$\begingroup$ It seems that the crucial piece of the puzzle is: why do shrinkage methods decrease the variance? (That they introduce some bias is more or less obvious.) You simply state that they do; can you provide some intuition for that? $\endgroup$– amoebaCommented Nov 2, 2015 at 21:33
-
2$\begingroup$ @Stephan Kolassa So adding the penalization term accounting for the size of the coefficients adds a little bit of bias but reduces variability because it penalizes large coefficients, which will generally have more variability than smaller coefficients. Is that correct? Then, ultimately we are not so concerned about getting the 'correct' value for any particular coefficient, we are just interested in the overall predictive ability of the model? $\endgroup$ Commented Nov 2, 2015 at 21:37
-
3$\begingroup$ @aspiringstatistician: Your second sentence is right on the mark. (Recall George Box about "wrong but useful" models.) I wouldn't worry all that much about whether large parameter estimates are shrunk more than small ones. First, this will depend on standardization. Second, if your large parameter values are well estimated (i.e., with low error), then they won't necessarily be shrunk a lot. Regularization "prefers" to shrink those parameters that are badly defined, i.e., which have a high variance. $\endgroup$ Commented Nov 2, 2015 at 21:46
-
3$\begingroup$ +1. Good luck with the paper! @aspiringstatistician: Very good observation about shrinkage not being concerned with getting the correct model; this is exactly right (and is worth contemplating): correctly specified model can have worse predictive ability than the regularized and "less true" one (see Appendix on page 307 of this paper for an example). $\endgroup$– amoebaCommented Nov 2, 2015 at 21:56
-
7$\begingroup$ +1. Just wanted to add, that while the question was about the intuition behind regularized models, it feels a little incomplete not to mention the Bayesian derivation of these models. For example, when comparing ridge regression to simple MLE, in most applications it is seems natural to me to think of the effect being drawn from a normal distribution, as opposed to a uniform (improper) distribution. So seeing these techniques both as special cases of MAP estimation makes it clear why one would choose ridge regression. $\endgroup$ Commented Nov 2, 2015 at 22:35
$\begingroup$
$\endgroup$
3
Just to add something to @Kolassa's fine answer, the whole question of shrinkage estimates is bound up with Stein's paradox. For multivariate processes with $p \geq 3$, the vector of sample averages is not admissible. In other words, for some parameter value, there is a different estimator with lower expected risk. Stein proposed a shrinkage estimator as an example. So we're dealing with the curse of dimensionality, since shrinkage does not help you when you have only 1 or 2 independent variables.
Read this answer for more. Apparently, Stein's paradox is related to the well-known theorem that a Browian motion process in 3 or more dimensions is non-recurrent (wanders all over the place without returning to the origin), whereas the 1 and 2 dimensional Brownians are recurrent.
Stein's paradox holds regardless of what you shrink towards, although in practice, it does better if you shrink towards the true parameter values. This is what Bayesians do. They think they know where the true parameter is and they shrink towards it. Then they claim that Stein validates their existence.
It's called a paradox precisely because it does challenge our intuition. However, if you think of Brownian motion, the only way to get a 3D Brownian motion to return to the origin would be to impose a damping penalty on the steps. A shrinkage estimator also imposes a sort of damper on the estimates (reduces variance), which is why it works.
-
$\begingroup$ Do you have a reference for the connection between Stein's paradox and Brownian processes? $\endgroup$ Commented Sep 11, 2016 at 16:27
-
1$\begingroup$ Follow my link under "Read this answer for more". There is a link in that response to a paper that makes the connection. $\endgroup$– PlacidiaCommented Sep 12, 2016 at 0:04
-
2$\begingroup$ bayes estimators are admissible by the complete class theorem: it has nothing to do with the JS estimator directly. However, the result that JS dominates sample mean did make people more interested in studying bayes estimators. (I'm objecting to the claim that bayesians "claim that Stein validates their existence.") $\endgroup$ Commented Sep 16, 2017 at 15:55
$\begingroup$
$\endgroup$
@Kolassa has a great mathematical answer. For a more intuitive visual answer here is a picture. I'm doing simple linear regression here with a slope and y-intercept. A population of 17 points are loosely correlated. At random I picked two points and created a regression. In general, 2 points is not enough observations and my regression lines are going to vary wildly in shape and quality. However, the r^2 error will be perfect, the line hits both of my test points. The solid lines (R1 through R5) represent these regressions. The dashed lines (G1 through G5) represent the regression with a shrinkage effect applied.
- Shrinkage is going to shrink the slope towards zero. This isn't an arbitrary value. We are stating that this parameter is less likely to have an effect. In my 2d linear regression we are stating the values are less likely to be correlated. It is a way of softening our result and fighting against overfit. It makes sense when we only have a few points out of the total sample that we're more likely to see false correlation.
- Shrinkage doesn't always yield a better result. Going from R3 to G3 we ended up with a poorer estimate of the final regression. It is simply more likely to yield a better regression.
- Shrinkage isn't just a matter of rotating the final regression line towards zero. When you change the slope you need to change the y-intercept as well. In this case we're taking a line that goes exactly through both points and ended up with a line that goes somewhere through the middle.
- You can see that the variance of the dashed lines is considerably lower than the variance of the solid lines as we would expect.
- Imagine there was no noise. Shrinkage would be terrible. Any two points we pick would give us the perfect line. If we applied shrinkage we would only end up with a worse result.
- If you want further explanation then Josh Starmer with StatQuest has a great video here
