19
$\begingroup$

The word shrinkage gets thrown around a lot in certain circles. But what is shrinkage, there does not seem to be a clear definition. If I have a time series (or any collection of observations of some process) what are the different ways I can measure some type of empirical shrinkage on the series? What are the different types of theoretical shrinkage I can talk about? How can shrinkage help in prediction? Can people provide some good insight or references?

$\endgroup$
3
  • 2
    $\begingroup$ Steyergerg: Application of Shrinkage Techniques in Logistic Regression Analysis: A Case Study and Shrinkage and penalized likelihood as methods to improve predictive accuracy are good places to start. Neither is open source (I think) but google will find original articles. $\endgroup$
    – charles
    Commented Mar 7, 2015 at 2:19
  • 5
    $\begingroup$ Any form of regularization of an estimator that moves (shrinks) an estimate (generally toward 0 or some other 'null'/known value); indeed, regularization that moves a collection of estimates toward each other is also a kind of shrinkage (it moves the parameters representing their differences toward 0). If you haven't already seen it, the Wikipedia article may be helpful. $\endgroup$
    – Glen_b
    Commented Mar 7, 2015 at 5:21
  • $\begingroup$ What about empirical Shrinkage. Suppose I have a time series I'm fitting a model to. Can I talk about some type of shrinkage between the in sample fit and the out of sample performance? $\endgroup$
    – Wintermute
    Commented Mar 7, 2015 at 14:25

2 Answers 2

7
$\begingroup$

In 1961 James and Stein published an article called "Estimation with Quadratic Loss" https://projecteuclid.org/download/pdf_1/euclid.bsmsp/1200512173 . While it doesn't specifically coin the term shrinkage, they discuss minimax estimators for high dimensional (actually even for a 3 parameter location) statistics that have less risk (expected loss) than the usual MLE (each component the sample average) for normal data. Bradley Efron calls their finding "the most striking theorem of post-war mathematical statistics". This article has been cited 3,310 times.

Copas in 1983 writes the first article Regression, Prediction and Shrinkage to coin the term "shrinkage". It's defined implicitly in the abstract:

The fit of a regression predictor to new data is nearly always worse than its fit to the original data. Anticipating this shrinkage leads to Stein‐type predictors which, under certain assumptions, give a uniformly lower prediction mean squared error than least squares.

And in all successive research, it seems that shrinkage refers to the operating characteristics (and estimates thereof) for out-of-sample validity of prediction and estimation in the context of finding admissible and/or minimax estimators.

$\endgroup$
4
$\begingroup$

This is about regularization. Suppose you would like to fit a curve and you use a square loss function (you can pick different). By fit you would like to recover the parameters that govern the process which generated that curve. Now imagine that you would like to fit this curve using 100th polynomial (just for example). You are pretty likely going to overfit or capture every kink and noise of the curve. In addition, your prediction capabilities outside the given training data interval will be likely very poor. So, regularization term is added to the objective function with some weight multiplied by the regularization factor - $l_1$, $l_2$ or custom. In the case of $l_2$, which is arbuably simpler to understand, this will have an effect that the large parameter values will be forced to reduce aka shrink. You can think of regularization or shrinkage as driving your algorithm to a solution which might be a better solution.

$\endgroup$
3
  • 2
    $\begingroup$ Is "regularization" the same as "shrinkage". If not, how are they distinct? Also, what is l_2 ? $\endgroup$ Commented Mar 11, 2021 at 22:26
  • $\begingroup$ @Harvey Motulsky: Yes, regularization is the same as shrinkage, basically. And l_2 really refers to least squares distance! $\endgroup$ Commented Aug 20, 2021 at 15:03
  • 1
    $\begingroup$ The Wikipedia page, as well as the other answer to this question, imply that "shrinkage" refers to the fact that the fitness of a predictor learned from data decreases ("shrinks") when applied to new data. Regularization is one way of doing that. It looks to me that people started seeing methods decreasing a parameter's value in order to deal with shrinkage, and assumed that the "shrinking" taking place was this decreasing of the parameter value, but that is not the original meaning of the term. This drifting of the meaning may help explain the confusion expressed by the original question. $\endgroup$
    – user118967
    Commented Sep 24, 2021 at 7:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.