What is shrinkage? The word shrinkage gets thrown around a lot in certain circles. But what is shrinkage, there does not seem to be a clear definition. If I have a time series (or any collection of observations of some process) what are the different ways I can measure some type of empirical shrinkage on the series? What are the different types of theoretical shrinkage I can talk about? How can shrinkage help in prediction? Can people provide some good insight or references?
 A: In 1961 James and Stein published an article called "Estimation with Quadratic Loss" https://projecteuclid.org/download/pdf_1/euclid.bsmsp/1200512173 . While it doesn't specifically coin the term shrinkage, they discuss minimax estimators for high dimensional (actually even for a 3 parameter location) statistics that have less risk (expected loss) than the usual MLE (each component the sample average) for normal data. Bradley Efron calls their finding "the most striking theorem of post-war mathematical statistics". This article has been cited 3,310 times.
Copas in 1983 writes the first article Regression, Prediction and Shrinkage to coin the term "shrinkage". It's defined implicitly in the abstract:

The fit of a regression predictor to new data is nearly always worse
  than its fit to the original data. Anticipating this shrinkage leads
  to Stein‐type predictors which, under certain assumptions, give a
  uniformly lower prediction mean squared error than least squares.

And in all successive research, it seems that shrinkage refers to the operating characteristics (and estimates thereof) for out-of-sample validity of prediction and estimation in the context of finding admissible and/or minimax estimators.
A: This is about regularization. Suppose you would like to fit a curve and you use a square loss function (you can pick different). By fit you would like to recover the parameters that govern the process which generated that curve. Now imagine that you would like to fit this curve using 100th polynomial (just for example). You are pretty likely going to overfit or capture every kink and noise of the curve. In addition, your prediction capabilities outside the given training data interval will be likely very poor. So, regularization term is added to the objective function with some weight multiplied by the regularization factor - $l_1$, $l_2$ or custom. In the case of $l_2$, which is arbuably simpler to understand, this will have an effect that the large parameter values will be forced to reduce aka shrink. You can think of regularization or shrinkage as driving your algorithm to a solution which might be a better solution.
