The word shrinkage gets thrown around a lot in certain circles. But what is shrinkage, there does not seem to be a clear definition. If I have a time series (or any collection of observations of some process) what are the different ways I can measure some type of empirical shrinkage on the series? What are the different types of theoretical shrinkage I can talk about? How can shrinkage help in prediction? Can people provide some good insight or references?
This is about regularization. Suppose you would like to fit a curve and you use a square loss function (you can pick different). By
fit you would like to recover the parameters that govern the process which generated that curve. Now imagine that you would like to fit this curve using 100th polynomial (just for example). You are pretty likely going to overfit or capture every kink and noise of the curve. In addition, your prediction capabilities outside the given training data interval will be likely very poor. So, regularization term is added to the objective function with some weight multiplied by the regularization factor - l_1, l_2 or custom. In the case of l_2, which is arbuably simpler to understand, this will have an effect that the large parameter values will be forced to reduce aka shrink. You can think of regularization or shrinkage as driving your algorithm to a solution which might be a better solution.