The word shrinkage gets thrown around a lot in certain circles. But what is shrinkage, there does not seem to be a clear definition. If I have a time series (or any collection of observations of some process) what are the different ways I can measure some type of empirical shrinkage on the series? What are the different types of theoretical shrinkage I can talk about? How can shrinkage help in prediction? Can people provide some good insight or references?

  • 1
    $\begingroup$ Steyergerg: Application of Shrinkage Techniques in Logistic Regression Analysis: A Case Study and Shrinkage and penalized likelihood as methods to improve predictive accuracy are good places to start. Neither is open source (I think) but google will find original articles. $\endgroup$ – charles Mar 7 '15 at 2:19
  • 3
    $\begingroup$ Any form of regularization of an estimator that moves (shrinks) an estimate (generally toward 0 or some other 'null'/known value); indeed, regularization that moves a collection of estimates toward each other is also a kind of shrinkage (it moves the parameters representing their differences toward 0). If you haven't already seen it, the Wikipedia article may be helpful. $\endgroup$ – Glen_b Mar 7 '15 at 5:21
  • $\begingroup$ What about empirical Shrinkage. Suppose I have a time series I'm fitting a model to. Can I talk about some type of shrinkage between the in sample fit and the out of sample performance? $\endgroup$ – Wintermute Mar 7 '15 at 14:25

This is about regularization. Suppose you would like to fit a curve and you use a square loss function (you can pick different). By fit you would like to recover the parameters that govern the process which generated that curve. Now imagine that you would like to fit this curve using 100th polynomial (just for example). You are pretty likely going to overfit or capture every kink and noise of the curve. In addition, your prediction capabilities outside the given training data interval will be likely very poor. So, regularization term is added to the objective function with some weight multiplied by the regularization factor - l_1, l_2 or custom. In the case of l_2, which is arbuably simpler to understand, this will have an effect that the large parameter values will be forced to reduce aka shrink. You can think of regularization or shrinkage as driving your algorithm to a solution which might be a better solution.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.