When you fit a model to a "learning" data set, and then apply the same model to new data, the coefficient of determination shrinks, and this "shrinkage" is a measure of how well a model predicts in new data versus the original data. LASSO methods are used in estimating predictive models, and they rely on "shrinkage" of coefficient estimates for model selection. Is this just an irritating and unfortunate coincidence that the same term refers to two different things in the prediction world, or do the two uses have something to do with each other? I am interested in model selection methods that help minimize that first use of shrinkage, maximizing r-squared in out of sample data. When I first saw the LASSO method ("Least Absolute Shrinkage...") I thought it was a short cut to a better predictor. Seems like you still have to do something else to minimize shrinkage of the r-squared. I've done a lot of cross-validation work, but tried playing with a LASSO routine for the first time this week. Is it the right direction to go?
-
3$\begingroup$ I've never heard anyone use the word "shrinkage" in the first sense. You are referring to "generalization error" there. $\endgroup$– Matthew DruryCommented Oct 21, 2016 at 13:51
-
$\begingroup$ It must be used more commonly in Psychology and Educational Statistics. Those articles are where I first encountered the term: Yin, P., & Fan, X. (2001). Estimating R-2 shrinkage in multiple regression: A comparison of different analytical methods. JOURNAL OF EXPERIMENTAL EDUCATION, 69(2), 203–224. If I can find search terms that help me separate the two, then I'll be happy. $\endgroup$– General TurgidsonCommented Oct 21, 2016 at 17:01
1 Answer
The term shrinkage is used in regression modelling to denote two, somewhat related ideas.
The first relates to the slope of a calibration plot, which is a plot of observed responses against predicted responses. When parameter estimates are derived from one dataset and then applied to predict the outcome on an independent dataset, overfitting will cause the slope of the calibration plot (i.e. shrinkage factor) to be less than one, a result of regression to the mean. Typically low predictions will be too low and high predictions will be too high.
The second meaning of shrinkage (which is what the OP is asking) is a statistical estimation method that preshrinks regression coefficients towards zero so that the calibration plot for new data will not need shrinkage as its calibration slope will be one. Penalized maximum likelihood estimation is a general shrinkage procedure, in which both LASSO regression and Ridge regression are special cases.
More details can be found in:
F. E. Harrell, Regression Modelling Strategies, P75-78. https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html
