Proof of "Shrinkage" in Statistics

Question

I have the following question regarding the Proof of "Shrinkage" in Statistics.

Up to the present moment, I always thought that "Shrinkage" was a synonymous term with "Regularization", and referred to a technique in statistics used to combat the "Bias-Variance Tradeoff" within predictive models. In the context of fitting statistical models to complex data : Simple Models tend to be unable to adequately model complex patterns (high bias, low variance) whereas Complex Models tend to better model complex patterns, but have a bad tendency to overfit the data:

Research over the years suggests that the amount of overfitting experienced by complex models can be reduced by "shrinking" the model parameters towards zero. This can be achieved by add a regularization term to the loss function of the regression model (below: right). On the left hand side of the picture below, the regularization term is represented by the green shapes. In a model with 2 parameters (e.g. theta_0, theta_1), the regularization term "pushes" thetha_0 and thetha_1 towards the (0,0) coordinates, thus shrinking their values:

My Question:

• Are there any standard proofs suggesting that Shrinkage is able to "shrink" multiple parameters towards 0 (i.e. promise sparse results)?

• Are there any standard proofs that suggest overfitting is more likely to be caused by models with many parameters compared to fewer parameters? (modern ML models like GPT3 are said to have "millions of parameters" and perform without overfitting)

• In general, are there any standard proofs that highlight the usefulness and utility of Shrinkage ?

Thanks!

References

• https://en.wikipedia.org/wiki/Shrinkage_(statistics)

Answer 1

• Are there any standard proofs suggesting that Shrinkage is able to "shrink" multiple parameters towards 0 (i.e. promise sparse results)?

If you take regression with $\ell_1$ penalty

$$ \underset{\boldsymbol\beta}{\operatorname{arg\,min}} \;\| \boldsymbol{y} - \mathbf{X}\boldsymbol\beta \|_2^2 + \lambda \| \boldsymbol\beta \|_1 $$

Then for regularization to have no effect (be the same as regular linear regression), the $\lambda \| \boldsymbol\beta \|_1$ term would need to be always zero. The term is minimized by minimizing the weights.

• Are there any standard proofs that suggest overfitting is more likely to be caused by models with many parameters compared to fewer parameters? (modern ML models like GPT3 are said to have "millions of parameters" and perform without overfitting)

Model with more parameters is more flexible so it can more easily fit perfectly to the (training) data. That said, there’s some recent research showing the “double descent” phenomenon, where this does not hold. However keep in mind that we are concerned with number of parameters relatively to size of data and the number of patterns in the data to be learned, where language data consists of millions of samples and there’s an infinite number of words and their combinations that are possible, so relatively the number of parameters of GPT3 is not necessarily huge.

• In general, are there any standard proofs that highlight the usefulness and utility of Shrinkage ?

Yes, you can find them in any machine learning handbook since regularization is always discussed.