I have the following question regarding the Proof of "Shrinkage" in Statistics.
Up to the present moment, I always thought that "Shrinkage" was a synonymous term with "Regularization", and referred to a technique in statistics used to combat the "Bias-Variance Tradeoff" within predictive models. In the context of fitting statistical models to complex data : Simple Models tend to be unable to adequately model complex patterns (high bias, low variance) whereas Complex Models tend to better model complex patterns, but have a bad tendency to overfit the data:
Research over the years suggests that the amount of overfitting experienced by complex models can be reduced by "shrinking" the model parameters towards zero. This can be achieved by add a regularization term to the loss function of the regression model (below: right). On the left hand side of the picture below, the regularization term is represented by the green shapes. In a model with 2 parameters (e.g. theta_0, theta_1), the regularization term "pushes" thetha_0 and thetha_1 towards the (0,0) coordinates, thus shrinking their values:
Are there any standard proofs suggesting that Shrinkage is able to "shrink" multiple parameters towards 0 (i.e. promise sparse results)?
Are there any standard proofs that suggest overfitting is more likely to be caused by models with many parameters compared to fewer parameters? (modern ML models like GPT3 are said to have "millions of parameters" and perform without overfitting)
In general, are there any standard proofs that highlight the usefulness and utility of Shrinkage ?