Is the shrinkage techique all about finite samples?

Question

With the right choice of the penalty, the ridge estimator in the linear regression model $$y=X \beta +\varepsilon, \quad E[X \varepsilon]=0, \quad \beta \in \mathbb{R}^k$$ will have a smaller MSE that the OLS estimator. The asymptotic distributions of the ridge estimator and the OLS estimator (that is, the distribution limits of $\sqrt{n}(\widehat{\beta}_{OLS}-\beta)$ and $\sqrt{n}(\widehat{\beta}_{ridge}-\beta)$) are the same.

Is shrinkage always just improving the properties of the estimator in a finite sample?

Does it make sense to use shrinkage in large samples (in the case of a finite-dimensional parameter of interest)?

Answer 1

Shrinkage is all about improving finite sample behavior. That's pretty important: I've never analyzed a data set that is not finite.

But more seriously, shrinkage methods are not just useful for data sets with relatively small samples sizes. They are very useful for data sets in which many of the variables have relatively little information about them. For example, you may have 1,000,000 rows of data, which I would say is a lot. But if you have 100,000 covariates, you don't have that much data relative to the number of parameters you are estimating. Or suppose you only have 10 variables, but one of them is 0 in all but a single case. Then all your inference about that single variable is essentially based on a single observation!

These are the types of cases that shrinkage methods can be useful for.

It can be helpful to realize that most penalized methods can be seen as simply putting a prior on the penalized variables and using MAP estimates. In the Bayesian setting, we know that priors can be helpful when the data is not informative, but (thankfully) gets drowned out when the data is very informative.