Why shrinkage estimators?

Question

Iam trying to understand the usage of lasso and ridge regression. The advantage of both methods is that we get a lower variance in comparisson to the ols estimation and thus we get a better prediction.

But i dont understand how the estimators become smaller, because lambda is positive in the equation that we try to minimize to get the coeffiecients?

I also dont understand why we are interessted in smaller coeffiecents. I think its because of smaller variance but i have no explenation for it. Where is the link betweend smaller coefficients and smaller variance?

Answer 1

The aim of shrinkage of parameters is to obtain better generalisation for your linear estimator.

It is not true that you get lower variance with an OLS estimator. If you have a lot of data an OLS estimator is likely to perform better than a RIDGE or LASSO estimator.

We are interested in smaller variance on unseen data. If we start to model the noise with our linear model then on new samples, we will have larger error, because noise in this interpretation cannot be modelled.

I think the easiest way to interpret why shrinkage is useful is to consider that your data has some inherent noise. If this inherent noise has an impact larger than some of your coefficients, then it would be safe to assume that those coefficients just fit the noise in the data, and not the effects that can be modelled by your linear model.

Another way to understand is by the principle of Occam's razor. Say that your linear model is able to explain the same data point with multiple linear combinations of the functions that you've used in your linear estimator. If two solutions are equivalent, a sparser solution is desired, because it uses less of your functions to model the effects.

In some cases (i.e. when you have less data than parameters, so called ill-conditioned problem) you can't even do ordinary least squares, and in that LASSO and RIDGE still works.

Answer 2

Have a look at the following generalized form of regression:

$$ \arg \min_{x} \frac{1}{ {\sigma}_{n}^{2} } {\left\| A x - b \right\|}_{2}^{2} + \lambda R \left( x \right) $$

We can say this is a form of Maximum a Posteriori Estimation (MAP) where the first term is the Likelihood for Gaussian Model and the second term as the prior where $ \lambda $ is a parameterization of the prior.

For ridge regression we model a prior of Gaussian Distribution of $ x $ (With zero mean). In that case $ \lambda = \frac{1}{ {\sigma}_{x}^{2} } $ where $ {\sigma}_{x} $ is the STD of the prior model.

In the Lasso Case the prior is Laplacian with $ \lambda $ as the model parameter.

Answer 3

I suggest ,you go through the following answers:-

(1.) In Ridge regression and LASSO, why smaller $\beta$ would be better?

(2.)Geometric interpretation of penalized linear regression

There it is explained very nicely.