# Rationale behind shrinking regression coefficients in Ridge or LASSO regression

I understand that with Ridge or Lasso regression we are trying to shrink regression coefficients, and we specify the amount of shrinking we need by varying alpha. But I cannot understand the intuition or rationale behind doing it? as we no longer fit the best line.

Here is the general intuition behind shrinking co-efficients in linear regression. Borrowing figures and equations from Pattern Recognition and Machine Learning by Bishop.

Imagine that you have to approximate the function, $$y = sin(2\pi x)$$ from $$N$$ observations. You can do this using linear regression, which approximates the $$M$$ degree polynomial,

$$y(x, \textbf{w}) = \sum_{j=0}^{M}{w_j x^j}$$ by minimizing the error function,

$$E( \textbf{w}) = \frac{1}{2} \sum_{n=1}^{N}{\{ y(x_n, \textbf{w}) - t_n \}^2}$$

By choosing different values of $$M$$, one can fit different degree polynomials of varying complexity. Here are some example fits (red lines) and the corresponding values of $$M$$. Blue dots represent the observations and the green line is the true underlying function. The goal is to fit a polynomial which closely approximates the underlying function (green line).

Watch what happens with the high degree polynomial, $$M=9$$. This polynomial gives the minimum error, since it passes through all the points. But this is not a good fit, because the model is fitting to the noise structure of the data, rather than the underlying function. Since the overall goal of linear regression is to be able to predict $$t$$ for an unknown value of $$x$$, you will be screwed with the model with the high degree polynomial!

Further, let's take a look at the values of the regression coefficients. Watch how the values of the coefficients exploded for the higher degree polynomial.

The solution to this problem is regularization! Where, the error function to be minimized, is redefined as follows:

$$E'( \textbf{w}) = \frac{1}{2} \sum_{n=1}^{N}{\{ y(x_n, \textbf{w}) - t_n \}^2} + \frac{\lambda}{2} \Vert \textbf{w} \Vert^2$$

This gives us the formulation of Ridge regression. The inclusion of the penalty term, $$L2$$ norm, discourages the values of the regression coefficients from reaching high values, thus preventing over-fitting.

Lasso has a similar formulation, where the penalty term is the $$L1$$ norm. Inclusion of the lasso penalty term has an interesting effect -- it drives some of the coefficients to zero giving a sparse solution.

$$E''( \textbf{w}) = \frac{1}{2} \sum_{n=1}^{N}{\{ y(x_n, \textbf{w}) - t_n \}^2} + \frac{\lambda}{2} \Vert \textbf{w} \Vert^1$$

The question is what we really mean by "fit the best line." Yes, a standard unpenalized regression will be the "best line" to fit the data sample that you have. It might not, however, be the "best line" for a new sample from the population, as that standard unpenalized regression might pick up quirks that are peculiar to the particular data sample that you have. Penalizing the regression coefficients trades off a "worse" fit to the present data sample for a better fit to future samples, particularly important if the regresssion model is to be used for predictions.

In response to comment:

1. I don't think anyone can guarantee that penalization will always perform better than unpenalized regression, but penalization does help to avoid some potentially important problems particularly in the context of a predictive model. An Introduction to Statistical Learning is a good introduction to this and to many other issues in statistical modeling. Try repeating the modeling (both standard and penalized) on multiple bootstrap samples from your data set and see how well the standard and penalized models fit the full data set to get an idea about how this might play out in your situation.

2. The issue of scaling data for regularization is complicated. The usual approach is to standardize all predictors to unit standard deviation before penalization. The idea is to treat all predictors as equally as possible; you don't want different penalization if, for example, you measure lengths in millimeters versus kilometers. But standardization isn't straightforward with categorical predictors, and sometimes it's important to ensure that some predictors aren't penalized. There is also the issue of how the penalization coefficient is chosen. So how the coefficients of particular predictors will differ between penalized and standard regression models depends on details of the analysis. In ridge regression they will typically be lower in magnitude in the penalized model, but in LASSO the complete removal of some predictors from the penalized model might require predictors correlated to the removed predictors to have higher-magnitude coefficients than they would in the standard regression.

• Thanks for your inputs. Can you also advise penalizing guarantee better results? Moreover, even after regularization, if we don't scale features, we may still end up getting large coefficients, not bigger than before regularization. isn't it? – AjitKrish Jan 12 '18 at 1:27
• @AjitKrish I've expanded my answer to address your comment. – EdM Jan 12 '18 at 15:56

Shrinkage method will limit the value of regression coefficients, which will avoid model overfitting.

From Elements of Statistical Learning

When there are many correlated variables in a linear regression model, their coefficients can become poorly determined and exhibit high variance. A wildly large positive coefficient on one variable can be canceled by a similarly large negative coefficient on its correlated cousin. By imposing a size constraint on the coefficients [...] this problem is alleviated.

The ridge regression squared penalty term can be thought as follows. The residual sum of square (RSS) writes

$$RSS(\beta)=\sum_{i=1}^{n}[y_{i}-\beta_{0}-\sum_{j=1}^{p}(x_{ij}\beta_{j})]^{2}$$.

We add p dummy variables $$y_{j+n}=0$$ for $$j=1..p$$ and corresponding inputs $$x_{n+i,j}$$ such that $$x_{n+j,j}=\sqrt{\lambda}$$ and $$x_{n+i,j}=0, i\neq j$$. Rewriting the RSS including those p variables we obtain $$RSS^{ridge} = RSS(\beta) + \lambda \sum_{j=1}^{p}\beta_{j}^{2}$$.

When $$\lambda$$ increases, non-zero elements of the $$p$$ added variables also increase, and their influence in the regression increase. When $$\lambda \to \infty$$ the coefficients tend to zero.

Ridge regression is mathematically equivalent to a Bayesian regression with a gaussian prior with $\beta$ centered around 0 (see page 11 of the Gaussian Process book here). I think that's helpful as a way to understand when to deploy ridge regression.
If you are worried that the data may generate unnaturally large $\beta$ (say, many $x$ share a high linear correlation) you might think of the ridge regression as a way to limit that problem. It tends to produce smaller $\beta$ than the corresponding OLS.

Lasso regression tends to set a lot of $\beta$ to zero. It is useful in studies where you have many potential $x$ but only a few observations (say, genetics) or where you don't have a strong theoretical reason to believe that all $x$ matter.

I think the most common approach is actually to do both lasso and ridge at the same time (the elastic nets approach).
R has the beautiful covnet package precisely for this purpose (including self-tuning by cross-validation).

Of course each problem is different and what you should do is to withhold some data and test prediction error after the fits are done.

• "Ridge regression is mathematically equivalent to a Bayesian regression with a prior assuming all $\beta$ are zero." I don't think this is the case. If the prior were that all $\beta$ are exactly zero, then the data wouldn't matter: Our posteriors would always be that all $\beta$ are always zero. And most of the time, priors for coefficients are still centered around zero. So perhaps it is setting all $\beta$ to zero, with a narrow scale? Still then, it doesn't seem to map conceptually onto ridge or LASSO, which are fundamentally just adding penalties to $\beta$ estimates. – Mark White Jan 11 '18 at 4:52
• Yes, pardon, of course it's not prior 100% zero, it's a gaussian prior. But they are mathematically equivalent: for the proper proof check page 11 of the GP book: gaussianprocess.org/gpml/chapters/RW.pdf – CarrKnight Jan 11 '18 at 4:59