About

Ridge Regression is a technique which penalizes the size of regression coefficients in order to deal with multicollinear variables or ill-posed statistical problems. It is based on the Tikhonov regularization named after the mathematician Andrey Tikhonov.

Given a set of training data $(x_1,y_1),...,(x_n,y_n)$ where $x_i \in \mathbb{R}^{J}$, the estimation problem is:

$$\min_\beta \sum\limits_{i=1}^{n} (y_i - x_i'\beta)^2 + \lambda \sum\limits_{j=1}^J \beta_j^2$$

for which the solution is given by

$$\widehat{\beta}_{ridge} = (X'X + \lambda I)^{-1}X'y$$

which is similar to the OLS estimator but including the tuning parameter $\lambda$ and the Tikhonov matrix (in this case $I$, the identity matrix but other choices are possible). Note that, unlike the OLS estimator, the ridge regression estimator is always invertible even if there are more parameters in the model than degrees of freedom and hence there always exists a unique solution to the estimation problem.

Bayesian derivation

Ridge regression is equivalent to Bayesian linear regression assuming a Normal prior on $\beta$. Define the likelihood:

$$L(X,Y;\beta,\sigma^2) = \prod_{i=1}^{n}\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(y_i - \beta^Tx_i)^2}{2\sigma^2}}$$

And using a normal prior with mean 0 and variance $\alpha I_p$ on $\beta$:

$$\pi(\beta) \sim N(0,\alpha I_p)$$

Using Bayes rule, we calculate the posterior distribution:

$$P(\beta | X,Y) \propto L(X,Y;\beta,\sigma^2)\pi(\beta) $$ $$ \propto \big[\prod_{i=1}^{n}\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(y_i - \beta^Tx_i)^2}{2\sigma^2}}\big]e^{-\frac12\beta^T(\alpha^2 I_p)^{-1}\beta}$$

Maximizing the posterior is equivalent to minimizing the negative of the log of the posterior (after some algebra):

$$log (P(\beta | X,Y)) \propto -\frac12\big(\frac{1}{\sigma^2}\sum_{i=1}^{n}(y_i - \beta^Tx_i)^2 + \frac{1}{\alpha}\beta^T\beta\big)$$ $$\propto \sum_{i=1}^{n}(y_i - \beta^Tx_i)^2 + \frac{\sigma^2}{\alpha}\sum_{j=1}^{p}\beta^2 $$

Where $\frac{1}{\alpha}$ is the tuning parameter, corresponding to the choice of $\lambda$ from above.

The tuning parameter $\lambda$ determines the degree of shrinkage of the regression coefficients. The idea is to introduce some degree of bias in order to improve the variance (see bias-variance trade-off). In cases of highly multicollinear variables a small increase in bias to trade off for a lower variance can have a substantial effect.

The bias of the ridge regression estimator is $$Bias(\widehat{\beta}) = -\lambda (X'X + \lambda I)^{-1} \beta$$ It is always possible to find $\lambda$ such that the MSE of the ridge regression estimator is smaller than that of the OLS estimator.

Note that as $\lambda \rightarrow 0, \beta_{ridge} \rightarrow \beta_{ols}$ and as $\lambda \rightarrow \infty, \beta_{ridge} \rightarrow 0$. It is therefore important how to choose the value for $\lambda$. Common methods for this decision include the use of information criteria (AIC or BIC) or (generalized) cross-validation.