Ridge regression: regularizing towards a value

Question

The traditional ridge regression estimate is

$$ \hat{\beta}_{ridge} = (X^TX+\lambda I)^{-1} X^T Y $$

which comes from adding the penalty term $\lambda ||\beta||^2_2$.

I have been struggling to find literature on regularizing towards a particular value. In particular, I have looked at a ridge regression model that uses the form of penalty $\lambda ||\beta-B||^2_2$ where $B$ is the initial estimate of $\beta$ under the setting of Iteratively Reweighted Least Squares. In turn, the ridge regression estimate is

$$ \hat{\beta}_{ridge} = (X^TX+\lambda I)^{-1} (X^T Y + \lambda B). $$

The lambda parameter is also chosen to be very large ($\lambda=100000$) which makes it seem to me that the estimate is trying to converge to $B$.

Why regularize towards a value? Does this change the interpretation of $\beta$?

Any comments and/or citations would be greatly appreciated. Thanks!

Answer 1

We have the cost function

$$\| \mathrm y - \mathrm X \beta \|_2^2 + \gamma \| \beta - \beta_0 \|_2^2$$

where $\gamma \geq 0$. The minimum is attained at

$$\hat{\beta} := ( \mathrm X^{\top} \mathrm X + \gamma \mathrm I )^{-1} ( \mathrm X^{\top} \mathrm y + \gamma \beta_0 )$$

Note that while $\mathrm X^{\top} \mathrm X$ may not be invertible, $\mathrm X^{\top} \mathrm X + \gamma \mathrm I$ is always invertible if $\gamma > 0$.

If $\gamma \gg 1$, then

$$\begin{array}{rl} \hat{\beta} &= ( \mathrm X^{\top} \mathrm X + \gamma \mathrm I )^{-1} ( \mathrm X^{\top} \mathrm y + \gamma \beta_0 )\\ &= ( \gamma^{-1} \mathrm X^{\top} \mathrm X + \mathrm I )^{-1} ( \gamma^{-1} \mathrm X^{\top} \mathrm y + \beta_0 )\\ &\approx ( \mathrm I - \gamma^{-1} \mathrm X^{\top} \mathrm X ) ( \beta_0 + \gamma^{-1} \mathrm X^{\top} \mathrm y )\\ &\approx ( \mathrm I - \gamma^{-1} \mathrm X^{\top} \mathrm X ) \beta_0 + \gamma^{-1} \mathrm X^{\top} \mathrm y\\ &= \beta_0 + \gamma^{-1} \mathrm X^{\top} \left( \mathrm y - \mathrm X \beta_0 \right)\end{array}$$

For large $\gamma$, we have the approximate estimate

$$\boxed{\tilde{\beta} := \beta_0 + \gamma^{-1} \mathrm X^{\top} \left( \mathrm y - \mathrm X \beta_0 \right)}$$

If $\gamma \to \infty$, then $\tilde{\beta} \to \beta_0$, as expected. Left-multiplying both sides by $\mathrm X$, we obtain

$$\mathrm X \tilde{\beta} = \mathrm X \beta_0 + \gamma^{-1} \mathrm X \mathrm X^{\top} \left( \mathrm y - \mathrm X \beta_0 \right)$$

and, thus,

$$\mathrm y - \mathrm X \tilde{\beta} = \left( \mathrm I - \gamma^{-1} \mathrm X \mathrm X^{\top} \right) \left( \mathrm y - \mathrm X \beta_0 \right)$$

which gives us $\mathrm y - \mathrm X \tilde{\beta}$, an approximation of the error vector for large but finite $\gamma$, in terms of $\mathrm y - \mathrm X \beta_0$, the error vector for infinite $\gamma$.

None of this seems particularly insightful or useful, but it may be better than nothing.

Answer 2

Conceptually it may help to think in terms of Bayesian updating: The penalty term is equivalent to a prior estimate $\beta_0$ with precision $\lambda$ (i.e. a multivariate Gaussian prior $\beta\sim\mathrm{N}_{\beta_0,\,I/\lambda}).$

In this sense a "very large" $\lambda$ does not correspond to any particular numerical value. Rather it would be a value which "dominates" the error, so numerically it must be large relative to some norm $\|X\|$ of the design matrix. So for your example we cannot say whether $\lambda=100000$ is "very large" or not, without more information.

That said, why might a "very large" value be used? A common case I have seen in practice is where the actual problem is equality constrained least squares, but this is approximated using Tikhonov Regularization with a "large $\lambda$". (This is slightly more general than your case, and would correspond to a "wide" matrix $\Lambda$, such that $\Lambda(\beta-\beta_0)=0$ could be solved exactly.)

Answer 3

I have an answer for "Why regularize towards a value? Does this change the interpretation of $\beta$?"

Transfer learning is a type of Machine Learning where knowledge from the source domain when performing a task is transfered to the target domain when performing the same task i.e. the task remains the same but datasets in the two domains differ.

One way to perform transfer learning is parameter sharing. The high level intuition is that target domain model parameters should be very close to source domain model parameters while still allowing for some uncertainty. Mathematically this intuition is captured by penalizing the deviation of the parameters i.e., $\lambda\|W_{target}−W_{source}\|^2_2$ , where, $λ$ is the penalization parameter and W's are a vector of model parameters.

I have used this approach to perform transfer learning for conditional random fields, look at Eq. 4 and related text.

I had a similar question for Ridge regression posted here on the interpretability of the closed form solution.

Answer 4

It is possible to understand it from a Bayesian point of view.

Ridge regularization for linear regression is a Bayesian method in disguise. See : https://en.wikipedia.org/wiki/Lasso_(statistics)#Bayesian_interpretation (it is easier to understand explained on the wikipedia"s Lasso page, but it's the same idea with Ridge).

The convention I use for regularization is the following. Minimize: $\left(\displaystyle\sum_{i=1}^N(y_i-\beta x_i)^2\right)+\lambda\|\beta-\beta_0\|^2$. Assume that the noise has variance $\sigma^2=1$ for simplicity (otherwise replace $\lambda$ by $\lambda/\sigma^2$ everywhere).

Regularization with coefficient $\lambda$ means assuming a normal prior $N(0;\frac{1}{\lambda}I)$: "I expect as a prior belief that the coefficients are small": The prior distribution is a normal distribution with mean $0$ and "radius" $\sqrt\frac{1}{\lambda}$. Regularizing towards $\beta_0$ means assuming a normal prior $N(\beta_0;\frac{1}{\lambda}I)$: "I expect as a prior belief that the coefficients are not far from $\beta_0$": the prior distribution is a normal distribution with mean $\beta_0$ and "radius" $\sqrt\frac{1}{\lambda}$.

This prior often results from a previous training that gave $\beta_0$ as an estimate. The strength of your belief $\lambda$ is the statistical power of your first training set. A big lambda means that you had previously a lot of information, your belief is only slightly changed for each new sample: a small update by sample.