8
$\begingroup$

The traditional ridge regression estimate is

$$ \hat{\beta}_{ridge} = (X^TX+\lambda I)^{-1} X^T Y $$

which comes from adding the penalty term $\lambda ||\beta||^2_2$.

I have been struggling to find literature on regularizing towards a particular value. In particular, I have looked at a ridge regression model that uses the form of penalty $\lambda ||\beta-B||^2_2$ where $B$ is the initial estimate of $\beta$ under the setting of Iteratively Reweighted Least Squares. In turn, the ridge regression estimate is

$$ \hat{\beta}_{ridge} = (X^TX+\lambda I)^{-1} (X^T Y + \lambda B). $$

The lambda parameter is also chosen to be very large ($\lambda=100000$) which makes it seem to me that the estimate is trying to converge to $B$.

Why regularize towards a value? Does this change the interpretation of $\beta$?

Any comments and/or citations would be greatly appreciated. Thanks!

$\endgroup$
  • 3
    $\begingroup$ I don't fully follow your question because it seems to concern several different things: regularization, IRLS, and focusing on a particular value. As far as the latter goes, though, if you were to replace $Y$ by $Y-XB$, then you can just apply standard Ridge Regression. Whether that's a good idea or not depends on what IRLS is accomplishing for your data: obviously the results could be extremely sensitive to the IRLS estimate. $\endgroup$ – whuber Oct 25 '16 at 20:12
  • $\begingroup$ I really just wanted to know the purpose of the penalty term they used and if the ridge estimate still has some interpretation. The $\beta$ here is a matrix in the paper but the $Y$ and $X$ are still vectors. What I can't seem to grasp is that in their final estimate of the predictor matrix. I would expect some structure where the diagonal dominates above and below have some contributions, think block diagonal. However, this is not the case so I wonder if the interpretation gets altered when using different combinations of penalty and ridge values. $\endgroup$ – CindyLhasapoo Oct 27 '16 at 1:46
  • $\begingroup$ I don't follow you, because it makes no mathematical or statistical sense for $X$ and $Y$ to be vectors and $\beta$ to be a matrix. Usually $X$ is the design matrix (it contains the values of all regressor variables), $Y$ is a vector (of responses), and $\beta$ is a vector of coefficients. If you wish to understand what the Ridge Regression estimate means, then review what Ridge Regression is: as I pointed out in my first comment, what you describe can be reformulated as a standard Ridge Regression model. $\endgroup$ – whuber Oct 27 '16 at 14:33
  • $\begingroup$ Regularize toward some other value than zero could be implemented using offsets, if the software implements that. $\endgroup$ – kjetil b halvorsen Nov 2 '17 at 14:41
  • $\begingroup$ At stats.stackexchange.com/a/311490/919, I provide the details of the argument that regularizing towards a particular value is the same thing as regularizing towards 0. That should help with the interpretation. $\endgroup$ – whuber Nov 2 '17 at 16:15
5
$\begingroup$

We have the cost function

$$\| \mathrm y - \mathrm X \beta \|_2^2 + \gamma \| \beta - \beta_0 \|_2^2$$

where $\gamma \geq 0$. The minimum is attained at

$$\hat{\beta} := ( \mathrm X^{\top} \mathrm X + \gamma \mathrm I )^{-1} ( \mathrm X^{\top} \mathrm y + \gamma \beta_0 )$$

Note that while $\mathrm X^{\top} \mathrm X$ may not be invertible, $\mathrm X^{\top} \mathrm X + \gamma \mathrm I$ is always invertible if $\gamma > 0$.

If $\gamma \gg 1$, then

$$\begin{array}{rl} \hat{\beta} &= ( \mathrm X^{\top} \mathrm X + \gamma \mathrm I )^{-1} ( \mathrm X^{\top} \mathrm y + \gamma \beta_0 )\\ &= ( \gamma^{-1} \mathrm X^{\top} \mathrm X + \mathrm I )^{-1} ( \gamma^{-1} \mathrm X^{\top} \mathrm y + \beta_0 )\\ &\approx ( \mathrm I - \gamma^{-1} \mathrm X^{\top} \mathrm X ) ( \beta_0 + \gamma^{-1} \mathrm X^{\top} \mathrm y )\\ &\approx ( \mathrm I - \gamma^{-1} \mathrm X^{\top} \mathrm X ) \beta_0 + \gamma^{-1} \mathrm X^{\top} \mathrm y\\ &= \beta_0 + \gamma^{-1} \mathrm X^{\top} \left( \mathrm y - \mathrm X \beta_0 \right)\end{array}$$

For large $\gamma$, we have the approximate estimate

$$\boxed{\tilde{\beta} := \beta_0 + \gamma^{-1} \mathrm X^{\top} \left( \mathrm y - \mathrm X \beta_0 \right)}$$

If $\gamma \to \infty$, then $\tilde{\beta} \to \beta_0$, as expected. Left-multiplying both sides by $\mathrm X$, we obtain

$$\mathrm X \tilde{\beta} = \mathrm X \beta_0 + \gamma^{-1} \mathrm X \mathrm X^{\top} \left( \mathrm y - \mathrm X \beta_0 \right)$$

and, thus,

$$\mathrm y - \mathrm X \tilde{\beta} = \left( \mathrm I - \gamma^{-1} \mathrm X \mathrm X^{\top} \right) \left( \mathrm y - \mathrm X \beta_0 \right)$$

which gives us $\mathrm y - \mathrm X \tilde{\beta}$, an approximation of the error vector for large but finite $\gamma$, in terms of $\mathrm y - \mathrm X \beta_0$, the error vector for infinite $\gamma$.

None of this seems particularly insightful or useful, but it may be better than nothing.

$\endgroup$
  • $\begingroup$ Rewrite $\beta-\beta_0=\alpha$ and $y=z-X\beta_0$. Now you have the usual Ridge Regression setup for the cost $||z-X\alpha||^2+\gamma||\alpha||^2$, allowing you to write down the solution immediately. $\endgroup$ – whuber Oct 25 '16 at 20:46
3
$\begingroup$

Conceptually it may help to think in terms of Bayesian updating: The penalty term is equivalent to a prior estimate $\beta_0$ with precision $\lambda$ (i.e. a multivariate Gaussian prior $\beta\sim\mathrm{N}_{\beta_0,\,I/\lambda}).$

In this sense a "very large" $\lambda$ does not correspond to any particular numerical value. Rather it would be a value which "dominates" the error, so numerically it must be large relative to some norm $\|X\|$ of the design matrix. So for your example we cannot say whether $\lambda=100000$ is "very large" or not, without more information.

That said, why might a "very large" value be used? A common case I have seen in practice is where the actual problem is equality constrained least squares, but this is approximated using Tikhonov Regularization with a "large $\lambda$". (This is slightly more general than your case, and would correspond to a "wide" matrix $\Lambda$, such that $\Lambda(\beta-\beta_0)=0$ could be solved exactly.)

$\endgroup$
  • $\begingroup$ Usually Ridge Regression is carried out only after standardizing the columns of $X$, allowing $\lambda$ to have some intrinsic meaning. $\endgroup$ – whuber Oct 25 '16 at 20:47
  • $\begingroup$ @whuber thank you for the information, which I did not know. $\endgroup$ – GeoMatt22 Oct 25 '16 at 20:56
  • $\begingroup$ I think it is $N_{\beta_0,I/\lambda}$ if the second parameter stands for the covariance matrix. $\endgroup$ – Benoit Sanchez Jul 20 '17 at 18:21
  • $\begingroup$ @BenoitSanchez thanks! Don't know what I was thinking, as I called it precision just 5 words before :) $\endgroup$ – GeoMatt22 Jul 20 '17 at 18:33
  • $\begingroup$ I was writing a similar answer when I read yours and thought one of us might be missing something :-) $\endgroup$ – Benoit Sanchez Jul 20 '17 at 18:40
2
$\begingroup$

I have an answer for "Why regularize towards a value? Does this change the interpretation of $\beta$?"

Transfer learning is a type of Machine Learning where knowledge from the source domain when performing a task is transfered to the target domain when performing the same task i.e. the task remains the same but datasets in the two domains differ.

One way to perform transfer learning is parameter sharing. The high level intuition is that target domain model parameters should be very close to source domain model parameters while still allowing for some uncertainty. Mathematically this intuition is captured by penalizing the deviation of the parameters i.e., $\lambda\|W_{target}−W_{source}\|^2_2$ , where, $λ$ is the penalization parameter and W's are a vector of model parameters.

I have used this approach to perform transfer learning for conditional random fields, look at Eq. 4 and related text.

I had a similar question for Ridge regression posted here on the interpretability of the closed form solution.

$\endgroup$
1
$\begingroup$

It is possible to understand it from a Bayesian point of view.

Ridge regularization for linear regression is a Bayesian method in disguise. See : https://en.wikipedia.org/wiki/Lasso_(statistics)#Bayesian_interpretation (it is easier to understand explained on the wikipedia"s Lasso page, but it's the same idea with Ridge).

The convention I use for regularization is the following. Minimize: $\left(\displaystyle\sum_{i=1}^N(y_i-\beta x_i)^2\right)+\lambda\|\beta-\beta_0\|^2$. Assume that the noise has variance $\sigma^2=1$ for simplicity (otherwise replace $\lambda$ by $\lambda/\sigma^2$ everywhere).

Regularization with coefficient $\lambda$ means assuming a normal prior $N(0;\frac{1}{\lambda}I)$: "I expect as a prior belief that the coefficients are small": The prior distribution is a normal distribution with mean $0$ and "radius" $\sqrt\frac{1}{\lambda}$. Regularizing towards $\beta_0$ means assuming a normal prior $N(\beta_0;\frac{1}{\lambda}I)$: "I expect as a prior belief that the coefficients are not far from $\beta_0$": the prior distribution is a normal distribution with mean $\beta_0$ and "radius" $\sqrt\frac{1}{\lambda}$.

This prior often results from a previous training that gave $\beta_0$ as an estimate. The strength of your belief $\lambda$ is the statistical power of your first training set. A big lambda means that you had previously a lot of information, your belief is only slightly changed for each new sample: a small update by sample.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.