1
$\begingroup$

Is it possible to penalize coefficients toward a number other than zero in a ridge regression in R?

For example, let's say I have dependent variable Y and independent variables X1,X2,X3, and X4. Because of the multicollinear nature of the ivs, ridge regression is appropriate. But say I'm fairly certain that the coefficient of X1 is near 5, X2 is near 1, X3 is near -1, and X4 is near -5.

Is there a ridge package and method in R where I can implement penalties on the coefficients of the ivs that penalize them toward those numbers instead of 0? I'd love to see an example in R with my example data, if possible. Thank you.

$\endgroup$
  • $\begingroup$ For a solution that also works with logistic regression or other generalized linear models, see here $\endgroup$ – kjetil b halvorsen Jan 4 at 12:07
3
$\begingroup$

A simple way to do this is to subtract the "centering value" of the coefficient times its associated variable from the left-hand side. To go with your example,

$Y = \beta_1X_1 + \beta_2X_2 + \beta_3X_3 + \beta_4X_4 + e$

Assume the coefficient values should be centered at (5,1,-1,-5) respectively. Then:

$Y - 5X_1 -X_2 +X_3 +5X_4 = (\beta_1-5)X_1 + (\beta_2-1)X_2 + (\beta_3+1)X_3 + (\beta_4+5)X_4 + e$

and, redefining terms, you have:

$Y^* = \beta_1^*X_1 + \beta_2^*X_2 + \beta_3^*X_3 + \beta_4^*X_4 + e$

A standard ridge regression would shrink the $\beta_i^*$ towards 0, which is equivalent to shrinking the original $\beta_i$ towards the specified centering values. to see this, consider a fully-shrunk $\beta_4^* = 0$, then $\beta_4+5 = 0$ and therefore $\beta_4 = -5$. Shrinkage accomplished!

$\endgroup$
  • $\begingroup$ What if you want to do it with logistic regression? $\endgroup$ – David Masip Jan 4 at 11:10
1
$\begingroup$

Ridge regression sets a normal prior centered at zero for each parameter. To penalize coefficients towards different values, just center the priors around your target instead of around 0. I think you can accomplish this using the bayesGLM function of the arm package with the following parameters:

  • prior.mean : set this to vector containing the values you want to use as the centers for your priors (the values you want to penalize "towards")
  • prior.df : set this to Inf to get normal priors. Possibly not necessary. There should be a way to control your prior variance, but it's not clear to me how this would be accomplished from looking at the documentation.

http://cran.r-project.org/web/packages/arm/arm.pdf

$\endgroup$
  • $\begingroup$ Hmm, I'm sure I'm missing something but I tried this and it gave me the same coefficients as a normal lm() regression $\endgroup$ – Chris Nov 19 '13 at 5:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.