6
$\begingroup$

I'm using scikit learn's linear model to do ridge regression. Ridge regression penalizes parameters for moving away from zero. I want to penalize for moving away from a certain prior, with each parameter having a different prior.

Is this possible with scikit learn's linear model? I know there's a BayesianRidge module there, but I'm not sure what it does.

$\endgroup$
10
$\begingroup$

Ridge regression looks like:

$$ \min_{\beta}||Y-X\beta||^2 + \lambda_1 ||\beta||^2 $$

If you want to instead compute

$$ \beta^* = \arg\min_{\beta}||Y-X\beta||^2 + \lambda_1 ||\beta - \beta_0||^2 $$

I guess you could just turn this into shrinking towards zero using the new variable

$$\theta = \beta - \beta_0.$$

So you'd solve:

$$ \theta^* := \arg\min_{\theta}||Y-X\beta_0-X \theta||^2 + \lambda_1 ||\theta||^2 $$

Then apply the change of variables again (i.e., $\beta^* := \theta^* + \beta_0$).

So to recap, if I have some black box function $\text{RidgeRegression}(Y,X, \lambda)$, I can use it to solve for an arbitrary prior $\beta_0$ simply by calling $\text{RidgeRegression}(Y-X\beta_0, X, \lambda)$.

$\endgroup$
  • 3
    $\begingroup$ Unless I'm missing something, $\text{RidgeRegression}(Y - X \beta_0, X, \lambda)$ gives you $\theta^*$, but you want $\beta^*$, so you actually want to replace $\text{RidgeRegression}(Y, X, \lambda)$ with $\text{RidgeRegression}(Y - X \beta_0, X, \lambda) + \beta_0$. $\endgroup$ – Dougal Jun 30 '14 at 0:30
0
$\begingroup$

What's posted in the only answer by Dapz does not do what it's supposed to do. If I choose a value > 0 for any of the $\beta_0$, say the "i-th", the corresponding $\beta^*$ of "i" will be lower than with standard ridge regression, instead of higher as it should be (because we penalize for moving away from something > 0, instead of moving away from 0).

$\endgroup$
  • $\begingroup$ can you post some reproducible code? $\endgroup$ – Cam.Davidson.Pilon Jan 11 '13 at 19:49
  • $\begingroup$ Did you intend to write this as a comment beneath Dapz's reply? Also, I've merged your two accounts but you will need to register this one to to benefit from all the advantages of the Stack Exchange community. $\endgroup$ – chl Jan 11 '13 at 22:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.