I'm using ridge regression for estimating the fair value of variable $Y$ against a vector of correlated predictor variables $X_i$, based on past observations of both $Y$ and $X_i$. Let's assume that both Y and Xi are standardized (mean $= 0$, std $= 1$).

Problem is that ridge regression introduces bias into the estimate of $Y$, and the further the predictor vector is from zero, the greater the bias. In order to correct for the bias, I perform another regression of $Y$ onto one variable, which is the linear sum of predictors with coefficients calculated from the ridge regression.

It seems that this procedure both benefits from the stable coefficients calculated using ridge and from the subsequent bias-correction, which looks like a double win.

Is this procedure "valid"? Does anybody do the same? Is there any literature on such kind of bias correction?

  • 1
    $\begingroup$ How are you calculating the shrinkage parameter of the ridge regression? $\endgroup$ – jbowman Aug 11 '14 at 19:52
  • $\begingroup$ It sounds wrong...as though perhaps your shrinkage parameter is too strong...which leads to jbowmans question $\endgroup$ – seanv507 Aug 11 '14 at 21:34
  • $\begingroup$ Thanks for comments. To be clear, let me provide you with exact formulas I use for calculation: betta_ridge = (X'X + kI)^(-1) * X'*y, where betta_ridge is a vector of regression coefficients. After that, I calculate bias-correction regression: y = gamma * y_hat (no constant term since all variables are standardized), where y_hat is the estimate from the ridge regression: y_hat = X * betta_ridge. So we get: y_est = gamma * x * betta_ridge. If k is greater than zero, than gamma is greater than one. 2 jbowman: I consider k as an input, rather than an estimated parameter here. $\endgroup$ – Student Aug 12 '14 at 7:30
  • $\begingroup$ so that is the problem- you cannot consider it an input...you have to search for the right k, because eg for large enough k the optimal weights would be zero $\endgroup$ – seanv507 Aug 12 '14 at 14:24

Unfortunately, this procedure turns out wrong. Based on Monte-Carlo tests, simple ridge regression performs better than "bias-corrected" variant.

| cite | improve this answer | |
  • 3
    $\begingroup$ If you do not like the bias introduced by ridge regression, maybe you should use OLS. $\endgroup$ – Michael M Aug 12 '14 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.