For some reason, I can't seem to find the formula for prediction intervals in ridge regression anywhere. I know that the coefficient estimates are biased, but are the predictions (dependent variable) biased as well?

  • 1
    $\begingroup$ Since the predictions are a linear function of the coefficients, it would be surprising if they weren't biased. $\endgroup$ – Glen_b Jun 19 '13 at 23:21
  • $\begingroup$ @Glen_b Biased in what way, though? I don't think this necessarily follows... Consider 2 predictors, x1 and x2, s.t. x1 = x2 + (very, very small noise), and a truth function y = x1 + x2. OLS might return you something like y = 3000x1 - 2998x2, and RR will probably give you something close to y = x1 + x2. In my poor understanding of bias, I don't think either of these would give biased estimates... $\endgroup$ – Jake Jun 20 '13 at 15:52
  • $\begingroup$ While it's possible to construct situations where parameters are biased but the biases cancel, this won't generally be the case for ridge regression, where the parameters are shrunk toward zero. As a result, it really would be surprising if the result weren't biased. You may, however, be mistakenly interpreting me as suggesting that bias is a bad thing. Ceteris paribus, perhaps (though I think far too much is made of bias even then), but the thing is the "ceteris" is really not "paribus"; what we're doing (as typical of regularization) is trading a bit of bias for a lot of variance $\endgroup$ – Glen_b Jun 20 '13 at 22:52
  • $\begingroup$ I think this related answer on confidence interval for ridge should also address your question. $\endgroup$ – WillZ Dec 28 '16 at 0:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.