For some reason, I can't seem to find the formula for prediction intervals in ridge regression anywhere. I know that the coefficient estimates are biased, but are the predictions (dependent variable) biased as well?

  • 1
    $\begingroup$ Since the predictions are a linear function of the coefficients, it would be surprising if they weren't biased. $\endgroup$
    – Glen_b
    Jun 19, 2013 at 23:21
  • $\begingroup$ @Glen_b Biased in what way, though? I don't think this necessarily follows... Consider 2 predictors, x1 and x2, s.t. x1 = x2 + (very, very small noise), and a truth function y = x1 + x2. OLS might return you something like y = 3000x1 - 2998x2, and RR will probably give you something close to y = x1 + x2. In my poor understanding of bias, I don't think either of these would give biased estimates... $\endgroup$
    – Jake
    Jun 20, 2013 at 15:52
  • 1
    $\begingroup$ While it's possible to construct situations where parameters are biased but the biases cancel, this won't generally be the case for ridge regression, where the parameters are shrunk toward zero. As a result, it really would be surprising if the result weren't biased. You may, however, be mistakenly interpreting me as suggesting that bias is a bad thing. Ceteris paribus, perhaps (though I think far too much is made of bias even then), but the thing is the "ceteris" is really not "paribus"; what we're doing (as typical of regularization) is trading a bit of bias for a lot of variance $\endgroup$
    – Glen_b
    Jun 20, 2013 at 22:52
  • 1
    $\begingroup$ I think this related answer on confidence interval for ridge should also address your question. $\endgroup$
    – WillZ
    Dec 28, 2016 at 0:18