In OLS, residuals are guaranteed to sum to zero because that's how OLS is essentially defined/derived since the residual vector is orthogonal to the column space spanned by by the $p$ independent variables.

However, when you add the regularization sum, my intuition tells me that residuals will no longer sum to zero. After all, the solution to OLS should be the ONLY result such that residuals sum to zero. So when you regularize, you are surely to get a different result, and hence residuals no longer sum to zero? Is this understanding correct?

I haven't seen this talked about.

What is the significance of residuals no longer summing to zero for ridge regression? Is there an analogous visualization of Ridge regression as there is for OLS (e.g., orthogonality of residuals to column space of $X$, $\hat{y}$ being a projection of observed $y$ onto the column space of $X$)

In addition, even though the offset term is not regularized, I believe that it is likely that ridge regression will produce a different offset than OLS.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.