# Why does regularization wreck orthogonality of predictions and residuals in linear regression?

Following up on this question...

In ordinary least squares, the predictions and residuals are orthogonal. $$\sum_{i=1}^n\hat{y}_i (y_i - \hat{y}_i) = 0$$

If we estimate the regression coefficients using some other method but the same model, such as using regularization, why, intuitively, should that wreck the orthogonality?

I wrote a comprehensive explanation on this question in my site. It might be useful for readers.

I'll talk about the ridge regularization here because it can be shown to neatly use the same equations used to derive the OLS solution (see this answer).

The coefficients in ridge regression (with penalty weighting $$\lambda$$) are simply:

$$\beta = (X^TX+\lambda\mathbb I)^{-1}X^Ty$$

The solution to the OLS can be obtained just as well by setting $$\lambda = 0$$.

The use of the normal equations to the ridge problem can be recovered from and correspond to an augmentation of $$X$$. Concatenating new virtual samples formed by a identity matrix:

$$\matrix{ X_\text{new}=\left[\matrix{ X_\text{old} \\ \sqrt{\lambda}\mathbb I_{p\times p} }\right] \qquad Y_\text{new}=\left[\matrix{ Y_\text{old} \\ \mathbf 0_{p\times1} }\right] }$$

If we do that, it can be quite straightforwardly shown that:

$$\beta = (X_\text{old}^TX_\text{old}+\lambda\mathbb I)^{-1}X^T_\text{old} y_\text{old} = (X_\text{new}^TX_\text{new})^{-1}X_\text{new}^T y_\text{new}$$

Thus, since we are using the normal equations to derive the solution to ridge regression, the property of orthogonal residuals and predictions is kept intact.

But notice that, now, predictions involve these virtual samples. That's why, when looking only at the real samples, this orthogonality is not guaranteed: you are missing part of the puzzle by not taking into account these "virtual" samples.

An image might help. In this image, we see a geometric view of the fitting.

• Least squares finds a solution in a plane that has the closest distance to the observation.

(more general a higher dimensional plane for multiple regressors and a curved surface for non-linear regression)

In this case, the vector between observation and solution is perpendicular to the plane (a space spanned be the regressors), and perpendicular to the regressors.

• Regularized regression finds a solution in a restricted set inside the the plane that has the closest distance to the observation.

In this case, the vector between observation and solution is not anymore perpendicular to te plane and not anymore perpendicular to the regressors.

But, there is still some sort of perpendicular relation, namely the vector of the residuals is in some sense perpendicular to the edge of the circle (or whatever other surface that is defined by te regularization) ### The model of $$\hat{y}$$

Our model gives estimates of the observations, $$\hat{y}$$, the observations as function of parameters $$\beta_i$$.

$$\hat{y} = f(\beta)$$

In our image this is a linear function with two parameters $$\beta_0$$ and $$\beta_1$$

(you can of course generalize this to a large size of coefficients and observations, for simplicity we regard three observations and two coefficients such that we can plot it)

$$\begin{bmatrix} \hat{y}_{1} \\ \hat{y}_{2} \\ \hat{y}_{3} \end{bmatrix} = \beta_0 \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}+ \beta_1 \begin{bmatrix} 0 \\ 1 \\ 2 \end{bmatrix}$$

The possible solutions of the model, defined by this linear sum, is represented by the red plane in the image.

Note that this plane in the image relates to the possible solutions of $$y_i = \beta_0 + \beta_1 x_i$$, when $$[x_1,x_2,x_3] = [0,1,2]$$. So we plotted the space of all possible $$y_i$$ (which is a 3D-space, and more generally a n-dimensional space) and the possible solutions that the model allows is a plane inside this space.

### Finding the best model with least squares

The model allows any solution in the plane spanned by the model (in the image this is the 2D red plane, in general this can be a higher dimensional plane, and also it does not need to be linear).

The least-squares method will select the 'solution' $$\hat{y} = \hat\beta_0 + \hat\beta_1 x_1$$ that has the lowest difference in terms of the squares of the residuals.

In geometric terms, this is equal to finding the point in the plane that has the smalles euclidian distance to the observed value. This smallest difference is achieved when the vector of residuals is orthogonal to the plane.

### Finding the best model with ridge regression (or other regularization)

When we apply a penalty then this is similar to applying some constraint like 'the sum of vectors can not be above some value'. In the image this is represented by the purple drawing.

The solution is still inside the plane, but also inside the circle. Now the estimate solution is still a representing a shortest distance between the space of solutions and the observation. But the optimal solution is not anymore orthogonal projection onto the red plane. It is instead the shortes distance to the purple circles.

Think in geometrical terms: the OLS fit is the projection of $$Y$$ on the space spanned by the columns of $$X$$, hence the residual vector is orthogonal to that space. If you regularize, performing ridge regression or otherwise, you will in general move your fit away from the projection and destroy orthgonality.

A book which I learned most of this from and is (in my opinion) difficult to surpass is Seber, G.A.F. Linear Regression Analysis, Wiley. I used the ca. 1980 edition, but there is a newer version. Grab a copy if you can.

• Although this is true, it strikes me as a mere re-assertion of the result the OP wants to understand. It would be helpful to explain why this particular way of altering the projection will generally destroy this specific type of orthogonality.
– whuber
Oct 29 '20 at 21:47
• @whuber, sorry I don't have a clue on that. I was just stating that, since the projection is unique, anything which moves the solution away from it will break orthogonality. Oct 30 '20 at 8:24

One way to derive the least squares estimate of $$\beta$$ (the vector of regression coefficients) is that it is the one and only value of $$\beta$$ (*) that would make the error vector orthogonal to every predictor, and hence orthogonal to linear combinations of the predictors, which is what the predicted values $$\hat{y}$$ are. See, for example, section 2 of these course notes.

From that perspective, we can see that any estimate of $$\beta$$ other than the least squares estimate (*) -- for example, any estimate that has been regularized toward 0 to some extent -- will not have this orthogonality property.

(*) Aside from $$\hat{\beta} = 0$$, which leads to $$\hat{y}=0$$, which is always orthogonal to any other vector

• I like your approach but the logic looks faulty: it's possible to devise a method of estimating $\beta$ that differs from OLS but nevertheless guarantees the orthogonality expressed in the question. What you can conclude is that when all the orthogonality conditions are satisfied (which is just saying $\hat\beta$ solves the Normal equations) you have, of course, the OLS estimate.
– whuber
Oct 29 '20 at 21:46
• @whuber Thanks for your comments, they're always appreciated. I guess the key here is in distinguishing the estimation method and the estimate. I made a claim about estimation methods, which was not quite right, because for example MLE of $\beta$ under a normal errors assumption will lead to the same estimate as OLS and therefore have the same orthogonality property. I have edited my post so that the claim is focused on the estimate itself and whether it differs from the least squares estimate, regardless of what estimation method got us there. I think this resolves your concern? Oct 30 '20 at 14:33
• I understand what you are claiming and think the idea is right, but I can't help thinking of counterexamples. For instance, let $\hat\beta$ be the OLS estimate and let $\hat\beta^\prime$ be some other estimate, such as the constant vector $0.$ Use a procedure in which you randomly choose between $\hat\beta$ and $\hat\beta^\prime.$ With probability $1/2,$ then, your result will satisfy the orthogonality conditions. (Interestingly, the zero estimate is always orthogonal too...) This is why your conclusion ".. will not have this orthogonality property" doesn't logically follow.
– whuber
Oct 30 '20 at 14:45
• @whuber Haha, okay, I'll add a parenthetical note about the zero vector Oct 30 '20 at 14:54
• The issue isn't really the zero vector: replace that with anything you like. The point is that in my example, orthogonality still holds half the time. You can make it hold for a definite subset of cases. For instance, use the OLS estimator generally, but if one of your response values equals $\pi$ exactly, double your estimate. In most cases--indeed, perhaps almost surely--orthogonality will hold, even though this is not the OLS estimator.
– whuber
Oct 30 '20 at 16:34