I'm reading this PDF.

It shows how to obtain the OLS estimator and its properties.

It is said that from the normal equations we obtain $X' e = 0$.

Where $X$ is the design matrix and $e$ is the vector of residuals.

Then, at page 4, it claims the following property: "The observed values of X are uncorrelated with the residuals."

Proof: Indicating the regressors by $x_k$, $X' e = 0$ implies $x'_k e = 0$.

So far so good.

Then it goes on to say "In other words, each regressor has zero sample correlation with the residuals."

I didn't understand that.

By definition of sample correlation:

$$r_{x_pe} = \frac{\sum_{i=1}^n x_{pi} e_i - n \bar{x_p} \bar{e}}{n s^{'}_{x_p} s^{'}_e}$$

We have proven that $\sum_{i=1}^n x_{pi} e_i = 0$. But then there is another term at the numerator.

Except if the mean of the residuals equal 0. That is, if $\bar{e}=0$.

But that would require the constant, as shown in point 3 of page 4.


1 Answer 1


You are right.

Maybe because most regressions do contain a constant, the property $X'e=0$ (often called, more precisely, "orthogonality") and the terminology "uncorrelatedness" are often used interchangeably, when they do amount to the same thing only if the regression contains a constant (or, more precisely, if the residuals have mean zero, which can also be the case if the regressors can be linearly combined into a constant, say with an exhaustive set of dummies).

A little numerical illustration:

n <- 10
y <- rnorm(n)
x <- rnorm(n)

regwcst <- lm(y~x)
regwocst <- lm(y~x-1)
d1 <- c(rep(1,5), rep(0,5)) # two exhaustive dummies
d2 <- 1-d1
regwdumm <- lm(y~x-1+d1+d2)

> crossprod(x, resid(regwcst))  # all numerically zero
[1,] -2.081668e-17

> crossprod(x, resid(regwdumm))
[1,] -1.249001e-16

> crossprod(x, resid(regwocst))
[1,] 1.804112e-16

> cor(x, resid(regwcst))        # numerically zero
[1] -2.721791e-17

> cor(x, resid(regwocst))       # not numerically zero
[1] 0.01718539

> cor(x, resid(regwdumm))       # numerically zero

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.