0
$\begingroup$

In OLS, we have the following

$$ X^Tr = 0 $$ where $X \in \mathbb{R}^{n \times p}$ data matrix and $r \in \mathbb{R}^{n}$ is the vector of residuals.

The above equal shows that the column of samples corresponding to the j-th independent variable, where $j \in {1, \ldots, p}$$, is orthogonal to the residual vector.

But does this necessarily imply that the 2 are uncorrelated. Covariance is essentially a centered inner product, but in this case $X$ is not necessarily centered. If it is was centered, then the above result can be used to conclude uncorrelatedness, but if it is not known whether $X$ is centered, then can we still make the assertion of uncorrelatedness?

$\endgroup$
3
  • $\begingroup$ I don't think $X^Tr=0$ in itself implies they're uncorrelated. But since you also know that the mean of the residuals is zero, you can show that the covariance is zero. \begin{align} cov(X_i, r) = E[(X_i - \mu_{X_i})(r)] \\ = E[X_ir] - \mu_{X_i}E[r] \\ = \frac{1}{n}\langle X_i, r\rangle - \mu_{X_i}*0 = 0\\ \end{align} $\endgroup$
    – 24n8
    Commented Jul 16, 2020 at 5:52
  • $\begingroup$ This derivation chooses a single variable, but there's no loss of generality. Also note that $r$ lies in the nullspace of $X^T$ $\endgroup$
    – 24n8
    Commented Jul 16, 2020 at 5:54
  • 1
    $\begingroup$ @lamanon why don't you write it as an answer? $\endgroup$
    – gunes
    Commented Jul 16, 2020 at 12:08

1 Answer 1

0
$\begingroup$

I don't think $X^Tr = 0$ itself is sufficient to show that all the regressors are uncorrelated with the residuals. However, for OLS with an intercept, you know that

$$ \sum_i^n r_i = 0 \\ \therefore E[r_i] = 0 \\ $$

You can then use this information to derive that the covariance is zero between any regressor and the residual, hence correlation is zero, as follows: \begin{align} \text{cov}(X_j, r) = E[X_jr] - E[X_j]E[r] && \forall X_j \in \{X_1, \ldots, X_p\}\\ =\frac{1}{n}\langle X_j, r \rangle - E[X_j]*0 \\ = 0 - 0 = 0 \end{align}

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.