Getting all zero correlations,$\rho_{ij}=\frac{\mathbb cov(e_i,e_j)}{(V(e_i)V(e_j))^{1/2}}$

Question

Consider the general regression model $$Y=X\beta+\epsilon$$ where,

$Y$ is an $(n\times 1)$ vector of observations,

$X$ is an $(n\times p)$ matrix of known form,

$\beta$ is a $(p\times 1)$ vector of parameters,

$\epsilon$ is an $(n\times 1)$ vector of errors.

The fitted values are $$\hat Y=X\hat\beta=HY$$

$H$ denotes hat matrix and $e$ denotes residual.

In the book Applied regression Analysis by Draper/Smith, it is written that :

$\mathbb V(e_i)$ is given by the $i$th diagonal element $(1-h_{ii})$, and $\mathbb cov(e_i,e_j)$ is given by the $(i,j)$th element $(-h_{ij})$ of the matrix $(I-H)\sigma^2$.The correlation between $e_i$ and $e_j$ is given by,$$\rho_{ij}=\frac{\mathbb cov(e_i,e_j)}{(V(e_i)V(e_j))^{1/2}}$$

The values of these correlations thus depend entirely on the elements of the matrix $\ X$,since $\sigma^2$ cancels.In situations where we "design our experiment",that is,choose our $\ X$ matrix ,we thus have the opportunity to affect these correlations.We cannot get all zero correlations, of course,because the $n$ residuals carry only $(n-p)$ degrees of freedom and are linked by the normal equations.

My question is :

$\bullet$ Why have they said We cannot get all zero correlations? Why is the attempt to get all zero correlations, that is, where is the benefit if we get all zero correlations?

$\bullet$When will all correlations be zero theoretically ? [since by the line We cannot get all zero correlations , i understood that we cannot practically make it zero.].

$\bullet$Perhaps my answers of above two questions lies in the line because the $n$ residuals carry only $(n-p)$ degrees of freedom and are linked by the normal equations.But i am not understanding the line.

Answer 1

The normal equations are: $$ (X'X)\hat{\beta} = X'y.$$ As you note, $\hat{y} = X(X'X)^{-1}X'y=H\,y$. The residuals are $\hat{e} = (I-H)y$. One of the interesting properties of $H$ and $I-H$ is that they are idempotent, that is $H^2=H$.

Now, note that the covariance matrix of $\hat{e}$ is $\sigma^2 (I-H)$. As Draper and Smith note, the rank of $I-H$ is $n-p$. If you solve the characteristic equation $$\det \left[(1-\lambda)I+H\right] = 0 $$ you find that there are $n-p$ roots of 1 and $p$ roots of 0. Associated with these characteristic roots are $n$ characteristic vectors, $V = [K|N]$ where $K$ is an $n$ by $n-p$ matrix of ch. vectors corresponding to the unit roots and $N$ is a $n$ by $p$ matrix of ch. vectors corresponding to the $0$ roots. $V$ is an orthogonal matrix, so $VV' = I = KK' + NN'$. The first term is a scalar multiple ($\sigma^{-2}$) of the covariance of $\hat{e}$. Note that if the residuals are uncorrelated, then the nullity ($NN'$) is necessarily also uncorrelated. What is more, if that is the case, then $p$ of the residuals must have 0 variance. This is not the case, so the residuals cannot be uncorrelated.

One can construct sets of uncorrelated residuals, but the sets have size $n-p$. That is, we must sacrifice information about $p$ residuals to make the remaining residuals uncorrelated. The consensus in the literature is that there is very little benefit in residual diagnostic processes and the costs are thus too high. (The construction is based on the characteristic roots and vectors of $I-H$.)