Are estimates of regression coefficients uncorrelated?

Question

Consider a simple regression (normality not assumed): $$Y_i = a + b X_i + e_i,$$ where $e_i$ is with mean $0$ and standard deviation $\sigma$. Are the Least Square Estimates of $a$ and $b$ uncorrelated?

Answer 1

This is an important consideration in designing experiments, where it can be desirable to have no (or very little) correlation among the estimates $\hat a$ and $\hat b$. Such lack of correlation can be achieved by controlling the values of the $X_i$.

---

To analyze the effects of the $X_i$ on the estimates, the values $(1,X_i)$ (which are row vectors of length $2$) are assembled vertically into a matrix $X$, the design matrix, having as many rows as there are data and (obviously) two columns. The corresponding $Y_i$ are assembled into one long (column) vector $y$. In these terms, writing $\beta = (a,b)^\prime$ for the assembled coefficients, the model is

$$\mathbb{E}(Y) = X \cdot \beta$$

The $Y_i$ are (usually) assumed to be independent random variables whose variances are a constant $\sigma^2$ for some unknown $\sigma \gt 0$. The dependent observations $y$ are taken to be one realization of the vector-valued random variable $Y$.

The OLS solution is

$$\hat\beta = \left(X^\prime X\right)^{-1} X^\prime y,$$

assuming this matrix inverse exists. Thus, using basic properties of matrix multiplication and covariance,

$$\text{Cov}(\hat\beta) = \text{Cov}\left(\left(X^\prime X\right)^{-1} X^\prime Y\right) = \left(\left(X^\prime X\right)^{-1} X^\prime\sigma^2 X \left( X^\prime X \right)^{-1\prime} \right) = \sigma^2 \left(X^\prime X\right)^{-1}. $$

The matrix $\left(X^\prime X\right)^{-1}$ has just two rows and two columns, corresponding to the model parameters $(a,b)$. The correlation of $\hat a$ with $\hat b$ is proportional to the off-diagonal elements of $(X^\prime X)^{-1},$ which by Cramer's Rule are proportional to the dot product of the two columns of $X$. Since one of the columns is all $1$s, whose dot product with the other column (consisting of the $X_i$) is their sum, we find

$\hat a$ and $\hat b$ are uncorrelated if and only the sum (or equivalently the mean) of the $X_i$ is zero.

This orthogonality condition frequently is achieved by recentering the $X_i$ (by subtracting their mean from each). Although this will not alter the estimated slope $\hat b$, it does change the estimated intercept $\hat a$. Whether or not that is important depends on the application.

---

Tthis analysis applies to multiple regression: the design matrix will have $p+1$ columns for $p$ independent variables (an additional column consists of $1$s) and $\beta$ will be a vector of length $p+1$, but otherwise everything goes through as before.

In conventional language, two columns of $X$ are called orthogonal when their dot product is zero. When one column of $X$ (say column $i$) is orthogonal to all the other columns, it is an easily demonstrated algebraic fact that all off-diagonal entries in row $i$ and column $i$ of $(X^\prime X)^{-1}$ are zero (that is, the $ij$ and $ji$ components for all $j\ne i$ are zero). Consequently,

Two multiple regression coefficient estimates $\hat\beta_i$ and $\hat\beta_j$ are uncorrelated whenever either (or both) of the corresponding columns of the design matrix are orthogonal to all other columns.

Many standard experimental designs consist of choosing values of the independent variables to make the columns mutually orthogonal. This "separates" the resulting estimates by guaranteeing--before any data are ever collected!--that the estimates will be uncorrelated. (When the responses have Normal distributions this implies the estimates will be independent, which greatly simplifies their interpretation.)