Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Consider a simple regression (normality not assumed): $Y_i = a + b X_i + e_i$ where $e_i$ is with mean 0 and standard deviation $\sigma$. Are the Least Square Estimates of $a$ and $b$ uncorrelated?

share|improve this question
1  
What do you think? en.wikipedia.org/wiki/Ordinary_least_squares, section "Finite sample properties". This question was answered many times on this site. –  mpiktas Jun 25 at 13:27

1 Answer 1

up vote 5 down vote accepted

This is an important consideration in designing experiments, where it can be desirable to have no (or very little) correlation among the estimates $\hat a$ and $\hat b$. Such lack of correlation can be achieved by controlling the values of the $X_i$.


To analyze the effects of the $X_i$ on the estimates, the values $(1,X_i)$ (which are row vectors of length $2$) are assembled vertically into a matrix $X$, the design matrix, having as many rows as there are data and (obviously) two columns. The corresponding $Y_i$ are assembled into one long (column) vector $y$. In these terms, writing $\beta = (a,b)^\prime$ for the assembled coefficients, the model is

$$\mathbb{E}(Y) = X \cdot \beta$$

The $Y_i$ are (usually) assumed to be independent random variables whose variances are a constant $\sigma^2$ for some unknown $\sigma \gt 0$. The dependent observations $y$ are taken to be one realization of the vector-valued random variable $Y$.

The OLS solution is

$$\hat\beta = \left(X^\prime X\right)^{-1} X^\prime y,$$

assuming this matrix inverse exists. Thus, using basic properties of matrix multiplication and covariance,

$$\text{Cov}(\hat\beta) = \text{Cov}\left(\left(X^\prime X\right)^{-1} X^\prime Y\right) = \left(\left(X^\prime X\right)^{-1} X^\prime\sigma^2 X \left( X^\prime X \right)^{-1\prime} \right) = \sigma^2 \left(X^\prime X\right)^{-1}. $$

The matrix $\left(X^\prime X\right)^{-1}$ has just two rows and two columns, each corresponding to the model parameters $(a,b)$. The correlation of $\hat a$ with $\hat b$ is proportional to the off-diagonal elements, which are just the dot products of the two columns of $X$. Since one of the columns is all $1$s, whose dot product with the other column (containing the of $X_i$) is their sum, we find

$\hat a$ and $\hat b$ are uncorrelated if and only the sum (or equivalently the mean) of the $X_i$ is zero.

This condition frequently is achieved by recentering the $X_i$ (by subtracting their mean from each). Although this will not alter the estimated slope $\hat b$, it does change the estimated intercept $\hat a$. Whether or not that is important depends on the application.


All this analysis applies to multiple regression: the design matrix will have $p+1$ columns for $p$ independent variables (remember, an additional column consists of $1$s) and $\beta$ will be a vector of length $p+1$, but otherwise everything goes through exactly as before. In conventional language, two columns of $X$ are called orthogonal when their dot product is zero. Thus,

Two multiple regression coefficient estimates $\hat\beta_i$ and $\hat\beta_j$ are uncorrelated if and only if the corresponding columns of the design matrix are orthogonal.

Many standard experimental designs consist of choosing values of the independent variables to make the columns orthogonal. This "separates" the resulting estimates by guaranteeing--before any data are ever collected!--that the estimates will be uncorrelated. (When the responses have Normal distributions this implies the estimates will be independent, which greatly simplifies their interpretation.)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.