Is it an assumption of the normal linear model that explanatory variables are uncorrelated with the errors?

Question

Some books seem to include an assumption for the normal linear model which I have never seen before. They say that there must be no correlation between between the explanatory variables and the errors. I was wondering if this assumption is true and if so, where it comes from.

Answer 1

I wouldn't quite call this an assumption of the linear model. Instead, I would say that this is an assumption you are making when you interpret the results of a linear model in a particular way. In other words, when the stated condition holds, there may be multiple possible interpretations of which some are legitimate and some are not.

The assumption, "no correlation between between the explanatory variables and the errors" refers to the lack (or existence) of endogeneity. Standard methods for estimating a linear model from data (e.g., ordinary least squares and maximum likelihood estimation) force the errors to have mean $0$. This affects the coefficient estimates that result.

Consider a case where the level of a response variable, $Y$, is a function of three variables, $X_1,\ X_2,\ \& \ X_3$. Further, imagine that $X_2$ and $X_3$ are correlated, but you only include $X_1$ and $X_2$ in your model. (Perhaps you've never even heard of $X_3$ and no one has ever thought to use it to understand why certain values of $Y$ seem to occur in the world.) When a variable is not included in a model, its effects are collapsed into the error term. Thus, you now have a variable, $X_2$, that is correlated with the error term, in violation of the 'assumption' stated above.

So what is the result of this? Part of the effect of $X_3$ gets mixed in with the effect of $X_2$ in the model's estimate of the coefficient for $X_2$. That is, the violation of this assumption leads to coefficient estimates that are biased when considered to be estimates of the direct effects of the coefficients on the response. However, these are unbiased estimates of the marginal association between your variables and the response. To understand this more fully, it may help you to read my answer here: Estimating $b_1x_1+b_2x_2$ instead of $b_1x_1+b_2x_2+b_3x_3$.

Answer 2

I find the phrase "no correlation" potentially misleading, because I have noticed that sometimes the word correlation is used narrowly, to refer to the covariance between two random variables and whether it is zero or not, and sometimes it is used more broadly, to indicate whether there exists (or not) some unspecified stochastic dependence between the two.

For the normal linear regression model, different conditions guarantee different properties of the estimator, deemed desirable ones. The model is

$$y_i = \mathbf x'\beta + u_i, \;\; i=1,...,n$$ and also in matrix notation (we will need them both)

$$\mathbf y = \mathbf X\beta + \mathbf u$$

The Ordinary Least-Squares estimator (which is also maximum likelihod under normality) is, in matrix notation,

$$\hat \beta = \left(\mathbf X' \mathbf X\right)^{-1}\mathbf X' \mathbf y = \beta + \left(\mathbf X' \mathbf X\right)^{-1}\mathbf X' \mathbf u$$

To examine Unbiasedness we have

$$E(\hat \beta) = \beta + E\left[\left(\mathbf X' \mathbf X\right)^{-1}\mathbf X' \mathbf u\right]$$ and using the law of Iterated Expectations

$$E(\hat \beta) -\beta= E\left(E\left[\left(\mathbf X' \mathbf X\right)^{-1}\mathbf X' \mathbf u\mid\mathbf X\right] \right)$$

$$= E\left(\left(\mathbf X' \mathbf X\right)^{-1}\mathbf X'E\left[ \mathbf u\mid\mathbf X\right] \right)$$

So in order for the estimator to be unbiased we require $E\left[ \mathbf u\mid\mathbf X\right]=0$. Note that the condition requires that the conditional expected value of each $u_i$ conditional on all $\mathbf X$ (i.e. on all the random variables forming the sample, and not only those belonging to the $i$-th observation), is zero. This is not usually called "being uncorrelated", but rather "error is mean-independent of the regressors" or "regressors are strictly exogenous to the error term". This specific property is needed for unbiasedness and consequently for the Gauss-Markov theorem to hold.

The property required for Consistency comes closer to being described verbally as "no correlation". For Consistency we require that

$${\rm plim} (\hat \beta-\beta) = {\rm plim}\left(\frac 1n\mathbf X' \mathbf X\right)^{-1}\cdot {\rm plim}\left(\frac 1n\mathbf X' \mathbf u\right) =0$$

Focusing on what interests us (and not on all regularity, etc conditions that must hold here) what we need is

$$\left(\frac 1n\mathbf X' \mathbf u\right) \xrightarrow{p} 0$$

If we write out explicitly the matrix product we will obtain a vector of $1$ column, with typical element

$$\frac 1n\sum_{i=1}^nX_{ki}u_i$$

where $X_k$ is the $k$-th regressor. As $n$ tends to infinity, and under some conditions the Law of Large Numbers will hold and we will have that

$$\frac 1n\sum_{i=1}^nX_{ki}u_i \rightarrow \frac 1n\sum_{i=1}^nE\big(X_{ki}u_i\big)$$

So for Consistency we require the (weaker) condition that $E\big(X_{ki}u_i\big) = 0, \forall k$, and for all $i$ separately. So we require that each regressor is contemporaneously orthogonal to the error term. The condition is weaker than the one for unbiasedness, because it does not require that, say, $E(X_{kj}u_i) =0$. In other words consistency may survive with a non-i.i.d sample. Also, $E\big(X_{ki}u_i\big) = 0$ does not imply the Strict Exogeneity condition.

Since moreover the error term is assumed zero-mean, "orthogonality" becomes equivalent to "no-correlation" (zero covariance), and this is why "no-correlation" comes closer to describing this condition (and so guarantees consistency) rather than the Strict Exogeneity condition that is required for unbiasedness.