Assumptions to derive OLS estimator

Question

Can someone briefly explain for me, why each of the six assumptions is needed in order to compute the OLS estimator? I found only about multicollinearity—that if it exists we cannot invert (X'X) matrix and in turn estimate overall estimator. What about the others (e.g., linearity, zero mean errors, etc.)?

Answer 1

You can always compute the OLS estimator, apart from the case when you have perfect multicollinearity. In this case, you do have perfect multilinear dependence in your X matrix. Consequently, the full rank assumption is not fulfilled and you cannot compute the OLS estimator, because of invertibility issues.

Technically, you do not need the other OLS assumptions to compute the OLS estimator. However, according to the Gauss–Markov theorem you need to fulfill the OLS assumption (clrm assumptions) in order for your estimator to be BLUE.

You can find an extensive discussion of the Gauss–Markov theorem and its mathematical derivation here:

http://economictheoryblog.com/2015/02/26/markov_theorem/

Furthermore, if you are looking for an overview of the OLS assumption, i.e. how many there are, what they require and what happens if you violate the single OLS assumption may find an elaborate discussion here:

http://economictheoryblog.com/2015/04/01/ols_assumptions/

I hope that helps, cheers!

Answer 2

The following is based on simple cross sections, for time series and panels it is somewhat different.

1. In the population, and therefore in the sample, the model can be written as: \begin{align} \newcommand{\Var}{\rm Var} \newcommand{\Cov}{\rm Cov} Y &= \beta_0 + \beta_1 x_1 + … + \beta_k x_k + u \\ &= X\beta + u \end{align} This is the linearity assumption, which is sometimes misunderstood. The model should be linear in the parameters - namely the $\beta_k$. You are free to do whatever you want with the $x_i$ themselves. Logs, squares etc. If this is not the case, then the model cannot be estimated by OLS - you need some other nonlinear estimator.
2. A random sample (for cross sections) This is needed for inference, and sample properties. It is somewhat irrelevant for the pure mechanics of OLS.
3. No perfect Collinearity This means that there can be no perfect relationship between the $x_i$. This is the assumption that ensures that $(X’X)$ is nonsingular, such that $(X’X)^{-1}$ exists.
4. Zero conditional mean: $E(u|X) = 0$. This means that you have properly specified the model such that: there are no omitted variables, and the functional form you estimated is correct relative to the (unknown) population model. This is always the problematic assumption with OLS, since there is no way to ever know if it is actually valid or not.
5. The variance of the errors term is constant, conditional on the all $X_i$: $\Var(u|X)=\sigma^2$ Again this means nothing for the mechanics of OLS, but it ensure that the usual standard errors are valid.
6. Normality; the errors term u is independent of the $X_i$, and follows $u \sim N(0,\sigma^2)$. Again this is irrelevant for the mechanics of OLS, but ensures that the sampling distribution of the $\beta_k$ is normal, $\hat{\beta_k} \sim N(\beta_k , \Var(\hat{\beta_k}))$.

Now for the implications.

1. Under 1 - 6 (the classical linear model assumptions) OLS is BLUE (best linear unbiased estimator), best in the sense of lowest variance. It is also efficient amongst all linear estimators, as well as all estimators that uses some function of the x. More importantly under 1 - 6, OLS is also the minimum variance unbiased estimator. That means that amongst all unbiased estimators (not just the linear) OLS has the smallest variance. OLS is also consistent.

2. Under 1 - 5 (the Gauss-Markov assumptions) OLS is BLUE and efficient (as described above).

3. Under 1 - 4, OLS is unbiased, and consistent.

Actually OLS is also consistent, under a weaker assumption than $(4)$ namely that: $(1)\ E(u) = 0$ and $(2)\ \Cov(x_j , u) = 0$. The difference from assumptions 4 is that, under this assumption, you do not need to nail the functional relationship perfectly.

Answer 3

A comment in another question raised doubts about the importance of the condition $E(\mathbf u \mid \mathbf X) =0$, arguing that it can be corrected by the inclusion of a constant term in the regression specification, and so "it can be easily ignored".

This is not so. The inclusion of a constant term in the regression will absorb the possibly non-zero conditional mean of the error term if we assume that this conditional mean is already a constant and not a function of the regressors. This is the crucial assumption that must be made independently of whether we include a constant term or not:

$$E(\mathbf u \mid \mathbf X) =const.$$

If this holds, then the non-zero mean becomes a nuisance which we can simply solve by including a constant term.

But if this doesn't hold, (i.e. if the conditional mean is not a zero or a non-zero constant), the inclusion of the constant term does not solve the problem: what it will "absorb" in this case is a magnitude that depends on the specific sample and realizations of the regressors. In reality the unknown coefficient attached to the series of ones, is not really a constant but variable, depending on the regressors through the non-constant conditional mean of the error term.

What does this imply? To simplify, assume the simplest case, where $E(u_i \mid \mathbf X_{-i})=0$ ($i$ indexes the observations) but that $E(u_i \mid \mathbf x_{i})=h(\mathbf x_i)$. I.e. that the error term is mean-independent from the regressors except from its contemporaneous ones (in $\mathbf X$ we do not include a series of ones).

Assume that we specify the regression with the inclusion of a constant term (a regressor of a series of ones).

$$\mathbf y = \mathbf a + \mathbf X\mathbf β + \mathbf ε $$

and compacting notation

$$\mathbf y = \mathbf Z\mathbf γ + \mathbf ε $$

where $\mathbf a = (a,a,a...)'$, $\mathbf Z = [\mathbf 1: \mathbf X]$, $\mathbf γ = (a, \mathbf β)'$, $\mathbf ε = \mathbf u - \mathbf a$.

Then the OLS estimator will be

$$\hat {\mathbf γ} = \mathbf γ + \left(\mathbf Z'\mathbf Z\right)^{-1}\mathbf Z'\mathbf ε$$

For unbiasedness we need $E\left[\mathbf ε\mid \mathbf Z\right] =0$. But

$$E\left[ ε_i\mid \mathbf x_i\right] = E\left[u_i-a\mid \mathbf x_i\right] = h(\mathbf x_i)-a$$

which cannot be zero for all $i$, since we examine the case where $h(\mathbf x_i)$ is not a constant function. So

$$E\left[\mathbf ε\mid \mathbf Z\right] \neq 0 \implies E(\hat {\mathbf γ}) \neq \mathbf γ$$

and

If $E(u_i \mid \mathbf x_{i})=h(\mathbf x_i)\neq h(\mathbf x_j)=E(u_j \mid \mathbf x_{j})$, then even if we include a constant term in the regression, the OLS estimator will not be unbiased, meaning also that the Gauss-Markov result on efficiency, is lost.

Moreover, the error term $\mathbf ε$ has a different mean for each $i$, and so also a different variance (i.e. it is conditionally heteroskedastic). So its distribution conditional on the regressors differs across the observations $i$.

But this means that even if the error term $u_i$ is assumed normal, then the distribution of the sampling error $\hat {\mathbf γ} - \mathbf γ$ will be normal but not zero-mean mormal, and with unknown bias. And the variance will differ. So

If $E(u_i \mid \mathbf x_{i})=h(\mathbf x_i)\neq h(\mathbf x_j)=E(u_j \mid \mathbf x_{j})$, then even if we include a constant term in the regression, Hypothesis testing is no longer valid.

In other words, "finite-sample" properties are all gone.

We are left only with the option to resort to asymptotically valid inference, for which we will have to make additional assumptions.

So simply put, Strict Exogeneity cannot be "easily ignored".