14
$\begingroup$

Can someone briefly explain for me, why each of the six assumptions is needed in order to compute the OLS estimator? I found only about multicollinearity—that if it exists we cannot invert (X'X) matrix and in turn estimate overall estimator. What about the others (e.g., linearity, zero mean errors, etc.)?

$\endgroup$
  • 5
    $\begingroup$ Related: What is a complete list of the usual assumptions for linear regression? $\endgroup$ – gung Apr 30 '15 at 18:08
  • 1
    $\begingroup$ Are you looking for a conceptual explanation, or do you need a mathematical demonstration? $\endgroup$ – gung Apr 30 '15 at 18:19
  • 4
    $\begingroup$ Ordinary least squares is a numerical procedure, you do not need much assumption to compute it (apart from invertibility). The assumptions are needed to justify inference based on it, see my answer yesterday: stats.stackexchange.com/questions/148803/… $\endgroup$ – kjetil b halvorsen Apr 30 '15 at 20:00
  • 1
    $\begingroup$ Exactly which "six assumptions" are you referring to? You mention only three. $\endgroup$ – whuber Apr 30 '15 at 20:05
  • $\begingroup$ I refer to 1) linearity 2) absence of multicollinearity 3) zero mean errors 4) spherical errors (homoscedasticity and non autocorrelation) 5) non-stochastic regressors and 6)normal distribution. So as I understood from the answer below, only first three are necessary in order to derive estimator and other are only needed to make sure the estimator is BLUE ? $\endgroup$ – Ieva Apr 30 '15 at 21:14
23
$\begingroup$

You can always compute the OLS estimator, apart from the case when you have perfect multicollinearity. In this case, you do have perfect multilinear dependence in your X matrix. Consequently, the full rank assumption is not fulfilled and you cannot compute the OLS estimator, because of invertibility issues.

Technically, you do not need the other OLS assumptions to compute the OLS estimator. However, according to the Gauss–Markov theorem you need to fulfill the OLS assumption (clrm assumptions) in order for your estimator to be BLUE.

You can find an extensive discussion of the Gauss–Markov theorem and its mathematical derivation here:

http://economictheoryblog.com/2015/02/26/markov_theorem/

Furthermore, if you are looking for an overview of the OLS assumption, i.e. how many there are, what they require and what happens if you violate the single OLS assumption may find an elaborate discussion here:

http://economictheoryblog.com/2015/04/01/ols_assumptions/

I hope that helps, cheers!

$\endgroup$
14
$\begingroup$

The following is based on simple cross sections, for time series and panels it is somewhat different.

  1. In the population, and therefore in the sample, the model can be written as: \begin{align} \newcommand{\Var}{\rm Var} \newcommand{\Cov}{\rm Cov} Y &= \beta_0 + \beta_1 x_1 + … + \beta_k x_k + u \\ &= X\beta + u \end{align} This is the linearity assumption, which is sometimes misunderstood. The model should be linear in the parameters - namely the $\beta_k$. You are free to do whatever you want with the $x_i$ themselves. Logs, squares etc. If this is not the case, then the model cannot be estimated by OLS - you need some other nonlinear estimator.
  2. A random sample (for cross sections) This is needed for inference, and sample properties. It is somewhat irrelevant for the pure mechanics of OLS.
  3. No perfect Collinearity This means that there can be no perfect relationship between the $x_i$. This is the assumption that ensures that $(X’X)$ is nonsingular, such that $(X’X)^{-1}$ exists.
  4. Zero conditional mean: $E(u|X) = 0$. This means that you have properly specified the model such that: there are no omitted variables, and the functional form you estimated is correct relative to the (unknown) population model. This is always the problematic assumption with OLS, since there is no way to ever know if it is actually valid or not.
  5. The variance of the errors term is constant, conditional on the all $X_i$: $\Var(u|X)=\sigma^2$ Again this means nothing for the mechanics of OLS, but it ensure that the usual standard errors are valid.
  6. Normality; the errors term u is independent of the $X_i$, and follows $u \sim N(0,\sigma^2)$. Again this is irrelevant for the mechanics of OLS, but ensures that the sampling distribution of the $\beta_k$ is normal, $\hat{\beta_k} \sim N(\beta_k , \Var(\hat{\beta_k}))$.

Now for the implications.

  1. Under 1 - 6 (the classical linear model assumptions) OLS is BLUE (best linear unbiased estimator), best in the sense of lowest variance. It is also efficient amongst all linear estimators, as well as all estimators that uses some function of the x. More importantly under 1 - 6, OLS is also the minimum variance unbiased estimator. That means that amongst all unbiased estimators (not just the linear) OLS has the smallest variance. OLS is also consistent.

  2. Under 1 - 5 (the Gauss-Markov assumptions) OLS is BLUE and efficient (as described above).

  3. Under 1 - 4, OLS is unbiased, and consistent.

Actually OLS is also consistent, under a weaker assumption than $(4)$ namely that: $(1)\ E(u) = 0$ and $(2)\ \Cov(x_j , u) = 0$. The difference from assumptions 4 is that, under this assumption, you do not need to nail the functional relationship perfectly.

$\endgroup$
  • $\begingroup$ I think you paint too dark a picture about the zero mean condition. If there were a bias, then minimizing the sum of the squared deviations would not be the appropriate thing to do, but on the other hand, you can capture the bias by shifting the regression equation (absorbing the bias into $\beta_0$), and then you do have mean 0. In other words, 4 is both impossible to verify and easy to ignore. $\endgroup$ – user3697176 Apr 30 '15 at 21:07
  • $\begingroup$ I am sorry, but I do not agree. Or maybe I just misunderstanding you? Could you either eloborate or give a reference. $\endgroup$ – Repmat May 1 '15 at 14:56
  • $\begingroup$ I am not talking about intentionally distorted estimation (such as ridge regression), which I believe the OP was not interested in. I am talking about a model of the form $y=\beta_0 +\beta_1x_1+\ldots+\beta_x x_n + \epsilon$ in which --- for some strange reason --- the residual $\epsilon$ has mean $\alpha\ne0$. In this case it is easy to do a formal transformation to $y=\alpha+\beta_0 +\beta_1x_1+\ldots+\beta_x x_n + \eta$, where the mean of $\eta$ is zero. $\endgroup$ – user3697176 May 3 '15 at 2:28
  • $\begingroup$ @user3697176 What you write is not correct. I just posted an answer to explain why. $\endgroup$ – Alecos Papadopoulos May 3 '15 at 15:38
  • $\begingroup$ If assumption 1 isn't satisfied, can't we still use OLS to estimate the population covariance (even though we know there's no linear relationship)? $\endgroup$ – max May 1 '16 at 0:23
7
$\begingroup$

A comment in another question raised doubts about the importance of the condition $E(\mathbf u \mid \mathbf X) =0$, arguing that it can be corrected by the inclusion of a constant term in the regression specification, and so "it can be easily ignored".

This is not so. The inclusion of a constant term in the regression will absorb the possibly non-zero conditional mean of the error term if we assume that this conditional mean is already a constant and not a function of the regressors. This is the crucial assumption that must be made independently of whether we include a constant term or not:

$$E(\mathbf u \mid \mathbf X) =const.$$

If this holds, then the non-zero mean becomes a nuisance which we can simply solve by including a constant term.

But if this doesn't hold, (i.e. if the conditional mean is not a zero or a non-zero constant), the inclusion of the constant term does not solve the problem: what it will "absorb" in this case is a magnitude that depends on the specific sample and realizations of the regressors. In reality the unknown coefficient attached to the series of ones, is not really a constant but variable, depending on the regressors through the non-constant conditional mean of the error term.

What does this imply? To simplify, assume the simplest case, where $E(u_i \mid \mathbf X_{-i})=0$ ($i$ indexes the observations) but that $E(u_i \mid \mathbf x_{i})=h(\mathbf x_i)$. I.e. that the error term is mean-independent from the regressors except from its contemporaneous ones (in $\mathbf X$ we do not include a series of ones).

Assume that we specify the regression with the inclusion of a constant term (a regressor of a series of ones).

$$\mathbf y = \mathbf a + \mathbf X\mathbf β + \mathbf ε $$

and compacting notation

$$\mathbf y = \mathbf Z\mathbf γ + \mathbf ε $$

where $\mathbf a = (a,a,a...)'$, $\mathbf Z = [\mathbf 1: \mathbf X]$, $\mathbf γ = (a, \mathbf β)'$, $\mathbf ε = \mathbf u - \mathbf a$.

Then the OLS estimator will be

$$\hat {\mathbf γ} = \mathbf γ + \left(\mathbf Z'\mathbf Z\right)^{-1}\mathbf Z'\mathbf ε$$

For unbiasedness we need $E\left[\mathbf ε\mid \mathbf Z\right] =0$. But

$$E\left[ ε_i\mid \mathbf x_i\right] = E\left[u_i-a\mid \mathbf x_i\right] = h(\mathbf x_i)-a$$

which cannot be zero for all $i$, since we examine the case where $h(\mathbf x_i)$ is not a constant function. So

$$E\left[\mathbf ε\mid \mathbf Z\right] \neq 0 \implies E(\hat {\mathbf γ}) \neq \mathbf γ$$

and

If $E(u_i \mid \mathbf x_{i})=h(\mathbf x_i)\neq h(\mathbf x_j)=E(u_j \mid \mathbf x_{j})$, then even if we include a constant term in the regression, the OLS estimator will not be unbiased, meaning also that the Gauss-Markov result on efficiency, is lost.

Moreover, the error term $\mathbf ε$ has a different mean for each $i$, and so also a different variance (i.e. it is conditionally heteroskedastic). So its distribution conditional on the regressors differs across the observations $i$.

But this means that even if the error term $u_i$ is assumed normal, then the distribution of the sampling error $\hat {\mathbf γ} - \mathbf γ$ will be normal but not zero-mean mormal, and with unknown bias. And the variance will differ. So

If $E(u_i \mid \mathbf x_{i})=h(\mathbf x_i)\neq h(\mathbf x_j)=E(u_j \mid \mathbf x_{j})$, then even if we include a constant term in the regression, Hypothesis testing is no longer valid.

In other words, "finite-sample" properties are all gone.

We are left only with the option to resort to asymptotically valid inference, for which we will have to make additional assumptions.

So simply put, Strict Exogeneity cannot be "easily ignored".

$\endgroup$
  • $\begingroup$ I'm not completely sure I understand this. Isn't assuming that the mean is a not a function of the regressors equivalent to assuming homoscedasticity? $\endgroup$ – Batman May 16 '16 at 4:01
  • $\begingroup$ @Batman To what part of my post are you referring to? $\endgroup$ – Alecos Papadopoulos May 16 '16 at 9:14
  • $\begingroup$ When you say "The inclusion of a constant term in the regression will absorb the possibly non-zero conditional mean of the error term if we assume that this conditional mean is already a constant and not a function of the regressors. This is the crucial assumption that must be made independently of whether we include a constant term or not." Isn't assuming that the conditional mean isn't a function of the regressors exactly what we're assuming when we assume homoscedasticity? $\endgroup$ – Batman May 16 '16 at 12:21
  • $\begingroup$ @Batman Homoskedasticity is an assumption about the variance. Assuming mean -independence does not imply that $E(u^2_j \mid \mathbf x)$ is also a constant, which is also needed for conditional homoskedasticity. In fact, mean-independence, $E(u \mid x) =const. $ together with conditional heteroskedasticity, $E(u^2 \mid x) = g(x)$ is a standard model variant. $\endgroup$ – Alecos Papadopoulos Dec 13 '16 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.