Many econometrics textbooks (e.g. Wooldridge, "Econometric analysis...") simply write something similar to: "If the population model is $y = xB + u$ and (1) $\text{Cov}(X,U) = 0$; (2) $X'X$ is full rank, then OLS consistently estimate parameters $B$". I've worked out the math behind consistency but I'm a bit lost in interpreting the meaning.

We're talking about consistent estimation, but estimation of what? Please tell me if following points are correct:

(1) If we have random sample $X,Y$ and $X'X$ is invertible, then we can always define Best Linear Predictor of $y$ given $x$. And then OLS always consistently estimate coefficients of Best Linear Predictor (because in BLP we have $\text{Cov}(u,x)=0$ from definition). Bottom line: we can always interpret OLS estimates as coefficients of BLP

(2) The only question is whether BLP corresponds to conditional expectation $\text{E}(y|x)$. If it does (for which we need $\text{E}(u|x) = 0$), then we can interpret OLS estimates as partial effects.

What is wrong with this or what am I missing? I don't get the point of stating the assumption of $\text{Cov}(u,x) = 0$, if this assumption is by definition fulfilled in case of Best Linear Predictor. And if we want to give structural interpretation to BLP coefficients (partial effects) we need stronger assumption of $\text{E}(u|x) = 0$ anyway. Is there any other interpretation we may have with zero covariance?


An estimator is consistent if $\hat{\beta} \rightarrow_{p} \beta$

Or $\lim_{n \rightarrow \infty} \mbox{Pr}(|\hat{\beta} - \beta| < \epsilon) = 1 $ for all positive real $\epsilon$.

Consistency in the literal sense means that sampling the world will get us what we want. There are inconsistent minimum variance estimators (failing to find the famous example by Google at this point).

Unbiased minimum variance is a good starting place for thinking about estimators. Sometimes, it's easier to understand that we may have other criteria for "best" estimators. There is the general class of minimax estimators, and there are estimators that minimize MSE instead of variance (a little bit of bias in exchange for a whole lot less variance can be good). These estimators can be consistent because they asymptotically converge to the population estimates.

The interpretation of the slope parameter comes from the context of the data you've collected. For instance, if $Y$ is fasting blood gluclose and $X$ is the previous week's caloric intake, then the interpretation of $\beta$ in the linear model $E[Y|X] = \alpha + \beta X$ is an associated difference in fasting blood glucose comparing individuals differing by 1 kCal in weekly diet (it may make sense to standardize $X$ by a denominator of $2,000$.

That is what you consistently estimate with OLS, the more that $n$ increases.

WRT #2 Linear regression is a projection. The predictors we obtain from projecting the observed responses into the fitted space necessarily generates it's additive orthogonal error component. These errors are always 0 mean and independent of the fitted values in the sample data (their dot product sums to zero always). This holds regardless of homoscedasticity, normality, linearity, or any of the classical assumptions of regression models.

  • 1
    $\begingroup$ Thank you for your answer. I understand the interpretation of $\beta$ if you define the model as $E(Y|X)$. However in such a case we could only interpret $\beta$ as a influence of number of kCals in weekly diet on in fasting blood glucose if we were willing to assume that $\alpha + \beta X$ is the true model for $E(Y|X)$ (and this assumption means that $E(u|X) = 0$. If the true model was $\alpha + \beta_1 X + \beta_2 X^2$ we could not give such a interpretation. Therefore I don't see what interpretation you can give to coefficient if you only assume zero covariance and not mean independence. $\endgroup$ – Michal Jun 13 '13 at 19:16
  • $\begingroup$ "we could only interpret β as a influence of number of kCals in weekly diet on in fasting blood glucose if we were willing to assume that α+βX is the true model": Not at all! Such is the importance of avoiding causal language. Even if the trend were curvilinear, the interpretation of $\beta$ is a population averaged parameter which estimates the first order trend of the kCal and glucose association. $E(u|X)=0$ even under curvillinearity for fixed $X$. As we know, 0 covariance does not imply independence. $\endgroup$ – AdamO Jun 13 '13 at 19:52
  • 1
    $\begingroup$ Sorry, I unnecessarily wrote "influence" - let's put the causality thing aside. You are absolutely right, $\beta$ can be interpreted as a population average partial effect of $X$. But that's also how you can interpret the coefficient of Best Linear Predictor. The thing is if true model is linear then this effect may be interpreted as effect for every individual, while in presence of quadratic trend population averaged parameter is the only interpretation (since people with kCal have different partial effect than the ones with low kCals). Anyway, this discussion helped me to understand this! $\endgroup$ – Michal Jun 13 '13 at 20:26
  • $\begingroup$ And I obviously consider stochastic $X$, otherwise how can we talk about covariance of $u$ and $X$?. $\endgroup$ – Michal Jun 13 '13 at 20:32
  • $\begingroup$ Prediction and inference are totally different things. A linear predictor is not an assumption about it's functional form being linear. If you're interested in quadratic effects, you must prespecify those terms. If I were to come up with a predictor $\hat{Y} = f(X)$, I find classical estimation theory to be complete bunk! Use smoothing splines, LOESS, or project pursuit regression to eschew any needs for assumptions about the shape of association. Evaluate models with cross validation, bootstrapping, and AIC. No need for asymptotic results to hold. $\endgroup$ – AdamO Jun 13 '13 at 20:43

When we talk about consistent estimation, we mean consistency of estimating the parameters $\beta$ from a regression like $$y = \alpha + \beta x + u$$ $\newcommand{\plim}{{\rm plim}}\newcommand{\Cov}{{\rm Cov}}\newcommand{\Var}{{\rm Var}}$ We don't know the true value of the slope of $x$ in this linear model, i.e. we don't know the true value of $\beta$. This is why we estimate it in the first place. When you estimate this model and if OLS is BLUE, then $$\plim\: \widehat{\beta}_{OLS} = \beta $$ This means, as your sample size becomes larger and larger, your estimate $\widehat{\beta}$ converges to the true value $\beta$. In this case you are "consistently" estimating this parameter. If you had the entire population as a sample, you would get $\widehat{\beta} = \beta$

As concerns your (1) and (2), $\Cov(X,u) = 0$ is one of the requirements for an estimator to be best, linear and unbiased (BLU). This doesn't mean that every estimator fulfills these requirements. If $\Cov(X,u) \neq 0$, OLS is biased (but it may still be "best", i.e. it has the smallest variance, and it will be "linear"). Consider again a linear regression model: $$y = \alpha + \beta x + \gamma d + u$$ Suppose that $\gamma \neq 0$, $\Cov(x,d) \neq 0$, and that $d$ is missing from the regression, so we only regress: $$y = \alpha + \beta x + e$$ This means that the $d$ is included in the error: $e = u + \gamma d$, and because $x$ is correlated with $d$, our OLS estimator is not BLUE anymore because $\Cov(x,e) \neq 0$ (since $d$ is inside $e$). Therefore, our estimate $\widehat{\beta}$ will be biased and inconsistent with $$\plim\: \widehat{\beta} = \beta + \gamma \frac{\Cov(x,d)}{\Var(x)}$$

As the sample size gets bigger and bigger, your estimate $\widehat{\beta}$ will not converge to the true value, i.e. it is inconsistently estimated. Instead it converges to the true value plus some bias (which depends on the size of $\gamma$, the correlation between $x$ and $d$ and the variance of $d$).

So you see that OLS is not BLUE by definition as you describe it in point (1). It is only BLUE if it fulfills the conditions set by the Gauss-Markov theorem. Concerning point (2), if OLS satisfies these conditions, then it is a best linear predictor of the conditional expectation.

  • 2
    $\begingroup$ The problem with this designation of "true" parameters in the case of model misspecification and working probability models is that these are not necessary conditions for consistent estimation. Even under model misspecification, the marginal parameter estimate $\beta_M$ from the working probability model $E[Y|X] = \alpha + \beta_M X$ can be consistently estimated even when $E[Y|X,W] = \alpha + \beta_C X + \gamma W$. As Tukey said, "An approximate answer to the exact question is infinitely more valuable that an exact answer to the approximate question." We may want to estimate $\beta_M$. $\endgroup$ – AdamO Jun 13 '13 at 18:14
  • $\begingroup$ Thank you very much for your answer. It fully adresses my doubts about my point (2). However, it does not really answer my doubts expressed in (1). Maybe I wasn't precise enough to state the question, let me rephrase it. When regressors are not perfectly collinear we can always define Best Linear Predictor (i.e. $\mathbb{L}(y|x) = x\beta$ where $\beta = argmin E(y-xb)^2$. Then the properties of BLP are such, that we can always write $y= x\beta + u$ (where $\beta$ is parameter of BLP) and in such a model $Cov(x,u) = 0$. Therefore we should always consistently estimate parameters of BLP, right? $\endgroup$ – Michal Jun 13 '13 at 19:11
  • $\begingroup$ Yes, but these may not correspond to the structural parameters of interest. $\endgroup$ – Christoph Hanck Jan 14 '16 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.