5
$\begingroup$

lets say I run the simple regression, $y_i = \beta_o + \beta_1x_i + \epsilon_i$.. Assume $cov(\epsilon,x)$=0

This yields the formula people write in terms of covariances for the slope parameter:

$\hat{\beta_1}$ = $\frac{\sum(x-\bar{x})y_i}{\sum({x-\bar{x})^2}}$

and then plugging in the true assumed dgp for y, we get:

= $\beta + \frac{\sum(x-\bar{x})\epsilon_i}{\sum({x-\bar{x})^2}}$

With this, I have a few questions.

  1. is this is now a statement not about the population, but the 'draw' of $\epsilon_i$'s we so happened to draw in this sample? so it is the numerator second term the $\textit{sample}$ covariance between epsilon and x? if true, can I think of each random sample as a given draw of $\epsilon_i$'s, and that draw is what drives the sampling variability of the estimator?

2.taking probability limits, the covaraince =0 seems to be sufficient for consistency of the estimator. however, is covariance only not sufficient for unbiasedness? is mean indepence of $\epsilon$ and x needec for finite sample properties?

  1. An also a question about thinking about 'identification'. if i think of the model above as the causal model, and I can say my ols is consistent, does that mean I have 'identified' the true $\beta_1$? so can it hink of the model not being identified if the $cov(\epsilon,x) \neq 0$, which would say that $\hat{\beta}$ converges in probability to the true $\beta_1$ + some other term? so I fail to isolate the underlying parameter?
$\endgroup$

1 Answer 1

3
+50
$\begingroup$

I will give you a bit more information about the slope estimator and then I will proceed to your individual questions. To do this, let's keep your simple regression model, but drop the assumption that the explanatory variables and error terms are uncorrelated. We can then see what happens if the correlation is non-zero. Since $\sum (x_i-\bar{x}) = \sum x_i - n \bar{x} = n \bar{x} - n \bar{x} = 0$ you can write the slope estimator as:

$$\begin{align} \hat{\beta}_1 &= \beta_1 + \frac{\sum (x_i-\bar{x}) \epsilon_i}{\sum (x_i-\bar{x})^2} \\[6pt] &= \beta_1 + \frac{\sum (x_i-\bar{x}) (\epsilon_i-\bar{\epsilon})}{\sum (x_i-\bar{x})^2} \\[6pt] &= \beta_1 + \frac{\tfrac{1}{n-1} \sum (x_i-\bar{x}) (\epsilon_i-\bar{\epsilon})}{\tfrac{1}{n-1} \sum (x_i-\bar{x})^2} \\[6pt] &= \beta_1 + \frac{s_{X, \varepsilon}^2}{s_X^2}, \\[6pt] \end{align}$$

where $s_{X, \varepsilon}^2 \equiv \tfrac{1}{n-1} \sum (x_i-\bar{x}) (\epsilon_i-\bar{\epsilon})$ denotes the "sample covariance" between the explanatory variables and errors. (Note that this is unobservable, since the true errors are unobservable.) This means that the estimation error in this estimator is equal to the ratio of this sample covariance over the sample variance of the explanatory variables.

Your interest lies in the case where you are random sampling values from a population, so it is legitimate to assume that the series of regression obesrvations (i.e., the series $(X_1,Y_1),(X_2,Y_2),(X_3,Y_3),...$) is exchangeable. Under that assumption, the sample variance and covariance converge to the true variance and covariance. Thus, if the explanatory variable has a non-zero variance (and obscuring the difference between weak and strong probabilisitic convergence) then we have:

$$\hat{\beta}_1 \rightarrow \beta_1 + \frac{\mathbb{C}(X,\epsilon)}{\mathbb{V}(X)}.$$

This gives us a general asymptotic result for the slope estimator under an assumption of exchangeability of the underlying regression data. You can see that if the explanatory variables are uncorrelated with the error terms (and the variance of the explanatory variables is non-zero) then the second term in this equation vanishes, which gives the desired consistency property. (Here I am intentionally obscuring the difference between weak and strong consistency; to get these you would apply either convergence in probability or almost sure convergence respectively.)


  1. Your equation for $\hat{\beta}_1$ writes this estimator as a function of the true slope parameter, the sample explanatory values and the (unobserved) error terms $\epsilon_1,...,\epsilon_n$. So yes, this equation uses only the error terms in the sample, and not any other error terms in the broader population. You can indeed think of each sample as a draw of explanatory variables and a corresponding draw of error terms for those samples, and this is sufficient to give you the second term in your equation (which is indeed what drives the variability of the estimator).

  2. The assumption that $\mathbb{C}(X,\epsilon)=0$ is sufficient for the above convergence result only when you assume exchangeability of the underlying series of regression observations. If you assume exchangeability then you can use all the standard convergence properties of sample variance/covariance under random sampling. If you do not assume this then you need some alternative direct assumption about the convergence of the second term in the equation for the slope estimator. In standard expositions of limit properties in regression, it is usual to impose some direct requirement on the series of explanatory variables to ensure that they do not "explode" in a way that screws up the convergence.$^\dagger$ In any case, assuming that your convergence result holds, this shows that the slope estimator converges to a constant. Thus, it is not just a statement of asymptotic unbiasedness --- it is an actual consistency result. As to whether the zero-covariance assumption is needed for "finite sample properties", it really depends on what properties you are talking about. Obviously, in any finite sample, that second term in the above equation is going to exert an influence on the slope estimator. If the explanatory variables and error terms are correlated, we would expect the slope coefficient to systematically under- or over-estimate the true slope parameter.

  3. In statistical parlance, the concept of "identification" pertains to whether or not the parameters affect the likelihood function. You seem to be using the term in a more informal sense, to refer to determining the true causal effect in the regression. In any case, it is certainly true that if you assume that the regression model is the true causal mechanism for the data, then consistent estimation of the parameters is equivalent to consistent estimation of the causal parameters. And yes, if you don't have a consistent estimator, then you are not identifying the true causal parameters.


$^\dagger$ Specifically, what you are trying to prevent is the case where the magnitude of the explanatory variables gets larger and larger, so that some finite set of data points always "dominates" the regression. For further information on this, have a look at standard convergence theorems for regression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.