Suppose we have a linear regression model: $$ y_{i}=x_{i}\beta+\epsilon_{i} $$ Where $i$ is an index for individuals $i=1...N.$ Now, the requirement for unbiased estimation of $\beta$ via OLS requires that $$ cov(x_{i,}\epsilon_{i})=0 $$ Is the assumption of $cov\left(x_{i},\epsilon_{i}\right)$ required to be held for each individual or across different individuals? The Method of Moments uses the moment condition that this is $0$ across individuals, but given that $y_{i},x_{i}$ have a joint distribution together, does it mean it is needed for within the same individual as well?
$\begingroup$
$\endgroup$
0
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Short answer: You need to assume zero covariance across all pairs of individuals. If you are seeking to obtain unbiasedness via a covariance assumption (as opposed to the more direct linearity assumption) then you also need to assume joint normality of the explanatory vectors and error vector. Details follow.
Bias of the OLS estimator: When looking at the required conditions for unbiasedness in regression, it helps to start by writing the form of the OLS estimator as an expansion using the model form, so that you can write an explicit formula for its bias. In vector form, your regression model is:
$$\boldsymbol{Y} = \boldsymbol{x} \boldsymbol{\beta} + \boldsymbol{\varepsilon}.$$
Now, the OLS estimator can be written as:
$$\begin{equation} \begin{aligned} \hat{\boldsymbol{\beta}} &= (\boldsymbol{x}^\text{T} \boldsymbol{x})^{-1} \boldsymbol{x}^\text{T} \boldsymbol{Y} \\[6pt] &= (\boldsymbol{x}^\text{T} \boldsymbol{x})^{-1} \boldsymbol{x}^\text{T} (\boldsymbol{x} \boldsymbol{\beta} + \boldsymbol{\varepsilon}) \\[6pt] &= \boldsymbol{\beta} + (\boldsymbol{x}^\text{T} \boldsymbol{x})^{-1} \boldsymbol{x}^\text{T} \boldsymbol{\varepsilon}. \\[6pt] \end{aligned} \end{equation}$$
Hence, conditional on the explanatory vector, the bias of the OLS estimator (as an estimator of $\boldsymbol{\beta}$) is:
$$\begin{equation} \begin{aligned} \text{Bias}(\hat{\boldsymbol{\beta}} | \boldsymbol{x}) &= (\boldsymbol{x}^\text{T} \boldsymbol{x})^{-1} \boldsymbol{x}^\text{T} \mathbb{E}(\boldsymbol{\varepsilon} | \boldsymbol{x}). \\[6pt] \end{aligned} \end{equation}$$
So, at this point we can see that the OLS estimator will be unbiased so long as this expression is zero; any assumption that yields the zero vector for this expression is a sufficient condition for unbiasedness.
Assumptions that yield an unbiased OLS estimator: The usual assumption used here in linear regression is the linearity assumption which holds that $\mathbb{E}(\boldsymbol{\varepsilon} | \boldsymbol{x}) = \mathbf{0}$. This assumption means that the posed form for the true regression function encompasses the true conditional expectation of the response variable. As we can easily see, this assumption is a sufficient condition to ensure that the OLS estimator is unbiased.
In your question, you are concerned with a different set of assumptions on the covariance of the explanatory variables and the error terms. Assumptions about the covariance can also lead to the linearity assumption under certain conditions. Specifically, suppose we break the design matrix down into rows, corresponding to individual data points:
$$\boldsymbol{x} = \begin{bmatrix} \boldsymbol{x}_{(1)} \\ \boldsymbol{x}_{(2)} \\ \vdots \\ \boldsymbol{x}_{(n)} \end{bmatrix}$$
Suppose we assume that $\boldsymbol{x}_{(1)}, ..., \boldsymbol{x}_{(n)} \sim \text{IID N}(\boldsymbol{\mu_X}, \boldsymbol{\Sigma_X})$ (i.e., the explanatory vectors for each data point are IID normal). Moreover, suppose we assume that these vectors, and the error terms, are jointly normally distributed. Using the rules for the conditional moments of a normal distribution we then have:
$$\mathbb{E}(\boldsymbol{\varepsilon} | \boldsymbol{x}_{(i)}) = \mathbb{E}(\boldsymbol{\varepsilon}) + \mathbb{C}(\boldsymbol{\varepsilon}, \boldsymbol{x}_{(i)}) \boldsymbol{\Sigma_X}^{-1} (\boldsymbol{x}_{(i)} - \boldsymbol{\mu_X}).$$
If we write the design matrix out as one long vector $\boldsymbol{x}$ (as an abuse of notation we will write this with the same notation as the original design matrix) then we have:
$$\mathbb{E}(\boldsymbol{\varepsilon} | \boldsymbol{x}) = \mathbb{E}(\boldsymbol{\varepsilon}) + \mathbb{C}(\boldsymbol{\varepsilon}, \boldsymbol{x}) (\boldsymbol{I} \otimes \boldsymbol{\Sigma_X})^{-1} (\boldsymbol{x} - \mathbf{1} \otimes \boldsymbol{\mu_X}).$$
We can now see that taking $\mathbb{C}(\boldsymbol{\varepsilon}, \boldsymbol{x}) = \mathbf{0}$ and $\mathbb{E}(\boldsymbol{\varepsilon}) = \mathbf{0}$ is a sufficient condition (in combination with the IID normality assumption) to obtain the linearity assumption, which in turn ensures that the OLS estimator is unbiased.
