This seems to be a very general question about the bias OLS produces for RHS variables with unequal variance, but I was not able to find an explicit solution anywhere.

Suppose we have realizations of a random variable $y_t \sim N(0, \sigma_y^2), t = 1, \ldots, N$.

And we have $x_{it}$, noisy estimators of $y_t$, for $i = 1, \ldots, k$

$x_{it} = y_t + \delta_{it}, \delta_{it} \sim N(0, \sigma_i^2)$

If we run an OLS regression on $y_t = \beta_1 x_{1t} + \beta_2 x_{2t} + \epsilon_t$

we get $\sum \hat\beta_i \neq 1$, which is not intuitive to me.

First, is this even a bias? Or is it just compensating for the unequal variance?

If we normalized everything properly so $\sigma_y^2 = \sigma_{x_i}^2$, we should get $E[\hat\beta_i] = 1 / k$ as $N \rightarrow \infty$.

I have empirical results from simulations below, but I was curious about the functional form of the asymptotic of $E[\hat\beta_i]$ as a function of $\sigma_y^2, \sigma_1^2, \ldots, \sigma_k^2$.

$f(\sigma_y, \sigma_1) \rightarrow \hat\beta_1, f(1,1) \rightarrow 1/2, f(1,2) \rightarrow1/5, f(2,1) \rightarrow 4/5$

$f(\sigma_y, \sigma_1, \sigma_2) \rightarrow (\hat\beta_1, \hat\beta_2), f(1,1,1) \rightarrow (1/3, 1/3), f(2,1,1) \rightarrow (4/9, 4/9), f(1,2,1) \rightarrow (1/9, 4/9)$

Furthermore, what if the RHS errors are correlated, such that $\sigma_{ij}^2 \neq 0$. How does it generalize for non-diagonal $\Sigma$?


Let's make it simple. Suppose we only have one $x$. You did not say, but I'll assume that $y$ and $\delta$ are uncorrelated. If we regress $y$ on $x$ we get for the slope coeficient:

\begin{align} \hat{\beta}&=\frac{\widehat{Cov}(y,x)}{\hat{V}(x)}\\ \strut \\ &\xrightarrow{P} \frac{Cov(y,x)}{V(x)}\\ \strut \\ &= \frac{\sigma^2_y}{\sigma^2_y+\sigma^2_\delta} < 1 \end{align}

The less-than-one thing isn't going away no matter what.

That's not very satisfying, though. What is the reason that OLS does not work properly? What is the violation of the assumptions of the classical linear regression model? Well, the true model is $x_i=1\cdot y_i + \delta_i$. If we estimated that model, we would get a slope estimate on $y$ which is unbiased, consistent, etc.

But that is not what we estimated, we estimated this model: $y_i=1\cdot x_i-\delta_i$. Now, that's also a true equation. Recall, though, that for OLS to give unbiased, consistent estimates of the slope paramenter, the $x$ variable has to be uncorrelated with the error term. The thing is that the covariance between $x_i$ and $\delta_i$ is not zero. When $x$ is high, that is often because $\delta$ is high---just look at how $x$ was made! In fact $Cov(x,\delta)=\sigma^2_\delta$ which means that $Cov(x,-\delta)=-\sigma^2_\delta$.

If you remember the formula for the endogeneity bias in the bivariate linear regression model, the bias is $\frac{Cov(x,\text{error term})}{V(x)}=\frac{-\sigma^2_\delta}{\sigma^2_\delta+\sigma^2_y}$. Lo and behold! That is exactly the amount that $\hat{\beta}$ is off by!

  • $\begingroup$ Yes. That makes total sense, it's a RHS correlation problem. $X'X \rightarrow \sigma_y^2 + \Sigma$, where $\sigma_y^2$ is scalar and $\Sigma$ is the correlation matrix of $\sigma_{ij}^2$. And $X'Y \rightarrow vec(\sigma_y^2)$ So $\hat\beta = (X'X)^{-1}X'Y \rightarrow (\sigma_y^2 + \Sigma)^{-1} vec(\sigma_y^2)$ is the explicit solution for generalized $\Sigma$. Thanks! $\endgroup$ – ken Jan 23 '14 at 23:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.