Bias of Random Predictor in Linear Regression

Question

Suppose we have a linear regression model with stochastic $X$ such that

$$ Y = \beta X + \epsilon,$$

and $X$ and $\epsilon$ follow a bivariate normal distribution with $\mu_x = \mu_{\epsilon} = 0$, unknown variances $\sigma_X^2$ and $\sigma_{\epsilon}^2$, and correlation $\rho = Corr(X, \epsilon)$. Also, let $(X_i, Y_i)$ be iid random samples for all $i \in \mathbb{N}$. How does one find the bias of the least squares estimator $\hat{\beta}$?

It seems to me that we no longer have that $\hat{\beta} = (X^TX)^{-1}X^TY$. First, $\rho = \frac{E[X\epsilon]}{\sigma_x\sigma_{\epsilon}}$, and then the needed assumption doesn't hold

$$E[\epsilon|X] \neq 0 ~~\Rightarrow~~ E[\epsilon|X] = \mu_x + \rho\frac{\sigma_x}{\sigma_{\epsilon}}(Y - \mu_{\epsilon}) = \rho Y\frac{\sigma_{x}}{\sigma_{\epsilon}} = \frac{E[X\epsilon]}{\sigma_{\epsilon}^2}Y.$$

So, I guess I would need to find the new derivation of $\hat{\beta}$ first, but am not sure how to proceed?

Answer 1

The least square estimator is always $\hat{\beta} = (X^TX)^{-1}X^TY$. It's just when $X$ and $\epsilon$ are correlated, the estimate is no longer unbiased.

Note that

$\hat{\beta} = (X^TX)^{-1}X^T(X\beta + \epsilon)$

$= \beta + (X^TX)^{-1} X^T\epsilon$

Thus the bias of $\hat{\beta}$ is:

$E[\hat{\beta} - \beta] = E (X^TX)^{-1} X^T\epsilon = E(\sum_i^n X_i^2)^{-1} \sum_i^n X_i \epsilon_i$

While I am not sure how to easily calculate this with finite sample size, at least we can see that when the sample size becomes large, the bias term

$(\sum_i^n X_i^2)^{-1} \sum_i^n X_i \epsilon_i = (\sum_i^n \frac{X_i^2}{n})^{-1} \sum_i^n \frac{X_i \epsilon_i}{n}$ will converge to $\frac{Cov(X, \epsilon)}{\sigma_X^2} = \frac{\rho \sigma_{\epsilon}}{\sigma_X}$