# Bias of Random Predictor in Linear Regression

Suppose we have a linear regression model with stochastic $$X$$ such that

$$Y = \beta X + \epsilon,$$

and $$X$$ and $$\epsilon$$ follow a bivariate normal distribution with $$\mu_x = \mu_{\epsilon} = 0$$, unknown variances $$\sigma_X^2$$ and $$\sigma_{\epsilon}^2$$, and correlation $$\rho = Corr(X, \epsilon)$$. Also, let $$(X_i, Y_i)$$ be iid random samples for all $$i \in \mathbb{N}$$. How does one find the bias of the least squares estimator $$\hat{\beta}$$?

It seems to me that we no longer have that $$\hat{\beta} = (X^TX)^{-1}X^TY$$. First, $$\rho = \frac{E[X\epsilon]}{\sigma_x\sigma_{\epsilon}}$$, and then the needed assumption doesn't hold

$$E[\epsilon|X] \neq 0 ~~\Rightarrow~~ E[\epsilon|X] = \mu_x + \rho\frac{\sigma_x}{\sigma_{\epsilon}}(Y - \mu_{\epsilon}) = \rho Y\frac{\sigma_{x}}{\sigma_{\epsilon}} = \frac{E[X\epsilon]}{\sigma_{\epsilon}^2}Y.$$

So, I guess I would need to find the new derivation of $$\hat{\beta}$$ first, but am not sure how to proceed?

• Hi: Is it stated somewhere that $X$ and $\epsilon$ are correlated ? I think you are assuming that but I'm not sure. ? Mar 10, 2020 at 2:11
• @mlofton Sorry, yes they are correlated: $\rho = Corr(X,\epsilon)$, which reduces a bit given the means are 0. Mar 10, 2020 at 2:30

The least square estimator is always $$\hat{\beta} = (X^TX)^{-1}X^TY$$. It's just when $$X$$ and $$\epsilon$$ are correlated, the estimate is no longer unbiased.

Note that

$$\hat{\beta} = (X^TX)^{-1}X^T(X\beta + \epsilon)$$

$$= \beta + (X^TX)^{-1} X^T\epsilon$$

Thus the bias of $$\hat{\beta}$$ is:

$$E[\hat{\beta} - \beta] = E (X^TX)^{-1} X^T\epsilon = E(\sum_i^n X_i^2)^{-1} \sum_i^n X_i \epsilon_i$$

While I am not sure how to easily calculate this with finite sample size, at least we can see that when the sample size becomes large, the bias term

$$(\sum_i^n X_i^2)^{-1} \sum_i^n X_i \epsilon_i = (\sum_i^n \frac{X_i^2}{n})^{-1} \sum_i^n \frac{X_i \epsilon_i}{n}$$ will converge to $$\frac{Cov(X, \epsilon)}{\sigma_X^2} = \frac{\rho \sigma_{\epsilon}}{\sigma_X}$$

• Nice answer. I'm pretty sure that the finite sample bias will just be the the empirical counterpart of your expression under : "Thus the bias of $\hat\beta$ is". So, one can calculate the finite sample bias by calculating the $\hat\epsilon_{i}$ and replacing the $\epsilon_{i}$ in your expression with those. Note though that, to calculate the $\hat{\epsilon}_{i}$, one would use the unbiased expression for $\hat{\beta}$, namely $(X^{T}X)^{-1} X^{T} Y$. Mar 10, 2020 at 19:15
• @mlofton I agree. But that is just an empirical estimate of the bias, and the quality of it heavily depends on the quality of your estimate of $\epsilon$. What I'm saying is that, since we are given the distributions of $X_i$'s and $\epsilon_i$'s, we should be able to work out the bias expression as a function of $\sigma_X, \sigma_{\epsilon}$ and $\rho$, without even seeing any data.
– Lii
Mar 12, 2020 at 3:25
• Thanks Lii: Just to make sure that I understand what you're saying: Do you mean, find the expectation and variance of the expression right before " will converge to". If so, I understand and thanks again. Note that I won't even attempt doing it. I'm happy enough just knowing what the goal would be :). Mar 13, 2020 at 4:32
• @mlofton Yes, exactly
– Lii
Mar 16, 2020 at 3:04