2
$\begingroup$

Suppose we have a linear regression model with stochastic $X$ such that

$$ Y = \beta X + \epsilon,$$

and $X$ and $\epsilon$ follow a bivariate normal distribution with $\mu_x = \mu_{\epsilon} = 0$, unknown variances $\sigma_X^2$ and $\sigma_{\epsilon}^2$, and correlation $\rho = Corr(X, \epsilon)$. Also, let $(X_i, Y_i)$ be iid random samples for all $i \in \mathbb{N}$. How does one find the bias of the least squares estimator $\hat{\beta}$?

It seems to me that we no longer have that $\hat{\beta} = (X^TX)^{-1}X^TY$. First, $\rho = \frac{E[X\epsilon]}{\sigma_x\sigma_{\epsilon}}$, and then the needed assumption doesn't hold

$$E[\epsilon|X] \neq 0 ~~\Rightarrow~~ E[\epsilon|X] = \mu_x + \rho\frac{\sigma_x}{\sigma_{\epsilon}}(Y - \mu_{\epsilon}) = \rho Y\frac{\sigma_{x}}{\sigma_{\epsilon}} = \frac{E[X\epsilon]}{\sigma_{\epsilon}^2}Y.$$

So, I guess I would need to find the new derivation of $\hat{\beta}$ first, but am not sure how to proceed?

$\endgroup$
2
  • $\begingroup$ Hi: Is it stated somewhere that $X$ and $\epsilon$ are correlated ? I think you are assuming that but I'm not sure. ? $\endgroup$
    – mlofton
    Mar 10, 2020 at 2:11
  • $\begingroup$ @mlofton Sorry, yes they are correlated: $\rho = Corr(X,\epsilon)$, which reduces a bit given the means are 0. $\endgroup$
    – jj8989
    Mar 10, 2020 at 2:30

1 Answer 1

3
$\begingroup$

The least square estimator is always $\hat{\beta} = (X^TX)^{-1}X^TY$. It's just when $X$ and $\epsilon$ are correlated, the estimate is no longer unbiased.

Note that

$\hat{\beta} = (X^TX)^{-1}X^T(X\beta + \epsilon)$

$= \beta + (X^TX)^{-1} X^T\epsilon$

Thus the bias of $\hat{\beta}$ is:

$E[\hat{\beta} - \beta] = E (X^TX)^{-1} X^T\epsilon = E(\sum_i^n X_i^2)^{-1} \sum_i^n X_i \epsilon_i$

While I am not sure how to easily calculate this with finite sample size, at least we can see that when the sample size becomes large, the bias term

$(\sum_i^n X_i^2)^{-1} \sum_i^n X_i \epsilon_i = (\sum_i^n \frac{X_i^2}{n})^{-1} \sum_i^n \frac{X_i \epsilon_i}{n}$ will converge to $\frac{Cov(X, \epsilon)}{\sigma_X^2} = \frac{\rho \sigma_{\epsilon}}{\sigma_X}$

$\endgroup$
4
  • 1
    $\begingroup$ Nice answer. I'm pretty sure that the finite sample bias will just be the the empirical counterpart of your expression under : "Thus the bias of $\hat\beta$ is". So, one can calculate the finite sample bias by calculating the $\hat\epsilon_{i}$ and replacing the $\epsilon_{i}$ in your expression with those. Note though that, to calculate the $\hat{\epsilon}_{i}$, one would use the unbiased expression for $\hat{\beta}$, namely $(X^{T}X)^{-1} X^{T} Y$. $\endgroup$
    – mlofton
    Mar 10, 2020 at 19:15
  • $\begingroup$ @mlofton I agree. But that is just an empirical estimate of the bias, and the quality of it heavily depends on the quality of your estimate of $\epsilon$. What I'm saying is that, since we are given the distributions of $X_i$'s and $\epsilon_i$'s, we should be able to work out the bias expression as a function of $\sigma_X, \sigma_{\epsilon}$ and $\rho$, without even seeing any data. $\endgroup$
    – Lii
    Mar 12, 2020 at 3:25
  • $\begingroup$ Thanks Lii: Just to make sure that I understand what you're saying: Do you mean, find the expectation and variance of the expression right before " will converge to". If so, I understand and thanks again. Note that I won't even attempt doing it. I'm happy enough just knowing what the goal would be :). $\endgroup$
    – mlofton
    Mar 13, 2020 at 4:32
  • $\begingroup$ @mlofton Yes, exactly $\endgroup$
    – Lii
    Mar 16, 2020 at 3:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.