In simple linear regression, what is the covariance between the error term and the residual?
Model: $y_i = \beta_0 +\beta_1 x_i + \varepsilon_i$
What will be the $\rm {cov}(\varepsilon_i,\ e_i)$, where $e_i = y_i - \hat y_i$?
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
1
Assuming the benchmark model, (deterministic or strictly exogenous regressors, homoskedastic and non-autocorrelated error), we have
$$\operatorname {Cov}(u_i,\hat u_i) = E(u_i\hat u_i) -E(u_i)E(\hat u_i) = E(u_i\hat u_i)$$
Also $$\hat u_i = y_i-\mathbf x_i'\hat\beta = y_i-\mathbf x_i'\left(\beta + (\mathbf X'\mathbf X)^{-1}\mathbf X'\mathbf u\right)=u_i - \mathbf x_i'(\mathbf X'\mathbf X)^{-1}\mathbf X'\mathbf u$$
so $$E(u_i\hat u_i) = E(u_i^2) - E(u_i\mathbf x_i'(\mathbf X'\mathbf X)^{-1}\mathbf X'\mathbf u)$$
Given the assumptions of the benchmark model, the above becomes
$$\operatorname {Cov}(u_i,\hat u_i)=E(u_i\hat u_i) = \sigma^2 - \mathbf x_i'(\mathbf X'\mathbf X)^{-1}\mathbf x_i\sigma^2 = \sigma^2\left(1-\mathbf x_i'(\mathbf X'\mathbf X)^{-1}\mathbf x_i\right)$$
(if regressors are assumed stochastic and the error assumptions are expressed conditional on the regressors, then we need first to apply the law of total variance to show that the unconditional variance equals the conditional one, and also use the law of iterated expectations to disentangle the $X$'s from the error term in the second term of the expression).
ADDENDUM
Responding to OP's request in a comment, since the regressor matrix contains a series of ones and one other regressor, we have that
$$\mathbf x_i'(\mathbf X'\mathbf X)^{-1}\mathbf x_i = [\begin{matrix} 1& x_{1i} \end{matrix}] \left[\begin{matrix} \sum_{i=1}^nx_{1i}^2 & -\sum_{i=1}^nx_{1i}\\ -\sum_{i=1}^nx_{1i} & n \end{matrix}\right] \left[\begin{matrix} 1 \\ x_{1i} \end{matrix}\right]\cdot \left(n\sum_{i=1}^nx_{1i}^2-\left(\sum_{i=1}^nx_{1i}\right)^2\right)^{-1}$$
The last term (a scalar) is the determinant of $\mathbf X'\mathbf X$. Vector-matrix multiplication does the rest.
answered Apr 21, 2014 at 12:36
-
$\begingroup$ Sorry I just started learning simple linear regression and I am not very familiar with the matrix form notation. Could you tell me what it would be in simple linear regression? $\endgroup$ Commented Apr 22, 2014 at 2:52
$\begingroup$
$\endgroup$
2
The covariance will be $\sigma^2$. I will elaborate:
$\hat{y}_i = \hat{\beta}_0 +\hat{\beta}_1x_1$ and $y_i = \beta_0+\beta_1 x_1 +\varepsilon_i$, suppose $\varepsilon_i \sim N(0,\sigma^2)$.
The residual: $$e_i = y_i-\hat{y}_i = \beta_0+\beta_1 x_1 +\varepsilon_i - (\hat{\beta}_0 +\hat{\beta}_1x_1) = (\beta_0 - \hat{\beta}_0) + (\beta_1-\hat{\beta}_1)x_1 + \varepsilon_i$$
Then:
$$\text{Cov}[e_i, \varepsilon_i] = \text{Cov}[(\beta_0 - \hat{\beta}_0) + (\beta_1-\hat{\beta}_1)x_1 + \varepsilon_i, \varepsilon_i] = \text{Cov}[\varepsilon_i, \varepsilon_i]=\text{Var}[\varepsilon_i]=\sigma^2$$
See full list of covariance properties: covariance and variance.
-
$\begingroup$ But should the last line be $\text{Cov}[\epsilon_i, \epsilon_i] = \sigma^2$ by gauss markov conditions? $\endgroup$ Commented Apr 21, 2014 at 3:49
-
1$\begingroup$ The terms $\beta_0$ and $\beta_1$ depend on $\varepsilon_1, ..., \varepsilon_n$, so the removal of these terms from the covariance is invalid. $\endgroup$– BenCommented Mar 26, 2018 at 1:07
$\begingroup$
$\endgroup$
Usually, the estimated parameters are denoted as either $\hat{\beta_i}$ or $b_i$, while $\beta_i$ is reserved for true values. So, your equation looks like the one with true parameters and errors. In such case, the covariance would be unknown.
If you meant to show the model specification, and $\epsilon_i$ denote residuals, not errors, then the answer is "zero by design". In OLS residuals will come uncorrelated with regressors. This doesn't mean that true errors are not correlated with residuals.
