# In simple linear regression, what is the covariance between the error term and the residual?

In simple linear regression, what is the covariance between the error term and the residual?

Model: $y_i = \beta_0 +\beta_1 x_i + \varepsilon_i$

What will be the $\rm {cov}(\varepsilon_i,\ e_i)$, where $e_i = y_i - \hat y_i$?

-

The covariance will be $\sigma^2$. I will elaborate:

$\hat{y}_i = \hat{\beta}_0 +\hat{\beta}_1x_1$ and $y_i = \beta_0+\beta_1 x_1 +\varepsilon_i$, suppose $\varepsilon_i \sim N(0,\sigma^2)$.

The residual: $$e_i = y_i-\hat{y}_i = \beta_0+\beta_1 x_1 +\varepsilon_i - (\hat{\beta}_0 +\hat{\beta}_1x_1) = (\beta_0 - \hat{\beta}_0) + (\beta_1-\hat{\beta}_1)x_1 + \varepsilon_i$$

Then:

$$\text{Cov}[e_i, \varepsilon_i] = \text{Cov}[(\beta_0 - \hat{\beta}_0) + (\beta_1-\hat{\beta}_1)x_1 + \varepsilon_i, \varepsilon_i] = \text{Cov}[\varepsilon_i, \varepsilon_i]=\text{Var}[\varepsilon_i]=\sigma^2$$

See full list of covariance properties: covariance and variance.

-
But should the last line be $\text{Cov}[\epsilon_i, \epsilon_i] = \sigma^2$ by gauss markov conditions? – user2350622 Apr 21 '14 at 3:49

Assuming the benchmark model, (deterministic or strictly exogenous regressors, homoskedastic and non-autocorrelated error), we have

$$\operatorname {Cov}(u_i,\hat u_i) = E(u_i\hat u_i) -E(u_i)E(\hat u_i) = E(u_i\hat u_i)$$

Also $$\hat u_i = y_i-\mathbf x_i'\hat\beta = y_i-\mathbf x_i'\left(\beta + (\mathbf X'\mathbf X)^{-1}\mathbf X'\mathbf u\right)=u_i - \mathbf x_i'(\mathbf X'\mathbf X)^{-1}\mathbf X'\mathbf u$$

so $$E(u_i\hat u_i) = E(u_i^2) - E(u_i\mathbf x_i'(\mathbf X'\mathbf X)^{-1}\mathbf X'\mathbf u)$$

Given the assumptions of the benchmark model, the above becomes

$$\operatorname {Cov}(u_i,\hat u_i)=E(u_i\hat u_i) = \sigma^2 - \mathbf x_i'(\mathbf X'\mathbf X)^{-1}\mathbf x_i\sigma^2 = \sigma^2\left(1-\mathbf x_i'(\mathbf X'\mathbf X)^{-1}\mathbf x_i\right)$$

(if regressors are assumed stochastic and the error assumptions are expressed conditional on the regressors, then we need first to apply the law of total variance to show that the unconditional variance equals the conditional one, and also use the law of iterated expectations to disentangle the $X$'s from the error term in the second term of the expression).

Responding to OP's request in a comment, since the regressor matrix contains a series of ones and one other regressor, we have that

$$\mathbf x_i'(\mathbf X'\mathbf X)^{-1}\mathbf x_i = [\begin{matrix} 1& x_{1i} \end{matrix}] \left[\begin{matrix} \sum_{i=1}^nx_{1i}^2 & -\sum_{i=1}^nx_{1i}\\ -\sum_{i=1}^nx_{1i} & n \end{matrix}\right] \left[\begin{matrix} 1 \\ x_{1i} \end{matrix}\right]\cdot \left(n\sum_{i=1}^nx_{1i}^2-\left(\sum_{i=1}^nx_{1i}\right)^2\right)^{-1}$$

The last term (a scalar) is the determinant of $\mathbf X'\mathbf X$. Vector-matrix multiplication does the rest.

-
Sorry I just started learning simple linear regression and I am not very familiar with the matrix form notation. Could you tell me what it would be in simple linear regression? – user2350622 Apr 22 '14 at 2:52

Usually, the estimated parameters are denoted as either $\hat{\beta_i}$ or $b_i$, while $\beta_i$ is reserved for true values. So, your equation looks like the one with true parameters and errors. In such case, the covariance would be unknown.

If you meant to show the model specification, and $\epsilon_i$ denote residuals, not errors, then the answer is "zero by design". In OLS residuals will come uncorrelated with regressors. This doesn't mean that true errors are not correlated with residuals.

-