Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

In simple linear regression, what is the covariance between the error term and the residual?

Model: $y_i = \beta_0 +\beta_1 x_i + \varepsilon_i$

What will be the $\rm {cov}(\varepsilon_i,\ e_i)$, where $e_i = y_i - \hat y_i$?

share|improve this question

3 Answers 3

The covariance will be $\sigma^2$. I will elaborate:

$\hat{y}_i = \hat{\beta}_0 +\hat{\beta}_1x_1$ and $y_i = \beta_0+\beta_1 x_1 +\varepsilon_i$, suppose $\varepsilon_i \sim N(0,\sigma^2)$.

The residual: $$e_i = y_i-\hat{y}_i = \beta_0+\beta_1 x_1 +\varepsilon_i - (\hat{\beta}_0 +\hat{\beta}_1x_1) = (\beta_0 - \hat{\beta}_0) + (\beta_1-\hat{\beta}_1)x_1 + \varepsilon_i$$

Then:

$$\text{Cov}[e_i, \varepsilon_i] = \text{Cov}[(\beta_0 - \hat{\beta}_0) + (\beta_1-\hat{\beta}_1)x_1 + \varepsilon_i, \varepsilon_i] = \text{Cov}[\varepsilon_i, \varepsilon_i]=\text{Var}[\varepsilon_i]=\sigma^2$$

See full list of covariance properties: covariance and variance.

share|improve this answer
    
But should the last line be $\text{Cov}[\epsilon_i, \epsilon_i] = \sigma^2$ by gauss markov conditions? –  user2350622 Apr 21 at 3:49

Assuming the benchmark model, (deterministic or strictly exogenous regressors, homoskedastic and non-autocorrelated error), we have

$$\operatorname {Cov}(u_i,\hat u_i) = E(u_i\hat u_i) -E(u_i)E(\hat u_i) = E(u_i\hat u_i)$$

Also $$\hat u_i = y_i-\mathbf x_i'\hat\beta = y_i-\mathbf x_i'\left(\beta + (\mathbf X'\mathbf X)^{-1}\mathbf X'\mathbf u\right)=u_i - \mathbf x_i'(\mathbf X'\mathbf X)^{-1}\mathbf X'\mathbf u$$

so $$E(u_i\hat u_i) = E(u_i^2) - E(u_i\mathbf x_i'(\mathbf X'\mathbf X)^{-1}\mathbf X'\mathbf u)$$

Given the assumptions of the benchmark model, the above becomes

$$\operatorname {Cov}(u_i,\hat u_i)=E(u_i\hat u_i) = \sigma^2 - \mathbf x_i'(\mathbf X'\mathbf X)^{-1}\mathbf x_i\sigma^2 = \sigma^2\left(1-\mathbf x_i'(\mathbf X'\mathbf X)^{-1}\mathbf x_i\right)$$

(if regressors are assumed stochastic and the error assumptions are expressed conditional on the regressors, then we need first to apply the law of total variance to show that the unconditional variance equals the conditional one, and also use the law of iterated expectations to disentangle the $X$'s from the error term in the second term of the expression).

ADDENDUM
Responding to OP's request in a comment, since the regressor matrix contains a series of ones and one other regressor, we have that

$$\mathbf x_i'(\mathbf X'\mathbf X)^{-1}\mathbf x_i = [\begin{matrix} 1& x_{1i} \end{matrix}] \left[\begin{matrix} \sum_{i=1}^nx_{1i}^2 & -\sum_{i=1}^nx_{1i}\\ -\sum_{i=1}^nx_{1i} & n \end{matrix}\right] \left[\begin{matrix} 1 \\ x_{1i} \end{matrix}\right]\cdot \left(n\sum_{i=1}^nx_{1i}^2-\left(\sum_{i=1}^nx_{1i}\right)^2\right)^{-1}$$

The last term (a scalar) is the determinant of $\mathbf X'\mathbf X$. Vector-matrix multiplication does the rest.

share|improve this answer
    
Sorry I just started learning simple linear regression and I am not very familiar with the matrix form notation. Could you tell me what it would be in simple linear regression? –  user2350622 Apr 22 at 2:52

Usually, the estimated parameters are denoted as either $\hat{\beta_i}$ or $b_i$, while $\beta_i$ is reserved for true values. So, your equation looks like the one with true parameters and errors. In such case, the covariance would be unknown.

If you meant to show the model specification, and $\epsilon_i$ denote residuals, not errors, then the answer is "zero by design". In OLS residuals will come uncorrelated with regressors. This doesn't mean that true errors are not correlated with residuals.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.