Once expressed the simple OLS residual $e_i$ as a weighted sum of the noise terms:

\begin{equation}e_{i}=\sum_{j}\left(\delta_{i j}-\frac{1}{n}-\left(x_{i}-\overline{x}\right) \frac{x_{j}-\overline{x}}{n s_{X}^{2}}\right) \varepsilon_{j},\end{equation}

where $\delta_{i j}$ is 1 when $i=j$ and 0 otherwise, I need to prove that:

$$ E(e_{i}^{2})=\sigma^{2}\left(1-\frac{1}{n}-\frac{\left(x_{i}-\overline{x}\right)^{2}}{n s_{X}^{2}}\right).$$

However, I squared the first equation and then took the expected value and exploiting the fact that $E(\varepsilon_i\varepsilon_j) = 0$ for $i\ne j$, I arrive to:

$$ E(e_{i}^{2})=\sigma^{2}\sum_{j}\left(\delta_{i j}-\frac{1}{n}-\left(x_{i}-\overline{x}\right) \frac{x_{j}-\overline{x}}{n s_{X}^{2}}\right) ^2.$$

What I'm doing wrong?

  • $\begingroup$ Thanks, I corrected the question with: $E(\varepsilon_i\varepsilon_j) = 0.$ It was a typo, so I think $E(e_{i}^{2})=\sigma^{2}\sum_{j}\left(1-\frac{1}{n}-\left(x_{i}-\overline{x}\right) \frac{x_{j}-\overline{x}}{n s_{X}^{2}}\right) ^2$ is true. $\endgroup$ – pdb Mar 8 '19 at 10:58
  • $\begingroup$ OK, now simply develop the square. Alternatively use matrix computations involving the hat matrix which you expressed. $\endgroup$ – Yves Mar 8 '19 at 11:12
  • $\begingroup$ Once developed the square, things become messy, and I do not see how, from squared terms, I get to $\left(1-\frac{1}{n}-\frac{\left(x_{i}-\overline{x}\right)^{2}}{n s_{X}^{2}}\right).$ $\endgroup$ – pdb Mar 8 '19 at 13:36
  • $\begingroup$ OK then go for matrices. The hat matrix $\mathbf{H}:= \mathbf{X}[\mathbf{X}^\top\mathbf{X}]^{-1}\mathbf{X}^\top$ has element $H_{ij} = [1 + (x_i - \bar{x})(x_j - \bar{x})/ s_X^2] / n$ as you wrote. We have $\mathbf{e} = [ \mathbf{I} - \mathbf{H}] \boldsymbol{\varepsilon}$ and the main point is that $\mathbf{H}$ and $\mathbf{I} - \mathbf{H}$ are orthogonal projection matrices. $\endgroup$ – Yves Mar 8 '19 at 14:20
  • $\begingroup$ Ok, I computed the hat matrix and I have seen that it has the form you indicated. Then I computed $E(\sum e_i) = E(\mathbf{e'}\mathbf{e}) = E(\boldsymbol{\varepsilon}'[ \mathbf{I} - \mathbf{H}] \boldsymbol{\varepsilon})$ and saw that the terms that remain are those having form $\sigma^{2}\left(1-\frac{1}{n}-\frac{\left(x_{i}-\overline{x}\right)^{2}}{n s_{X}^{2}}\right)$, so I suppose I can express each $E(e_{i}^{2})$ in that way. So I think I messed up somewhere with the first multiplication $e_ie_i$. $\endgroup$ – pdb Mar 8 '19 at 17:03

Ok, the previous steps were correct and I arrived to the result I needed:

\begin{align} E(e_{i}^{2}) &=\sigma^{2}\sum_{j}\left(\delta_{i j}-\frac{1}{n}-\left(x_{i}-\overline{x}\right) \frac{x_{j}-\overline{x}}{n s_{X}^{2}}\right) ^2 \\ &=\sigma^{2}\left(1-\frac{1}{n} -2\frac{\left(x_{i}-\overline{x}\right)^{2}}{n s_{X}^{2}} -\frac{1}{n} + \frac{n}{n^2} + 2\frac{\left(x_{i}-\overline{x}\right)\overbrace{\sum_j\left(x_{j}-\overline{x}\right)}^{0}}{n^2s_{X}^{2}} + \\ + \frac{\left(x_{i}-\overline{x}\right)^{2}\overbrace{\sum_j\left(x_{j}-\overline{x}\right)^2}^{ns_{X}^{2}}}{n^2 s_{X}^{2}s_{X}^{2}}\right) \\ &=\sigma^{2}\left(1-\frac{1}{n} - \frac{\left(x_{i}-\overline{x}\right)^{2}}{n s_{X}^{2}} \right) \end{align}

Thanks to Yves for the precious insights.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.