I am trying to reproduce the equality of $R^2 = r_{y, \hat{y}}^2$ from this site. The author uses the equation $cov(\hat{y}, e) = 0$, which is what I am trying to explain.


$X$ invertible matrix of explanatory variables

$y$ explained variable

Linear Model: $y = X\beta + e$, assuming $\mathbb{E}[e] = 0$

$\hat{\beta} = (X'X)^{-1}X'y$ (least squares estimator)

$\hat{y} = X\hat{\beta}$

$\hat{y}'e = 0: \hat{y}'e = \hat{\beta}'X'e = \hat{\beta}'X'(y - X\beta) = \hat{\beta}'(X'y - X'X\beta = 0) \Rightarrow \hat{y}'e = 0$

Furthermore: $\mathbb{E}[\hat{\beta}] = \beta$ (without proof)

I found this solution here in the forum, however I wanted to present my own and ask if my argument is correct.

$$ \begin{align*} cov(\hat{y}, e) &= \mathbb{E}[(\hat{y} - \mathbb{E}[\hat{y}])'(e - \mathbb{E}[e])] \quad(\mathbb{E}[e] = 0,\text{per assumption}) \\ cov(\hat{y}, e) &= \mathbb{E}[(\hat{y} - \mathbb{E}[\hat{y}])'e] = \mathbb{E}[\hat{y}'e - \mathbb{E}[\hat{y}]'e] = \mathbb{E}[0 - \mathbb{E}[\hat{y}]'e] \\ \mathbb{E}[0 - \mathbb{E}[\hat{y}]'e] &= \mathbb{E}[\mathbb{E}[X\hat{\beta}]'e] = \mathbb{E}[\beta'X'e] = 0 \end{align*} $$

Please let me know if there is an error in my argument.


The reference to $e$ in the linked site is a reference to the residual of the model, not the error term, so the equation $y = x \beta + e$ is not valid. In fact, the predicted values are correlated with the error terms, so your attempted proof is trying to prove something that is not true.

Suppose we instead use the proper meaning of the variables here, and work with the standard vector notation for the model. Using that hat matrix $\mathbf{h} \equiv \mathbf{x} (\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T}$ and the regression equation $\mathbf{Y} = \mathbf{x} \boldsymbol{\beta} + \boldsymbol{\varepsilon}$ you can decompose the predicted response vector and residual vector as:

$$\begin{align} \hat{\mathbf{Y}} &= \mathbf{h} \mathbf{Y} \\[6pt] &= \mathbf{h} (\mathbf{x} \boldsymbol{\beta} + \boldsymbol{\varepsilon}) \\[6pt] &= \mathbf{h} \mathbf{x} \boldsymbol{\beta} + \mathbf{h} \boldsymbol{\varepsilon} \\[6pt] &= \mathbf{x} \boldsymbol{\beta} + \mathbf{h} \boldsymbol{\varepsilon}, \\[6pt] \mathbf{e} &= (\mathbf{I}-\mathbf{h}) \mathbf{Y} \\[6pt] &= (\mathbf{I}-\mathbf{h}) (\mathbf{x} \boldsymbol{\beta} + \boldsymbol{\varepsilon}) \\[6pt] &= (\mathbf{I}-\mathbf{h}) \boldsymbol{\varepsilon}. \\[6pt] \end{align}$$

You then have:

$$\begin{align} \mathbb{C}(\hat{\mathbf{Y}}, \mathbf{e}) &= \mathbb{C}(\mathbf{x} \boldsymbol{\beta} + \mathbf{h} \boldsymbol{\varepsilon}, (\mathbf{I}-\mathbf{h}) \boldsymbol{\varepsilon}) \\[6pt] &= \mathbb{C}(\mathbf{h} \boldsymbol{\varepsilon}, (\mathbf{I}-\mathbf{h}) \boldsymbol{\varepsilon}) \\[6pt] &= \mathbf{h} \mathbb{V}(\boldsymbol{\varepsilon}) (\mathbf{I}-\mathbf{h})^\text{T} \\[6pt] &= \sigma^2 \mathbf{h} (\mathbf{I}-\mathbf{h})^\text{T} \\[6pt] &= \sigma^2 \mathbf{h} (\mathbf{I}-\mathbf{h}) \\[6pt] &= \sigma^2 (\mathbf{h} - \mathbf{h}^2) \\[6pt] &= \sigma^2 (\mathbf{h} - \mathbf{h}) \\[6pt] &= \sigma^2 \mathbf{0} = \mathbf{0}. \\[6pt] \end{align}$$

(Incidentally, for the error term, you get $\mathbb{C}(\hat{\mathbf{Y}}, \boldsymbol{\varepsilon}) = \sigma^2 \mathbf{h}$ so the terms of those vectors are correlated.)

  • $\begingroup$ I think this demonstration is more complicated than necessary, in that the result holds true algebraically without reference to any postulated model $x\beta+\epsilon$, following from $(I-h)h=0$. $\endgroup$ Jun 29 '20 at 15:01
  • $\begingroup$ The reference to the postulated model form is used in the initial expansions for $\hat{\mathbf{Y}}$ and $\mathbf{e}$. $\endgroup$
    – Ben
    Jun 29 '20 at 22:05
  • $\begingroup$ Yes, what I was trying to say is that the OP seems to be about the sample covariance, but (imo, erroneously) uses expectations and population covariances to derive the result. $\endgroup$ Jun 30 '20 at 5:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.