Linear Regression, $\mathrm{Cov}(\hat{y},e)=0$, correct Argument?

Question

I am trying to reproduce the equality of $R^2 = r_{y, \hat{y}}^2$ from this site. The author uses the equation $cov(\hat{y}, e) = 0$, which is what I am trying to explain.

Notation:

$X$ invertible matrix of explanatory variables

$y$ explained variable

Linear Model: $y = X\beta + e$, assuming $\mathbb{E}[e] = 0$

$\hat{\beta} = (X'X)^{-1}X'y$ (least squares estimator)

$\hat{y} = X\hat{\beta}$

$\hat{y}'e = 0: \hat{y}'e = \hat{\beta}'X'e = \hat{\beta}'X'(y - X\beta) = \hat{\beta}'(X'y - X'X\beta = 0) \Rightarrow \hat{y}'e = 0$

Furthermore: $\mathbb{E}[\hat{\beta}] = \beta$ (without proof)

I found this solution here in the forum, however I wanted to present my own and ask if my argument is correct.

$$ \begin{align*} cov(\hat{y}, e) &= \mathbb{E}[(\hat{y} - \mathbb{E}[\hat{y}])'(e - \mathbb{E}[e])] \quad(\mathbb{E}[e] = 0,\text{per assumption}) \\ cov(\hat{y}, e) &= \mathbb{E}[(\hat{y} - \mathbb{E}[\hat{y}])'e] = \mathbb{E}[\hat{y}'e - \mathbb{E}[\hat{y}]'e] = \mathbb{E}[0 - \mathbb{E}[\hat{y}]'e] \\ \mathbb{E}[0 - \mathbb{E}[\hat{y}]'e] &= \mathbb{E}[\mathbb{E}[X\hat{\beta}]'e] = \mathbb{E}[\beta'X'e] = 0 \end{align*} $$

Please let me know if there is an error in my argument.

Answer 1

The reference to $e$ in the linked site is a reference to the residual of the model, not the error term, so the equation $y = x \beta + e$ is not valid. In fact, the predicted values are correlated with the error terms, so your attempted proof is trying to prove something that is not true.

Suppose we instead use the proper meaning of the variables here, and work with the standard vector notation for the model. Using that hat matrix $\mathbf{h} \equiv \mathbf{x} (\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T}$ and the regression equation $\mathbf{Y} = \mathbf{x} \boldsymbol{\beta} + \boldsymbol{\varepsilon}$ you can decompose the predicted response vector and residual vector as:

$$\begin{align} \hat{\mathbf{Y}} &= \mathbf{h} \mathbf{Y} \\[6pt] &= \mathbf{h} (\mathbf{x} \boldsymbol{\beta} + \boldsymbol{\varepsilon}) \\[6pt] &= \mathbf{h} \mathbf{x} \boldsymbol{\beta} + \mathbf{h} \boldsymbol{\varepsilon} \\[6pt] &= \mathbf{x} \boldsymbol{\beta} + \mathbf{h} \boldsymbol{\varepsilon}, \\[6pt] \mathbf{e} &= (\mathbf{I}-\mathbf{h}) \mathbf{Y} \\[6pt] &= (\mathbf{I}-\mathbf{h}) (\mathbf{x} \boldsymbol{\beta} + \boldsymbol{\varepsilon}) \\[6pt] &= (\mathbf{I}-\mathbf{h}) \boldsymbol{\varepsilon}. \\[6pt] \end{align}$$

You then have:

$$\begin{align} \mathbb{C}(\hat{\mathbf{Y}}, \mathbf{e}) &= \mathbb{C}(\mathbf{x} \boldsymbol{\beta} + \mathbf{h} \boldsymbol{\varepsilon}, (\mathbf{I}-\mathbf{h}) \boldsymbol{\varepsilon}) \\[6pt] &= \mathbb{C}(\mathbf{h} \boldsymbol{\varepsilon}, (\mathbf{I}-\mathbf{h}) \boldsymbol{\varepsilon}) \\[6pt] &= \mathbf{h} \mathbb{V}(\boldsymbol{\varepsilon}) (\mathbf{I}-\mathbf{h})^\text{T} \\[6pt] &= \sigma^2 \mathbf{h} (\mathbf{I}-\mathbf{h})^\text{T} \\[6pt] &= \sigma^2 \mathbf{h} (\mathbf{I}-\mathbf{h}) \\[6pt] &= \sigma^2 (\mathbf{h} - \mathbf{h}^2) \\[6pt] &= \sigma^2 (\mathbf{h} - \mathbf{h}) \\[6pt] &= \sigma^2 \mathbf{0} = \mathbf{0}. \\[6pt] \end{align}$$

(Incidentally, for the error term, you get $\mathbb{C}(\hat{\mathbf{Y}}, \boldsymbol{\varepsilon}) = \sigma^2 \mathbf{h}$ so the terms of those vectors are correlated.)