Covariance beween value and prediction in linear regression

Question

Equation (7.23) in the book "Elements of Statistical Learning" reads that for the model $Y = \beta \cdot X + \epsilon$ with $\epsilon \sim \mathcal N(0, \sigma^2)$, We have $$\sum _{i=1}^N \operatorname{Cov}(\hat y_i, y_i) = d \sigma^2,$$ where $d$ is the number of features and $N$ is the size of the training sample.

I don't understand how this can be true, since it seems evident that increasing the size of the training set should also increase this sum, as we add many terms in the sum while intuitively only decreasing the covariance a little. Moreover, I would like to know where I may find a proof of this fact, as it is just quoted in the book with seemingly no justification.

Answer 1

The other answer links to the proofs, but let's talk intuition.

Before we discuss linear regression, suppose you have a set of real numbers $\{x_1, \dots, x_m\}$ and you want to know how far you have to increase $x_1$ so that $\bar x_m$ increases by 1 unit. Do you have to move $x_1$ farther if $m = 10$ or if $m = 1000$? Clearly the answer is $m=1000$, which shows how the mean becomes less and less correlated with each particular input as we sample more and more. Asymptotically, if instead the $X_i$ are iid Gaussian RVs (for example, and switching to upper case to show they're RVs) with mean $\mu$, we have $\bar X_m \to_p \mu$ and constants are uncorrelated with everything, so in the limit our sample average is actually uncorrelated with each input observation.

Now, returning to linear regression, consider a single term $Cov(\hat y_i, y_i)$. This tells us how closely $\hat y_i$ tracks to $y_i$, and is exactly answering the question from the previous paragraph of how much $\hat y_i$ moves when $y_i$ moves. The more $\hat y_i$ changes by, the more flexible our model is (because changing a single observation can change its prediction), which is why $\sum_i Cov(\hat y_i, y_i)$ is a great definition of degrees of freedom.

Now imagine that we are fitting a linear regression and $N$ gets bigger and bigger, and we want to explore the effect of changing one particular observed value $y_0$. The more data we have, the farther we'll have to move $y_0$ in order to make an impact on our loss that's big enough for the trend line to move. In other words, as $N$ gets bigger, $Cov(\hat y_i, y_i)$ shrinks for each $i$ which is exactly what we saw with our $\bar X_m$ (and indeed, that corresponds exactly to an intercept-only linear regression). That's how this sum can work out, because even though we add more and more things, those individual things get smaller and smaller because our model complexity doesn't increase with $N$ (compare this to nonparametric methods like smoothing splines where as $N$ increases you can model a wider range of patterns).

Answer 2

This is more of a maths question, so here is the appropriate answer on Mathematics Stack Exchange:

https://math.stackexchange.com/a/266069/476670

The answer above shows that $\text{Cov}(\hat{\textbf{y}}, \textbf{y}) = \sigma^2\textbf{H}$, where $\textbf{H}$ is the hat matrix, $\textbf{X}\left(\textbf{X}^T\textbf{X}\right)^{-1}\textbf{X}^T$. We can extend this answer about the vector $\textbf{y}$ to your question about scalar $y_i$:

$$ \sum\limits_{i = 1}^N\text{Cov}(\hat{y}_i, y_i) = \text{tr}(\textbf{H})\sigma^2. $$

According to this other answer on Mathematics Stack Exchange, $\text{tr}(\textbf{H}) = \text{tr}(\textbf{I}_d) = d$.