# Notation: covariance on scalars?

I've recently seen the degrees-of-freedom for K-nearest neighbors regression specified like so:

$\frac{1}{\sigma^2}\sum_{i=1}^NCov(y_i,\hat{y}_i)$

But what does $Cov(y_i,\hat{y}_i)$ mean? Previously I've understood the notation $Cov(X,Y)$ to mean the covariance of the vectors $X$ and $Y$, resulting in a scalar. Here, the inputs $y_i$ and $\hat{y}_i$ are themselves scalars. I am confused.

• Can you provide some context regarding where you have recently seen this? Eg can you paste in a quote &/or provide a cite? May 18, 2015 at 4:37
• For a citable context, see page 77, equation 3.60, of Elements of Statistical Learning: statweb.stanford.edu/~tibs/ElemStatLearn/printings/… May 18, 2015 at 5:17
• I imagine gung was asking for a cite for in the expression $\text{Cov}(X,Y)$ only referring to vectors. Generally in an expression like $\text{Cov}(X,Y)$, $X$ and $Y$ represent random variables (which could indeed be scalar). May 18, 2015 at 10:18
• e.g. see here May 18, 2015 at 10:23
• and section 2] 1) for example : cs.cmu.edu/~epxing/Class/10708-15/notes/…, hard to understand.... the question is still open for me Dec 12, 2018 at 3:05

This is the standard definition of the covariance: $$\newcommand{\Cov}{\mathrm{Cov}} \newcommand{\E}{\mathbb{E}} \Cov(X, Y) = \E\left[ \left( X - \E[X] \right) \left( Y - \E[Y] \right) \right] ,$$ where here $X$ and $Y$ are real-valued random variables.

With an $n$-dimensional random vector $X$, the typical notation is $$\Cov(X) = \left[ \Cov(X_i, X_j) \right]_{ij} ,$$ i.e. you get an $n \times n$ matrix where the $(i, j)$th entry is the covariance between $X_i$ and $X_j$.

I'm not familiar with a notation $\Cov(X, Y)$, where $X$ and $Y$ are random vectors and the covariance is a scalar.

I guess that $y_i$ and $\hat{y}_i$ are vectors with respective means $\bar{y}_i$ and $\hat{\bar{y}}_i$ respectively.

Maybe they should have specified $1/\sigma^2$ $\sum\limits_{i=1}^n$ ($y_i$ - $\bar{y}_i$)($\hat{y}_i$ - $\hat{\bar{y}}_i$).

I will look into K-nearest neighbors regression and get back as I don't know the underlying theory.

• I haven't looked up the context, but $y_i$ is almost certainly the scalar dependent variable and $\hat y_i$ the output of the regression. May 18, 2015 at 8:11
• Oops sorry, i mentioned vectors and showed it in scalar notation. Yes, $y_i$ should be the i$^th$ element of vector Y, looks like. May 18, 2015 at 8:17

It follows from the definition of the effective degrees of freedom given by the trace of the hat matrix, $$\nu=\mathrm{tr}(H)$$; the diagonal of $$H$$ equals the ratio of the covariance between $$\hat{y}$$ and $$y$$ and the variance of $$y$$ (link).