I've recently seen the degrees-of-freedom for K-nearest neighbors regression specified like so:


But what does $Cov(y_i,\hat{y}_i)$ mean? Previously I've understood the notation $Cov(X,Y)$ to mean the covariance of the vectors $X$ and $Y$, resulting in a scalar. Here, the inputs $y_i$ and $\hat{y}_i$ are themselves scalars. I am confused.

  • 1
    $\begingroup$ Can you provide some context regarding where you have recently seen this? Eg can you paste in a quote &/or provide a cite? $\endgroup$ May 18, 2015 at 4:37
  • 1
    $\begingroup$ For a citable context, see page 77, equation 3.60, of Elements of Statistical Learning: statweb.stanford.edu/~tibs/ElemStatLearn/printings/… $\endgroup$
    – user77403
    May 18, 2015 at 5:17
  • 1
    $\begingroup$ I imagine gung was asking for a cite for in the expression $\text{Cov}(X,Y)$ only referring to vectors. Generally in an expression like $\text{Cov}(X,Y)$, $X$ and $Y$ represent random variables (which could indeed be scalar). $\endgroup$
    – Glen_b
    May 18, 2015 at 10:18
  • $\begingroup$ e.g. see here $\endgroup$
    – Glen_b
    May 18, 2015 at 10:23
  • $\begingroup$ and section 2] 1) for example : cs.cmu.edu/~epxing/Class/10708-15/notes/…, hard to understand.... the question is still open for me $\endgroup$
    – Tbertin
    Dec 12, 2018 at 3:05

3 Answers 3


This is the standard definition of the covariance: $$ \newcommand{\Cov}{\mathrm{Cov}} \newcommand{\E}{\mathbb{E}} \Cov(X, Y) = \E\left[ \left( X - \E[X] \right) \left( Y - \E[Y] \right) \right] ,$$ where here $X$ and $Y$ are real-valued random variables.

With an $n$-dimensional random vector $X$, the typical notation is $$ \Cov(X) = \left[ \Cov(X_i, X_j) \right]_{ij} ,$$ i.e. you get an $n \times n$ matrix where the $(i, j)$th entry is the covariance between $X_i$ and $X_j$.

I'm not familiar with a notation $\Cov(X, Y)$, where $X$ and $Y$ are random vectors and the covariance is a scalar.


I guess that $y_i$ and $ \hat{y}_i$ are vectors with respective means $ \bar{y}_i $ and $\hat{\bar{y}}_i$ respectively.

Maybe they should have specified $ 1/\sigma^2 $ $\sum\limits_{i=1}^n$ ($y_i$ - $ \bar{y}_i $)($ \hat{y}_i$ - $\hat{\bar{y}}_i$).

I will look into K-nearest neighbors regression and get back as I don't know the underlying theory.

  • $\begingroup$ I haven't looked up the context, but $y_i$ is almost certainly the scalar dependent variable and $\hat y_i$ the output of the regression. $\endgroup$
    – Danica
    May 18, 2015 at 8:11
  • $\begingroup$ Oops sorry, i mentioned vectors and showed it in scalar notation. Yes, $y_i$ should be the i$^th$ element of vector Y, looks like. $\endgroup$
    – KarthikS
    May 18, 2015 at 8:17

It follows from the definition of the effective degrees of freedom given by the trace of the hat matrix, $\nu=\mathrm{tr}(H)$; the diagonal of $H$ equals the ratio of the covariance between $\hat{y}$ and $y$ and the variance of $y$ (link).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.