1
$\begingroup$

Bellow is my deduction:

According to the definition of k-NN fit, we have $$\hat{Y}(x) = \frac{1}{k} \sum_{x_i \in N_k(x)}^{N}= \frac{1}{k}diag(a_1, a_2,..., a_N)y$$ where $N_k(x)$ is the neighborhood of $x$ defined by the k closest points $x_i$ in the training sample, and $diag(a_1, a_2,..., a_N)$ is a diagonal matrix, if $x_i \in N_k(x)$, $a_i=1$, else $a_i=0$.

Hence, the effective degrees-of-freedom $$df({S}) = trace({S})=trace(\frac{1}{k}diag(a_1, a_2,..., a_N))=\frac{trace(diag(a_1, a_2,..., a_N))}{K}=1$$


What's wrong with my deduction?

$\endgroup$
1
  • $\begingroup$ If $Y$ is multivariate normal, $\hat{Y}(x)$ is a linear combination of $k$-normal random variates. If the variance is unknown, the degrees of freedom depend on how you estimate the variance. Are you using $\sum_{i=1}^n (Y_i - \hat{Y}(x))^2 = SSYY$? $\endgroup$
    – AdamO
    Commented Nov 25, 2018 at 14:00

1 Answer 1

2
$\begingroup$

Sorry, I misunderstood the definition of $S$. Maybe below is a right deduction.

According to the definition of k-NN fit, we have $$\hat{Y}(x) = \frac{1}{k} \sum_{x_i \in N_k(x)}^{N}= \frac{1}{k}\boldsymbol{I}_N\boldsymbol{y}$$

where $N_k(x)$ is the neighborhood of $x$ defined by the k closest points $x_i$ in the training sample, $\boldsymbol{I_N}$ is a $1*N$ row vector and the i-th element $\boldsymbol{I_N}(i)=1$ if $x_i \in N_k(x)$, else $\boldsymbol{I_N}(i)=0$

Suppose we stack the outcomes $y_1, y_2, ..., y_N$ into a vector $\boldsymbol{y}$ , and similarly for the predictions $\boldsymbol{\hat{y}}$. Then the fitting method is one for which we can write $$\boldsymbol{\hat{y}}= \boldsymbol{Sy}$$ where $\boldsymbol{S}$ is an $N*N$ matrix depending on the input vectors $x_i$ but not on the $y_i$.

Hence, the effective degrees-of-freedom is $$df(\boldsymbol{S}) = trace(\boldsymbol{S})=trace(\frac{1}{k}\boldsymbol{I}_{N*N}) \approx \frac{N}{K}$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.