I was reading elements of statistical learning and it mentions let $\hat{f}(x)=Ave(y_i|x_i \in N_k(x))$ where $N_k(x)$ is the neighborhood containing the k points closest to x .

Then it says "under mild regularity conditions on the joint probability distribution $Pr(X, Y )$, one can show that as $N, k \to \infty$ such that $k/N \to 0$, $\hat{f}(x) \to E(Y |X = x)$.

Can someone guide how to begin proving this statement and what mild regularity conditions are required?

  • 2
    $\begingroup$ Might want to check stats.stackexchange.com/questions/96044/… for pointers in the answer. $\endgroup$
    – jbowman
    Oct 26, 2023 at 19:32
  • 2
    $\begingroup$ Although the details of "one can show that" is definitely not easy, the first step of approaching the problem is rewrite $\hat{f}(x)$ as the weighted sum of $Y_i$, i.e., $\hat{f}(x) = \sum_{i = 1}^n W_{n, i}Y_i$. One thing the author of ESL failed to make this statement sufficiently precise is that they did not specify the convergence mode: the conditions and methods used to prove $\hat{f}(x) \to_p f(x)$, $\hat{f}(x) \to_{a.s.} f(x)$, and $\hat{f}(x) \to_{L^p} f(x)$ are different, where $f(x) = E[Y|X = x]$ (if you are a mathematician, ESL is pretty painful to read...). $\endgroup$
    – Zhanxiong
    Oct 27, 2023 at 0:55


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.