Suppose a bivariate process is observed $(X_1, Y_1), \ldots, (X_n, Y_n)$ that follows the probability model of ordinary linear regression

$$Y_i = \alpha + \beta X_i + \epsilon_i, $$

and assume $\beta> 0$.

We further are interested in predicting response at a certain exposure level $X$. The distance for an observed $X_i$ from the observation level is noted as $\delta_i = |X_i - X|$, and its order statistic is $\delta_{(i)}$ so that $\delta_{(1)}$ is the closest observed exposure level to the target prediction $X$ WLOG. The rank order set index for the $k$-nearest neighbors is $i: \delta_{(j)} \le \delta_{(k)}$

Take the prediction of the response for a certain level $X$ as

$$ \hat{Y} = \sum_{i: \delta_{(j)} \le \delta_{(k)}} Y_i / k$$

If $X \ge X_{(n)}$ then $\hat{Y} = \sum Y_i I(X \ge X_{(n-k+1)}) / k $, that is the $k$ responses that have $X$ top $k$ ranked response. But $E[Y_i I(X_i \ge X_{(n-k+1)})] < X \beta$. That is to say, all the $k$ neighbors are the highest rank orders of the exposure, and thus the prediction is consistently lower than necessary.

Is this a correct understanding of $k$-nearest neighbors? Is this a known or discussed problem? Are there proposed corrections?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.