For a typical regression algorithm like linear regression, the model is


for instance. We can make predictions

y = 3



enter image description here

Picture above is an example from github.The green point is the 'test data' as the author shows. My question is how can we know the exact position of the 'test_data', since we do not know the y_label of 'test data'. If y_lable is not certain, there might be different neighbors of this 'test data'. Thus, how knn regression predict works?

Not very sure if I had upload the picture successfully, the picture is in the

1.3.4 k-neighbors regression variant model

of the above link


My question is how can we know the exact position of the 'test_data', since we do not know the y_label of 'test data'

The entire point of training the model is to predict the y_label from the feature precisely because we do not have access to the y_label at prediction time. You are right that we are not certain about the y_label even after fitting the model, but in principle if we have enough data (and the data is of high quality) then the predictions should be "good enough" (depending on what "good" means to you).

The mechanics of the prediction are quite straight forward. Find the $k$-nearest data points in the training set (where the distance is measured with respect to the feature, not the y_label) to the prediction point, and then take an arithmetic mean of the y_labels for the $k$-nearest data points. Return the arithmetic mean as the prediction.

  • $\begingroup$ Thanks for the answer, I wonder how the 'distance is measured with respect to the feature', can you give me an example? $\endgroup$ – jasonshu Sep 18 '19 at 2:21
  • $\begingroup$ Distance is usually the euclidean distance. In 1D (as in your example) the distance metric is $\vert x^* - x\vert$. Here, $x^*$ is the prediction feature. This is naturally extended to higher dimensions. You can read up about euclidean distance here: en.wikipedia.org/wiki/Euclidean_distance#Definition. Does that answer your question? $\endgroup$ – Demetri Pananos Sep 18 '19 at 2:31
  • $\begingroup$ That is very precise, I have fully understood how it works. Thank you very much. $\endgroup$ – jasonshu Sep 18 '19 at 2:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.