I need to train a regressor (in the machine learning sense). I have tried many different methods and so far nothing works better than just a constant prediction. In other words in looked like I have no dependency of target on features.

However, I checked if points that a close in terms of their features (next neighbours) tend to have closer values of targets than two randomly selected points and it is the case! So, I take it as an confirmation that features do change something. There is an dependency on the features.

I tried to utilise this observed "dependency" using k-NN method (k-nearest neighbours). So, I started from k = 1 and started to increase k. Accuracy has grown as I have increased k, until k became equal to the number of points in the data set. So, I end up again with the situation where I use an average over all targets as my prediction.

So, to summarise, the nearest neighbour tend to have closer target than a random "neighbour" but the average over all points is still better than average about closest points.

Can I use k-NN in this situation?


Instead of using the number of neighbours k, I have tried to use weighted averaging of target depending on the distance. The weights decay exponentially with increasing distance. I got the similar results, I get the best predictions if I do not let the weights to decay (which means that I take the very distant points with the same weight as the closes points).

So, I see that closest points have on average closer targets than random points (so, there is some dependency on features), however, I cannot manage to use this fact. Because of larger noise and small number of data points the model tries to average out all the points.

  • $\begingroup$ kNN uses a weighted average, check the documentation. Anyway, kNN with k equal to sample size seems pretty stupid to me, the results may just be random. $\endgroup$ – user2974951 Feb 7 at 11:09

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.