I need to train a regressor (in the machine learning sense). I have tried many different methods and so far nothing works better than just a constant prediction. In other words in looked like I have no dependency of target on features.

However, I checked if points that a close in terms of their features (next neighbours) tend to have closer values of targets than two randomly selected points and it is the case! So, I take it as an confirmation that features do change something. There is an dependency on the features.

I tried to utilise this observed "dependency" using k-NN method (k-nearest neighbours). So, I started from k = 1 and started to increase k. Accuracy has grown as I have increased k, until k became equal to the number of points in the data set. So, I end up again with the situation where I use an average over all targets as my prediction.

So, to summarise, the nearest neighbour tend to have closer target than a random "neighbour" but the average over all points is still better than average about closest points.

Can I use k-NN in this situation?


Instead of using the number of neighbours k, I have tried to use weighted averaging of target depending on the distance. The weights decay exponentially with increasing distance. I got the similar results, I get the best predictions if I do not let the weights to decay (which means that I take the very distant points with the same weight as the closes points).

So, I see that closest points have on average closer targets than random points (so, there is some dependency on features), however, I cannot manage to use this fact. Because of larger noise and small number of data points the model tries to average out all the points.

  • $\begingroup$ kNN uses a weighted average, check the documentation. Anyway, kNN with k equal to sample size seems pretty stupid to me, the results may just be random. $\endgroup$ – user2974951 Feb 7 at 11:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.