I need to train a regressor (in the machine learning sense). I have tried many different methods and so far nothing works better than just a constant prediction. In other words in looked like I have no dependency of target on features.
However, I checked if points that a close in terms of their features (next neighbours) tend to have closer values of targets than two randomly selected points and it is the case! So, I take it as an confirmation that features do change something. There is an dependency on the features.
I tried to utilise this observed "dependency" using k-NN method (k-nearest neighbours). So, I started from k = 1 and started to increase k. Accuracy has grown as I have increased k, until k became equal to the number of points in the data set. So, I end up again with the situation where I use an average over all targets as my prediction.
So, to summarise, the nearest neighbour tend to have closer target than a random "neighbour" but the average over all points is still better than average about closest points.
Can I use k-NN in this situation?
Instead of using the number of neighbours k, I have tried to use weighted averaging of target depending on the distance. The weights decay exponentially with increasing distance. I got the similar results, I get the best predictions if I do not let the weights to decay (which means that I take the very distant points with the same weight as the closes points).
So, I see that closest points have on average closer targets than random points (so, there is some dependency on features), however, I cannot manage to use this fact. Because of larger noise and small number of data points the model tries to average out all the points.