I am using
sklearn.neighbors.KNeighborsRegressor() to train over a set of very high dimensional input (721 dimensions) and continous output (a regression problem). My sample size is 2500. I heard of the curse of dimensionality so I tried implementing PCA to reduce the amount of features. The image below shows the $R^2$ score vs # neighbors for different number of features taken. By the way, these $R^2$ scores are the cross-validation scores reported by
GridSearchCV() and not an out-of-sample score, but they should be similar I guess. The little triangle marks indicate the optimal scores.
Now, I would have expected that lowering the amount of features would get me better scores, as it would help with the curse of dimensionality, but this is not the case. In fact, the more components you take, the better $R^2$ score I get. To be completely honest, the
GridSearchCV() reported that the best parameters were
n_components = 13, n_neighbors = 7, but as you can see in the picture, it might have been just a numerical luck. According to the plot, the performances I get are pretty much the same after 7 features. I would have expected the curves to be higher until some point, and then be lower due to the curse of dimensionality. Why is it not the case?
EDIT: Added a simpler plot of the cross-validation score vs PCA components. As mentioned, after some 10 components, the best scores are pretty much constant.